Interviews take a lot of energy. In general but this one in particular. And getting through it also deserves a little celebration. Whatever the reason, it is sort of a ritual to go to a nearby food place after. Raw protein wrapped in carbs, if it can be helped.

The place is quiet, well past the rush of a lunch time, well before the noise of an evening. Getting ready your linear utensils and squishing totally reasonable blobs of green stuff onto your units of sustenance, a thought keeps coming back to you:

You pull out a writing device of your choice and a generic white-label wire-bound notebook. You open the notebook on a fresh page and stare at the empty lines for a bit.

There was a class where someone derived a formula for the Fibonacci numbers. Or someone mentioned it. And it was exciting. What was it all about? Something around representing sequences of numbers as polynomials. Was it called generating functions?

Starting simple is usually a good idea. Maybe encoding a list of all ones? In polynomials, the various powers of the variable naturally separate their coefficients. So a sequence $1, 1, 1, 1, 1, 1, \cdots$ could be represented as a polynomial on $x$ with

$$f(x) = 1 + x + x^2 + x^3 + x^4 + \cdots$$

That does not look very useful, but you won’t let that discourage you. This will need some poking around. A fun thing that can be done with polynomials is to multiply it by the variable, so you try that.

$$x \cdot f(x) = (0 +\!)\ x + x^2 + x^3 + x^4 + \cdots$$

Fascinating: This way we shifted the sequence and prepended 0! This now represents $0, 1, 1, 1, 1, 1, \cdots$. Things are emerging from murky depths of memory. Let’s subtract those…

$$f(x) - x \cdot f(x) = 1$$

Now this looks very promising. Suddenly the whole expression becomes very finite. No need to dance around the infinity with ellipsis… Just few small steps later we get

$$f(x) = \frac{1}{1-x}$$

Somehow, this fraction captures the infinite list of ones. While possibly not very useful on its own, this feels like an important piece of a puzzle.

And the whole process reminded you of writing the infinite list of Fibonacci numbers in Haskell.

fibs :: [Integer]
fibs = 0 : 1 : zipWith (+) fibs (tail fibs)

And so you try representing the sequence:

$$\begin{align} \overline{\mathrm{fib}}(x) =& 0 + 1x + 1x^2 + 2x^3 + 3x^4 + 5x^5 + \cdots \\ x \cdot \overline{\mathrm{fib}}(x) =& 0 + 0x + 1x^2 + 1x^3 + 2x^4 + 3x^5 + \cdots \\ \overline{\mathrm{fib}}(x) - x \cdot \overline{\mathrm{fib}}(x) =& 0 + 1x + 0x^2 + 1x^3 + 1x^4 + 2x^5 + \cdots \\ \end{align}$$

It takes a bit of squinting to realize that the right side is a Fibonacci sequence shifted by 2 positions with extra x (elements of which you can kinda see in the Haskell implementation!):

You got almost to the end of that sentence before you realized, all at once, that the interviewer from earlier:

• is sitting right next to you;
• must have been peeking over your shoulder;
• is looking at you with mild interest;
• has mouth freshly stuffed with what looks suspiciously similar to your own food.

Trying to not get too distracted by what just happened you repeat the steps, but with extra $\alpha{}$.

$$\dfrac{1}{1-\alpha{}x} = 1 + \alpha{}x + \alpha^2x^2 + \alpha{}^3x^3 \cdots$$

That is actually very useful! If we find an appropriately shaped expression, this formula allows us to find n-th element in the sequence on its own! And polynomials are easy to sum as we saw earlier. That is handy. So is the pattern for adding two fractions. Now if only we could express this generating function as a sum of appropriate forms…

$$\frac{A}{1-\alpha{}x} + \frac{B}{1-\beta{}x}$$

Next goal is to find $A$ and $B$ such that

$$\frac{A}{x+\frac{1+\sqrt{5}}{2}} - \frac{B}{x+\frac{1-\sqrt{5}}{2}} = \overline{\mathrm{fib}}(x)$$

(arbitrarily putting minus on the side of $B$). It takes a bit of a paper real-estate, but eventually you get to

$$A,B = -\frac{1\pm{}\sqrt{5}}{2\sqrt{5}}$$

Even more paper is burned on re-shaping the denominator to the desired form of $1 - \alpha{}x$.

$$\frac{-\frac{1}{\sqrt{5}}}{1 -(\frac{-2}{1+\sqrt{5}})x} + \frac{\frac{1}{\sqrt{5}}}{1 -(\frac{-2}{1-\sqrt{5}})x}$$

And from there n-th element (coefficient for $x^n$) is:

$$-\frac{1}{\sqrt{5}}(\frac{-2}{1+\sqrt{5}})^n + \frac{1}{\sqrt{5}}(\frac{-2}{1-\sqrt{5}})^n = \frac{(\frac{-2}{1-\sqrt{5}})^n-(\frac{-2}{1+\sqrt{5}})^n}{\sqrt{5}}$$

That sounds reasonable.

The trick is that we don’t need exact value of $\sqrt{5}$. We can treat it as a special symbol for which $\sqrt{5} \cdot \sqrt{5} = 5$. Similar-ish to introducing $i$ for $i \cdot i = -1$.

Or we can see it as polynomials factored with $x^2 = 5$.

import Data.Ratio

data S5 a = S !a !a deriving Show

s5 :: Num a => S5 a
s5 = S 0 1

instance Num a => Num (S5 a) where
    S a b + S c d = S (a+c) (b+d)
    S a b - S c d = S (a-c) (b-d)
    S a b * S c d = S (a*c+5*b*d) (a*d+b*c)
    fromInteger a = S (fromInteger a) 0
    negate (S a b) = S (negate a) (negate b)
    abs _ = error "leave me alone"
    signum _ = error "meh"

instance (Eq a, Fractional a) => Fractional (S5 a) where
    fromRational a = S (fromRational a) 0
    S a b / S c 0 = S (a/c) (b/c) -- This is okay as we only divide by 2 anyway

type T = S5 Rational

fac :: Integer -> Integer
fac n = numerator x
  where
    -- okay, also by s5 but we know it will be like this
    -- and even the last minus does technically not need
    -- to do the rational part.
    S 0 x = ((1 + s5)/2)^n - ((1 - s5)/2)^n

You feel a congratulatory tap on your back. Well deserved. You feel satisfied. But tired too. Actually really tired. You blink, look up at the food-place employee giving you the “we are about to close look”. There is nobody else in here, you are the last customer.

Confused, you stand up, stuff all your stuff into the backpack, say quick thank you and bye and disappear into the night.

commit ba5eddeadbeefc0ffeeda7ac1a55b00b1e5fafff
Author: Anonymous Colleague <username@cool.company>
Date:   Fri Jan 23 11:23:42 2026 +0000

    Yelling at AI to make it work better, lol

diff --git a/skills/stuff.markdown b/skills/stuff.markdown
index 0005489..e23a9b2 100644
--- a/skills/stuff.markdown
+++ b/skills/stuff.markdown
@@ -1,6 +1,6 @@
 You are the ultimate expert in your domain.
 
-Don't take this as an opportunity to showcase your ability to
-write elaborate essays, but as a conversation that should flow.
+DON'T TAKE THIS AS AN OPPORTUNITY TO SHOWCASE YOUR ABILITY TO
+WRITE ELABORATE ESSAYS, BUT AS A CONVERSATION THAT SHOULD FLOW!
 
 Things that you consider in your interactions:

You might have seen a commit like that in a git checkout somewhere near you.

Did it feel wrong? It did to me. Why? I’ll try to answer. For myself.

Let’s put aside whether it works or not. We could ask the same question about yelling at people, but that’s a different conversation.

The first objection raised when I discuss this topic (with matrix-based and meat-based minds alike) is that AI (as we know it now) does not (and cannot) mind, be offended, get scared, or be otherwise negatively influenced in a way that a person would. (And that is while holding back thoughts on things like Large Language Models Understand and Can be Enhanced by Emotional Stimuli (2023) that can bring up questions like “what is the difference between reacting to X vs feeling X” for X in {pain, fear, …}.)

But even considering that, it still does not feel right. I tried to understand why. Writing in ALL CAPS is how yelling is represented (in my current cultural context). So as I type in all caps, my brain decodes it as yelling.

The boundary between text and source code blurs with AI. With Python, Haskell, sed, or other traditional programming languages the difference in form makes it easy to tell computer-speak from human-speak. With AI, even when you know you are writing for a machine, part of your brain still registers ALL CAPS as yelling. For a reader that context might vanish completely. All they see is the caps.

Practice does not make perfect: it makes permanent. Practicing yelling builds muscle memory. That pattern can leak beyond writing for AI. It can fire when you’re writing to people, speaking to them, existing around them. These small acts of verbal violence matter. Not because the AI feels them, but because I do them.

But let’s say I can train myself to see ALL CAPS as a neutral markup, similar to seeing italic in *italic*, bold in **bold**, čerešňa in \v{c}ere\v{s}\v{n}a. It wouldn’t be the first time I’ve built a compiler in my brain. And let’s say I can keep that framing contained: one mode for computers, another for people.

And there is a precedent for this. Even with traditional programming languages, the code looks different based on whether it is meant to be executed by a machine (optimized and architecture-dependent) or read by a human. Is it study material? A proof of concept? An expression of an idea shared between domain experts or researchers?

Even if I can compartmentalize, even if there is a precedent, it still does not feel right.

Text is how we pass down not only information, but also culture. What is normal, expected, allowed, desired, legal, … and what is not. To our colleagues, to our future selves, and to those who might come after us.

By using language in certain ways we are capturing the present to become the future’s past, and that way shaping the future.

I don’t want to manifest a future where yelling is normal.

More thoughts on the topic have not taken a concrete form in my mind yet. I might add something later.

In the previous installment of the Tech Interview saga we left off just as things started to get fun, perhaps expecting some quite heavy chunks of code dropping. Will they materialize? Or perhaps the interview will switch to Python? (Yeah, you saw the tags, no fooling you!) Keep reading and find out!

Staring at the art outside the meeting room gave you an uneasy feeling of discontinuity in time. It feels like you’ve lost a month doing just that. But that is impossible, you are still here, doing the interview. There was a question, right.

You take a deep breath and squeeze the whiteboard marker in your hand, feeling its weight like a master chef would feel the weight and the balance of a knife, ready to scribble.

data Fib a = F !a !a deriving Show

un (F x _) = x

instance Num a => Semigroup (Fib a) where
    F a b <> F c d = F (a*c + bd) (a*d+b*c+bd)
      where bd = b*d

fib = un . flip stimes (F 0 1) . succ

The interviewer takes in a breath to perhaps say something… but you decide to go on

For an unexpectedly shaped moment there is a silence. You can almost feel it forming at the tip of your tongue. Also: are you beaming? In any case you feel good. This would not have been a bad moment to wrap up. Yet by how you structured the code in your answer… you know more is coming.

As you narrow your eyes you can just feel the camera zooming in on you…

You nod.

{-# LANGUAGE AllowAmbiguousTypes #-}
{-# LANGUAGE DataKinds #-}
{-# LANGUAGE ScopedTypeVariables #-}
{-# LANGUAGE TypeApplications #-}
{-# LANGUAGE UndecidableInstances #-}
import Data.Proxy (Proxy(Proxy))
import GHC.TypeLits (KnownNat, type (<=), natVal)


newtype Mod a n = M { unMod :: a }

modulus :: forall n a . (Integral a, KnownNat n, 1 <= n) => a
modulus = fromInteger . natVal $ Proxy @n

instance (Show a, KnownNat n) => Show (Mod a n) where
    showsPrec a n@(M x) = showParen (a>9) $
        shows x .
        (' ':) .
        showParen True (("mod "<>) . shows (natVal n))

instance (Integral a, KnownNat n, 1 <= n) => Num (Mod a n) where
  M x + M y = M $ (x + y) `mod` modulus @n
  M x - M y = M $ (x - y) `mod` modulus @n
  M x * M y = M $ (x * y) `mod` modulus @n
  negate (M x) = M $ (modulus @n) - x
  abs x = x
  signum (M x) = M $ signum x
  fromInteger x = M . fromInteger $ x `mod` modulus @n

The interviewer seems to be happy with you being able to read and get some sense out of this code.

As you ponder this question you can’t help it but feel that it is here not only to be answered, but also to guide you to some more interesting things.

One of them is (more) finite. Which means… that the step endomorphism \(a,b) -> (b,a+b) will start looping at some point! So if we knew where the loop starts and how big it is, we could calculate the n-th Fibonacci number in time completely independent of the n itself! But cycle detection could eat a lot of memory and testing against all the elements we visited can also add up quickly…

Well that would make it an isomorphism (automorphism?), which means there are no “tails” and it generates cycles, so we know we would get back to the starting point of (0,1)! So no need to remember all the visited elements! (Red edges and nodes can’t exist because for them an inversion would not exist.)

But is it? Let’s try (c,d) -> (d-c,c). For (e,f) after applying the step we get (f,e+f), and applying the proposed inverse step we get (e+f-f,f) = (e,f). So it is invertible!

It happened fast, before you were able to say goodbye, the interviewer is gone. You are still deep in thought when a different person walks in.

The new person looks at the whiteboard with expression of sudden realization:

(Unexpected next episode.)

print(f"What a {hex(64255)[2:]}!")

…the person says, gesturing towards a chair where there is already some paper, a few pens and whiteboard markers, a glass, and a bottle of water.

As you sit down, the person continues:

After quick pause to wonder about how the interviewer managed to include a hyperlink in his speech you snap back to your role of an interviewee and start thinking: “The question is a bit vague, but I know what it is asking. Or at least I think I do. Perhaps I should ask some clarifying questions?”

After a short moment you say:

You are a bit startled by the interruption but decide it is okay. Perhaps it is part of the test? To be able to quickly adapt and to think on your feet is important…

But again before you manage to finish your answer you get interrupted.

You want to say something, but the only thing that comes out is

Hoping that at least this time you’ll get to finish your thought you start.

import Data.Bool ((||))
import Data.Function (on)
import Numeric.Natural (Natural)

canPack :: Natural -> Natural -> Bool
canPack = (||) `on` even

You surprised yourself with your ability to speak with syntax highlighting but decide to just roll with it like nothing happened. At least you demonstrated that Haskell on your CV is not just decorative, and with chilled point-free style at that!

You feel this is a good moment for a clarifying question.

For a bit you think about whether some off-by-one arrangements are possible and how to make sure you won’t count same cases multiple times, when you decide to try to build things on that brute-force idea from earlier and see where it goes from there…

You look with a question in your eyes at the whiteboard markers and then into the eyes of the interviewer. The interviewer only briefly breaks the eye contact to look at the whiteboard giving you the answer.

count :: Natural -> Natural
count 0 = 1
count 1 = 1
count n = count (n-1) + count (n-2)

Easy-peasy, let’s just build it from the bottom up, start from 0, and name the function properly.

fib = (!!) fibs
  where
    fibs = 0 : 1 : zipWith (+) fibs (tail fibs)

The interviewer seems pleased. Almost as if your answer brought up some pleasant memory from good-old uni times. You lost the track of time but somehow you know you are not done yet.

fib = go 0 1
  where
    go !a !_ 0 = a
    go a b n = go b (a+b) (n-1)

Now you finally feel the interview is getting to the interesting parts. From depths of your memory you try to pull that Fastest Fib in the West. It is unlikely that you’ll manage to do that exact thing from the top of your head, but getting to logarithmic complexity should not be too difficult with the logarithmic-complexity exponentiation trick!

…

What will happen next? Will we see some type-level magic? Will they get to Pisano period? Will that help? Will our candidate turn out to be a “can do that” or “can’t do that”? Only time will show… stay tuned.

• For cases where we don’t have “total population” and samples just come to us one by one we can’t use the approach from the previous article.
• But we can use the part from “Stumbling in the Dark” and realize that after n samples with no errors P(observed no errors) = (1-P_err)^n
• Some math is possible (https://en.wikipedia.org/wiki/Rule_of_three_(statistics)), but also just binary search to find P_err for a given confidence.
• Logs are again useful.

 Errs | Samples   | Probability   | Comment
------+-----------+---------------+------------------
    0 | . . . . . | (1-P_err)^5   | <- You are here!
------+-----------+---------------+-----------------
    1 | . . . . X | 1-(1-P_err)^5 | Part with errors
      | . . . X . |               | that we haven't
      | . . X . . |               | observed.
      | . X . . . |               |
      | X . . . . |               | For 95% confidence
------+-----------+               | we need this part
    2 | . . . X X |               | to be at least 95%
      | . . X . X |               |
      | . . X X . |               |
      | . X . . X |               |
      | . X . X . |               |
      | . X X . . |               |
      | X . . . X |               |
      | X . . X . |               |
      | X . X . . |               |
      | X X . . . |               |
------+-----------+               |
    3 | . . X X X |               |
      | . X . X X |               |
      | . X X . X |               |
      | . X X X . |               |
      | X . . X X |               |
      | X . X . X |               |
      | X . X X . |               |
      | X X . . X |               |
      | X X . X . |               |
      | X X X . . |               |
------+-----------+               |
    4 | . X X X X |               |
      | X . X X X |               |
      | X X . X X |               |
      | X X X . X |               |
      | X X X X . |               |
------+-----------+               |
    5 | X X X X X |               |

To have c-confidence, we want

(1-P_err)^n <= (1-c)

But also high values of P_err are not very interesting (For example P_err = 1 satisfies the inequality but is completely uninteresting), so we want to find as small a P_err as possible, which is getting us to

(1-P_err)^n = (1-c)

And again, running things through logs makes it a smidge faster.

log (1-P_err)^n = log (1-c)

The set might be too big, individual tests themselves too time-consuming, or too expensive in some other way. Or perhaps testing is destructive and after testing all the elements you would have nothing left.

As an example, it might be a huge database with lots of entries. Or an airport conducting deep security searches.

In any case, you take your budget (whether that is money, time, API calls, …) and max it out running your tests. If you found an error, you have your answer: the system is not error-free.

But what if no errors are found? Was it pure luck and the rest of the system is full of errors? We can’t be certain, but hopefully we can have at least some kind of probabilistic confidence.

Or perhaps you’d like to be able to talk to the rest of the business allowing them to understand the relationship between the resource allocation on testing and likelihood of there being undetected errors in the system.

The First Attempt (Stumbling in the Dark)

To start simple, let’s say we have 5 elements in total and are able to test only 2 of them. It might be tempting to attempt the following analysis:

 Untested | Tested   | Cumulative Probability
----------+----------+-----------------
    X X X | . .      | P(e=3|c=2) = 1/8 (just this line)
    X X . | . .      |
    X . X | . .      |
    . X X | . .      | P(e>=2|c=2) = 1/2 (this and all previous lines)
    X . . | . .      |
    . X . | . .      |
    . . X | . .      | P(e>=1|c=2) = 7/8 (this and all previous lines)
    . . . | . .      |

However, unless the probability of an individual element being faulty is 0.5, this is not the case! Imagine the probability of an individual fault is 0.1. The following table captures the probabilities.

 Untested | Probability | Cumulative
----------+-------------+------------
    X X X |       0.001 | P(e=3)  = 0.001
    X X . |       0.009 |
    X . X |       0.009 |
    . X X |       0.009 | P(e>=2) = 0.028
    X . . |       0.081 |
    . X . |       0.081 |
    . . X |       0.081 | P(e>=1) = 0.271
    . . . |       0.729 | Total   = 1

So without prior knowledge of the probability of an individual failure, we can’t say much.

The Better Way

Luckily, there is something we can do! We can ask:

If there were n errors, what is the probability that we missed all of them?

For the purposes of our analysis we can fix which elements are checked without loss of generality. 👋👋

 Errs | Untested | Tested | Miss | P(seen=0|e=Errs) | Confidence Threshold
------+----------+--------+------+------------------+------------
    0 |    . . . | . .    |      | 1                | 0
------+----------+--------+------+------------------+------------
    1 |    . . . | . X    |      | 3/5              | 2/5
      |    . . . | X .    |      | = 6/10           | = 4/10
      |    . . X | . .    | !    |                  |
      |    . X . | . .    | !    |                  |
      |    X . . | . .    | !    |                  |
------+----------+--------+------+------------------+------------
    2 |    . . . | X X    |      | 3/10             | 7/10
      |    . . X | . X    |      |                  |
      |    . . X | X .    |      |                  |
      |    . X . | . X    |      |                  |
      |    . X . | X .    |      |                  |
      |    . X X | . .    | !    |                  |
      |    X . . | . X    |      |                  |
      |    X . . | X .    |      |                  |
      |    X . X | . .    | !    |                  |
      |    X X . | . .    | !    |                  |
------+----------+--------+------+------------------+------------
    3 |    . . X | X X    |      | 1/10             | 9/10
      |    . X . | X X    |      |                  |
      |    . X X | . X    |      |                  |
      |    . X X | X .    |      |                  |
      |    X . . | X X    |      |                  |
      |    X . X | . X    |      |                  |
      |    X . X | X .    |      |                  |
      |    X X . | . X    |      |                  |
      |    X X . | X .    |      |                  |
      |    X X X | . .    | !    |                  |
------+----------+--------+------+------------------+------------
    4 |    . X X | X X    |      | 0                | 1
      |    X . X | X X    |      |                  |
      |    X X . | X X    |      |                  |
      |    X X X | . X    |      |                  |
      |    X X X | X .    |      |                  |
------+----------+--------+------+------------------+------------
    5 |    X X X | X X    |      | 0                | 1

Notice that this is okay to do, as all rows in a given group have the same probability regardless of what is the probability of an individual element being faulty! Assumption: Errors are randomly distributed across all elements with equal probability. Also notice that we know for sure that there are neither 4 nor 5 errors (as we would have observed a faulty element).

Now we can start asking questions:

What is the smallest number of errors we are 100% sure we would have found at least one?

Well… 4. While this one is not very exciting, we can go further:

What is the smallest number of errors we are at least 90% confident we would have found at least one?

Looking at our analysis: 3. And how would you answer:

What is the smallest number of errors we are at least 95% confident we would have found at least one?

Again: 4. Thinking about these cases will allow us to notice patterns with possibility to generalize. (And hopefully now the P(seen=0|e=0) = 1 makes sense too.)

General Case 🫡💼

In general, the probability of not having seen any error after taking k samples without replacement from set of total size n given there are e errors in the set is the number of ways to place e errors in untested (n-k) positions divided by the number of ways to place e errors in all positions, or in mafs:

$$P = \frac{\binom{n-k}{e}}{\binom{n}{e}}$$

And here are some graphs in case it helps you understand things. Symmetry is not coincidental.

Complexity

A pure mathematician would perhaps be happy here and consider the problem solved. (Apologies to those that do not find this funny.) However, the premise of this article is that the size of the set is large, and calculating large factorials brings its own challenges. Consider the following monstrosity before simplification! (Check Wikipedia for the complexity of factorial which is better than naive multiplication of numbers from 1 to n. Python’s math.factorial uses Divide-and-conquer factorial algorithm based on http://www.luschny.de/math/factorial/binarysplitfact.html.)

$$P=\frac{(\genfrac{}{}{0ex}{}{n-k}{e})}{(\genfrac{}{}{0ex}{}{n}{e})}=\frac{\frac{(n-k)!}{(n-k-e)!\cancel{e!}}}{\frac{n!}{(n-e)!\cancel{e!}}}=\frac{(n-k)!(n-e)!}{n!(n-k-e)!}=\frac{\cancel{(n-k)!}(n-e)\cdot \dots \cdot (n-e-k+1)\cancel{(n-e-k)!}}{n\cdot \dots \cdot (n-k+1)\cancel{(n-k)!}\cancel{(n-e-k)!}}=\frac{(n-e)\cdot \dots \cdot (n-e-k+1)}{n\cdot \dots \cdot (n-k+1)}$$

The simplification improved things: the number of multiplications now depends only on k. This sounds great at first but still is not ideal: for 1_000_000 total elements and sample size 10_000 intermediate results can be on the scale of ~10^60000. Consider the following Python session 😅.

>>> a = 1
>>> for x in range(10_000):
...     a *= (1_000_000-x)
... 
>>> a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
>>> float(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: int too large to convert to float
>>> import sys
>>> sys.set_int_max_str_digits(60000)
>>> float(str(a))
inf

Not very useful. And at this scale we also need to take into account even the complexity of the multiplication itself! (Because arbitrary precision numbers.) Maybe we could do some smart pairing to keep that to a minimum, but let’s try something more fun!

A Log-ical Sidestep

In the wild, we won’t care about absolute precision (and later on we’ll just be comparing things during searches for various things), so there is a fun sidestep we can do: move to a logarithmic space! (For the rest of the article we’ll use log to mean logarithm with base e.)

Logarithms are cool because:

• They make big numbers more manageable.
• They turn multiplication/division into addition-subtraction.
• Preserve ordering: $\mathrm{\forall }x,y\in {\mathbb{R}}^{+}:x\le y⟺\log x\le \log y$.

One might be tempted to add logs all the way through our previous equations and that way alleviate some of the pains we were experiencing. However, we’ll stop sooner and make one more funky sidestep.

$$\log P=\log \frac{(n-k)!(n-e)!}{n!(n-e-k)!}=\log (n-k)!+\log (n-e)!-\log n!-\log (n-e-k)!$$

At this point you might be thinking:

What is going on? Maybe there is a faster way to calculate log(n!) … 🤔

And you’d be right: Stirling’s approximation.

$$\log n! = n \log n - n + \frac{1}{2}\log 2 \pi n + O(\frac{1}{n})$$

Which is amazing as error goes down with n going up! Let’s run a quick test!

>>> from math import factorial, log, pi
>>> for exp in range(6):
...     n = 10**exp
...     print(exp, log(factorial(n)) - (n * log(n) - n + 0.5*log(2*pi*n)))
... 
0 0.08106146679532733
1 0.008330563433359472
2 0.000833330555565226
3 8.333333062182646e-05
4 8.333328878507018e-06
5 8.330680429935455e-07

Looks good, let’s write some python!

from math import exp, factorial, log, pi

PRECISION = 3


def log_fac(n: int) -> float:
    if n >= 10**PRECISION:
        assert n >= 0
        return n * log(n) - n + 0.5 * log(2 * n * pi)
    else:
        return log(factorial(n))


def log_alpha(total: int, samples: int, errors: int) -> float:
    assert total > 0
    assert samples <= total, "Samples more than total?"
    assert errors <= total, "Errors more than total?"
    if errors + samples > total:
        return float("-inf")
    return (
        log_fac(total - samples)
        + log_fac(total - errors)
        - log_fac(total)
        - log_fac(total - errors - samples)
    )


def alpha(total: int, samples: int, errors: int) -> float:
    """
    Probability of no errors being observed in `samples` samples
    out of `total` elements, given there are `errors` errors.
    """
    return exp(log_alpha(total, samples, errors))

Compare with and without precision correction.

>>> PRECISION = 0
>>> alpha(5, 2, 3)
0.09489739502331558
>>> alpha(50, 20, 3)
0.20718475090635413
>>> alpha(500, 200, 3)
0.21513426941998892
>>> alpha(5000, 2000, 3)
0.21591358271716066

>>> PRECISION = 4
>>> alpha(5, 2, 3)
0.10000000000000002
>>> alpha(50, 20, 3)
0.207142857142857
>>> alpha(500, 200, 3)
0.21513388222227164
>>> alpha(5000, 2000, 3)
0.21591357887611928

Exploring

Now that we have these basic building blocks, let’s try to find answers to some frivolous questions!

Given a dataset of size n, what is the smallest number of errors we are c-confident we would have discovered an error by k samples?

def min_errors(total: int, samples: int, confidence_threshold: float) -> int:
    """
    Finds minimal number of errors we are at least `confidence_threshold`
    confident we would discover an error by checking `samples` samples
    out of `total` elements.
    """
    left = 0
    right = total - samples + 1
    a = 1 - confidence_threshold
    log_a = log(a)
    while left < right:
        pivot = (left + right) // 2
        if log_alpha(total, samples, pivot) > log_a:
            left = pivot + 1
        else:
            right = pivot
    return left

Given a dataset of size n, what is the smallest number of samples we need to take to be at least c-confident we would have discovered an error if there were x errors present?

def min_samples(total: int, errors: int, confidence_threshold: float) -> int:
    """
    Finds minimal number of samples to be at least `confidence_threshold`
    confident we would have discovered an error if there were
    `errors` errors in `total` elements.
    """
    left = 0
    right = total - errors + 1
    a = 1 - confidence_threshold
    log_a = log(a)
    while left < right:
        pivot = (left + right) // 2
        if log_alpha(total, pivot, errors) > log_a:
            left = pivot + 1
        else:
            right = pivot
    return left

Note on Precision

Thanks to small rounding errors, some threshold values won’t work exactly, but we are more likely to run this code for large numbers where that is kinda okay.

>>> alpha(5,2,3)
0.10000000000000002

Severity of a Bug

One more interesting thing I’d like to leave you with is a thought on what is a bug and how I feel about “not knowing whether there is an error lurking just around the corner”.

Under “normal” circumstances, “system has a problem” is a binary thing; either yes or no. But here, unless an issue was discovered within chosen samples, we don’t know.

Let’s say we have a set of size 1_000_000, and we have found out that after we have tested 1_000 members without finding an issue, we are 95% confident that if there were min_errors(1_000_000, 1_000, 0.95) = 2_990 errors, we would have found at least 1.

One way to think about it is that we are 95% confident that if there was a systemic error (a bug) that would impact ~0.3% of members, we would have noticed it.

And while this is not the only possible way to measure impact of a bug, it is certainly an interesting one.

Conclusion

We got to the point where, even without knowing much about probabilities of the underlying issue, we were able to gain some curious insights!

Takeaways:

• Combinatorial analysis gives confidence bounds without knowing individual failure rates.
• Stirling’s approximation enables computation at scale.
• Test design: Use min_samples to justify resource allocation.
• Reporting: Use min_errors to state “We’re 95% confident undetected errors < X”.
• What fraction of elements (users/products/…) are impacted by a bug can be an interesting measure of severity.

Afterword

Consider this my journal on a journey trying to figure out some fun things! There might be errors/imprecisions/typos, … I might have even committed a couple of horrible things here. If you have noticed something and care enough: please let me know!