The T20 World Cup from NaN
Why our trivia engine never trusts itself — a post-mortem of the bug that shipped with a green validator.
The pill that said NaN
On our cricket site, every nation page carries a small coverage pill — the honest label that says which slice of history the numbers below actually cover. For India it should read: covers the ODI World Cup from 2003 and the T20 World Cup from 2007. For a while, in production, it read: the T20 World Cup from NaN.
The culprit was a one-line year parser. Our cricket editions are keyed like odi-2003 and t20-2024, and the parser extracted the year by stripping leading non-digits. On odi-2003 that works: strip odi-, parse 2003. But t20-2024 starts with a t followed by digits — so only the t was stripped, and Number("20-2024") is NaN. The same parser sat inside the sorter that orders a nation’s title chips, so “chronological” had silently degraded to alphabetical too. Nobody free-wrote a wrong fact. The machine computed one, twice, identically.
That last word is the whole story. Our validator — the thing that exists to catch exactly this — stayed green.
What this thing is
Scorewit is a family of five daily sports-trivia games — the men’s soccer World Cup, cricket’s World Cups, the World Series, Formula 1, and the NFL — each a single static page, six questions a day, the same round for everyone. Every answer, every reveal fact, every stats page is computed from an openly licensed dataset (openfootball, Cricsheet, Retrosheet, F1DB, nflverse) and linked to its source. Nothing is written by hand; nothing comes from a model’s memory of sport.
The pipeline, and why it’s deterministic
Each game is built by the same four-stage pipeline: ingest normalizes the upstream dataset into a committed snapshot; generate derives a question bank and the stats artifacts from it with a seeded RNG; validate re-derives everything and refuses to ship on any mismatch; render emits one self-contained HTML file. There are no clocks and no randomness outside the seed, so two builds from the same snapshot are byte-identical — which turns “did my refactor change behavior?” into a checksum comparison. The nightly refresh regenerates from the newest upstream release, and the git push is the deploy.
The firewall
The validate stage is the part we actually trust — so we don’t let it take the generator’s word for anything. For every question it re-derives the answer from raw match rows on its own code path: recount the podiums, re-walk the win streak, re-sort the standings. It checks that every wrong option is genuinely wrong, that every reveal fact carries the numbers it claims, that open-ended career claims are only made about retired drivers (a future race can’t falsify a closed career), and that every citation resolves into the pinned dataset release. Emit and verify compute the same value from different directions, and the build demands byte-equality between them. Disagreement means no deploy.
Teaching the machine to be interesting, safely
Raw stats are inert — “242 starts, 71 wins” means little to a casual fan. So the stats pages open with a composed lead: “wins nearly one race in three; on the podium more often than not.” Those lines come from an insight engine that is deliberately boring underneath: a library of templates, each a pure function of independently computed stats, each with a predicate that must hold before its line may fire. Wit is earned by thresholds — a midfield team never gets called dominant, because that template’s predicate won’t pass; a different, honest line fires instead. Every rounded figure is hedged (“roughly”, “nearly”, “better than”) by the same rounding helper, so approximation stays truthful. Superlatives — “only”, “no one else” — are emitted solely from computed ranks with strict uniqueness checks. Anything mid-tournament is suppressed until the edition completes, because an in-progress superlative is a claim tomorrow’s match can falsify. And entities with nothing to boast about get their genuine angle, never snark: the wit is about achievement, not absence. The validator recomposes each lead from its own independently derived stats and demands the same bytes.
Anatomy of the escape
So how did NaN walk through all of that? Because the coverage pill’s year parse was a shared helper. The page-emit code called it, and the validator — re-deriving the pill to compare — called the same function. Both sides computed NaN. Both sides produced the same wrong string. Byte-equality held, the build went green, and the bug shipped.
That’s the structural blind spot of any re-derivation guarantee: independence is only as deep as the deepest shared dependency. The moment emit and verify share a line of code, that line is validating itself. We hadn’t built a second deriver; we’d built one deriver with two callers and congratulated it for agreeing with itself.
The rule that came out of it
The fix for the parser was one line. The fix for the process is now FIREWALL.md in the engine repo, and it classifies every helper the firewall touches:
Derivations — anything that computes a fact from primitives (wins, titles, ranks, coverage boundaries) — may never be shared. The validator re-derives them on its own code path from raw rows, not even through the emit side’s convenience indexes. Pure formatters — total, fact-free transforms like a year parser or a number-to-word helper — may be shared, but only under three conditions at once: small enough to verify by reading; unit-tested in isolation against normal, boundary, and garbage inputs (a malformed input must throw or be handled, never a silent NaN); and anchored by ground-truth pins — hardcoded known-correct values asserted against the real dataset, like “cricket coverage is ODI from 2003 and T20 from 2007”, “Brazil has five World Cup titles”, “the F1 drivers’ record is seven”. A pin catches a shared-helper regression precisely where byte-equality can’t, because the pin doesn’t ask the code — it asks history. If a pin fires when the source legitimately changes, that’s a feature: someone updates it on purpose.
What byte-equality still doesn’t buy
Honesty about the guarantee, then. Byte-equality between emit and verify does not guarantee the shared-formatter class of bug is gone — pins and tests shrink it; nothing eliminates it. It does not guarantee the upstream dataset is right: if the source records the wrong scoreline, we will faithfully compute, verify, and cite the wrong scoreline. It does not validate editorial decisions — alias tables, quota choices, which archetypes exist — because there is no independent way to re-derive a decision. And it says nothing about taste: a line can be verified and still be a bad sentence. What it does buy is narrower and worth having: no fact reaches a page unless two code paths, written to be as independent as we can make them, computed the same thing from the same committed snapshot — and when that promise bends, as it did here, the failure is legible enough to become a rule.
The code is public
The engine behind all five games — pipeline, firewall, insight engine, and the FIREWALL.md this bug wrote — is open at github.com/aifounder7/scorewit-core. The bug described here is commit 6b1f930 in the cricket repo’s history, fix and confession in the same diff.