Every model we tested, and what won.
AI models do our reading, so which model reads matters. We score every candidate against an answer key before it reads anything we publish, and we publish every score: 24 tests, 159 results, 58 models and setups, including the ones that failed. Use them, check them, and tell us how to do better.
Read this first
A score means “agrees with our key”, not “is right”.
- AI models made the keys.Every answer key here was made by AI readers, except one check against labels that people made in another study. No person has checked our keys yet; that check is designed and waiting for funding.
- Small keys, on purpose.Most keys hold a few dozen serious errors, and some over-sample the rare cases so they can be tested at all. Each test says how its key was made.
- Chosen, then tested fresh.Where a setup was picked on one set of citations, the score that counts is the one on a fresh set it had never seen. Tests registered in advance say so.
- Costs are per 1,000 citations.List price means usage times the provider’s published price on the date given. Some models ran on included usage with no per-call charge.
Every test
From the first bake-off to this week.
Open a test to see every model and setting it tried, the score, the cost, and why we kept it or dropped it. All of it is in one CSV.
Census reading
Does a paper that cites a retracted paper use it as support, mention it in passing, or show it knows of the retraction? Scored as the share of citations where the model's label matches the answer key.
13 Oct 2026Phase 2 pilot bake-off v1 (census labelling task, codebook v1)Kept: nothing kept
Answer key. 73 random development citations labelled by one AI reader (Claude), codebook v1; AI-made; n=73
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| GLM 5.2Zhipu AI (Z.ai) | Cloudflare Workers AI | prompt codebook v1, one call per citation, default settings; model id cf/zai-org/glm-5.2 | candidate | accuracy (label = key)66 of 73 (90.4%) | – | $0.53 | CandidateTop single score on v1; re-tested on the 118-item key. |
| gpt-oss-120bOpenAI | Cloudflare Workers AI | prompt codebook v1, one call per citation, default settings; model id cf/openai/gpt-oss-120b | candidate | accuracy (label = key)65 of 73 (89.0%) | – | $0.41 | CandidateSecond on v1; re-tested on the 118-item key. |
| ClefCloudflare | Cloudflare Workers AI | prompt codebook v1, one call per citation, default settings; model id cf/cloudflare/clef | candidate | accuracy (label = key)63 of 73 (86.3%) | – | $0.12 | CandidateFast and cheap typed-choice model; re-tested on the 118-item key. |
| Kimi K2.6Moonshot AI | Cloudflare Workers AI | prompt codebook v1, one call per citation, default settings; model id cf/moonshotai/kimi-k2.6 | candidate | accuracy (label = key)61 of 73 (83.6%) | – | $0.14 | CandidateCaught every retraction-aware item; slow (about 59 s per call). |
| Gemma 4 26B A4BGoogle | Cloudflare Workers AI | prompt codebook v1, one call per citation, default settings; model id cf/google/gemma-4-26b-a4b-it | candidate | accuracy (label = key)61 of 73 (83.6%) | – | $0.05 | CandidateCheapest of the strong models; re-tested on the 118-item key. |
| JevTypeSafe | TypeSafe API (Cloudflare AI Gateway route returned HTTP 402) | prompt codebook v1, one call per citation, default settings; id typesafe/jev | candidate | accuracy (label = key)60 of 73 (82.2%) | – | $0.03 | CandidateTyped-choice model; re-tested on the 118-item key. |
| Nemotron 3 Super 120B A12BNVIDIA | Cloudflare Workers AI | prompt codebook v1, one call per citation, default settings; model id cf/nvidia/nemotron-3-120b-a12b | candidate | accuracy (label = key)60 of 73 (82.2%) | – | $0.27 | CandidateMid-pack; re-tested on the 118-item key. |
| DeepSeek V4 ProDeepSeek | Cloudflare Workers AI | prompt codebook v1, one call per citation, default settings; model id cf/deepseek-ai/deepseek-v4-pro-0813 | candidate | accuracy (label = key)58 of 73 (79.5%) | – | $0.51 | CandidateMid-pack and the dearest per call; re-tested on the 118-item key. |
| gpt-oss-20bOpenAI | Cloudflare Workers AI | prompt codebook v1, one call per citation, default settings; model id cf/openai/gpt-oss-20b | candidate | accuracy (label = key)57 of 73 (78.1%) | – | $0.28 | Not usedBelow the larger gpt-oss on the same items; not carried forward. |
| Llama 3.3 70B Instruct (fp8 fast)Meta | Cloudflare Workers AI | prompt codebook v1, one call per citation, default settings; model id cf/meta/llama-3.3-70b-instruct-fp8-fast | candidate | accuracy (label = key)55 of 73 (75.3%) | – | $0.19 | Not usedOver-called substantive use; not carried forward. |
| Llama 4 Scout 17B 16EMeta | Cloudflare Workers AI | prompt codebook v1, one call per citation, default settings; model id cf/meta/llama-4-scout-17b-16e-instruct | candidate | accuracy (label = key)54 of 73 (74.0%) | – | $0.14 | Not usedSecond lowest score; not carried forward. |
| GLM 4.7 FlashZhipu AI (Z.ai) | Cloudflare Workers AI | prompt codebook v1, one call per citation, default settings; model id cf/zai-org/glm-4.7-flash | candidate | accuracy (label = key)43 of 73 (58.9%) | – | $0.04 | Not usedLowest score (59%); not carried forward. |
| Qwen3 30BAlibaba (Qwen) | Cloudflare Workers AI | codebook v1 | candidate | not scorednot recorded | – | not recorded | Not usedReturned empty content, so it was dropped before scoring. |
Rows T01-01 to T01-13 in the CSV.
13 Oct 2026Phase 2 pilot bake-off v2 (census labelling task, codebook v2)Kept: Jev, Gemma 4 26B A4B, Clef, Kimi K2.6, Panel: Clef + Jev, Kimi K2.6 breaks ties (3-way split goes to Clef)
Answer key. 118 citations: 73 random development items plus 45 retraction-aware challenge items, labelled by Claude, codebook v2; AI-made; aware items over-sampled on purpose (27 aware)
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| JevTypeSafe | TypeSafe API (Cloudflare AI Gateway route returned HTTP 402) | prompt codebook v2, one call per citation, default settings; id typesafe/jev | panel member | accuracy (label = key)110 of 118 (93.2%) | – | $0.04 | ChosenBest single model (93%); panel member. |
| Gemma 4 26B A4BGoogle | Cloudflare Workers AI | prompt codebook v2, one call per citation, default settings; model id cf/google/gemma-4-26b-a4b-it | panel member | accuracy (label = key)108 of 118 (91.5%) | – | $0.07 | ChosenSecond best and cheapest; became the census panel's third model when Kimi proved too slow. |
| ClefCloudflare | Cloudflare Workers AI | prompt codebook v2, one call per citation, default settings; model id cf/cloudflare/clef | panel member | accuracy (label = key)107 of 118 (90.7%) | – | $0.18 | ChosenFast and cheap; panel member and tie-breaker. |
| Kimi K2.6Moonshot AI | Cloudflare Workers AI | prompt codebook v2, one call per citation, default settings; model id cf/moonshotai/kimi-k2.6 | panel member | accuracy (label = key)106 of 118 (89.8%) | – | $0.34 | ChosenCaught all 27 aware items; pilot tie-breaker; later dropped from the census as too slow. |
| GLM 5.2Zhipu AI (Z.ai) | Cloudflare Workers AI | prompt codebook v2, one call per citation, default settings; model id cf/zai-org/glm-5.2 | candidate | accuracy (label = key)105 of 118 (89.0%) | – | $0.61 | Not usedLower than the chosen three and the dearest per item. |
| DeepSeek V4 ProDeepSeek | Cloudflare Workers AI | prompt codebook v2, one call per citation, default settings; model id cf/deepseek-ai/deepseek-v4-pro-0813 | candidate | accuracy (label = key)103 of 118 (87.3%) | – | $0.63 | Not usedLower score at a high price. |
| gpt-oss-120bOpenAI | Cloudflare Workers AI | prompt codebook v2, one call per citation, default settings; model id cf/openai/gpt-oss-120b | candidate | accuracy (label = key)102 of 118 (86.4%) | – | $0.51 | Not usedLower score; 3 replies cut off; quotes often not found in the passage (69%). |
| Nemotron 3 Super 120B A12BNVIDIA | Cloudflare Workers AI | prompt codebook v2, one call per citation, default settings; model id cf/nvidia/nemotron-3-120b-a12b | candidate | accuracy (label = key)102 of 118 (86.4%) | – | $0.41 | Not usedLower score; missed the most aware items. |
| Panel: Clef + Jev, Kimi K2.6 breaks ties (3-way split goes to Clef)Cloudflare, TypeSafe, Moonshot AI | Cloudflare Workers AI; TypeSafe API | codebook v2; Kimi asked only when Clef and Jev differ | panel | accuracy (label = key)114 of 118 (96.6%) | – | not recorded | ChosenBeat every single model, so it labelled the pilot; chosen on these same 118 items, so the score is in-sample. |
Rows T02-01 to T02-09 in the CSV.
13 Oct 2026Phase 2 pilot blind holdoutKept: Panel: Clef + Jev, Kimi K2.6 breaks ties
Answer key. 73 random pilot citations (80 drawn, 7 skipped because the citation marker was not visible) labelled blind by Claude before seeing model output, codebook v2; AI-made
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Panel: Clef + Jev, Kimi K2.6 breaks tiesCloudflare, TypeSafe, Moonshot AI | Cloudflare Workers AI; TypeSafe API | codebook v2 (pilot panel) | panel | accuracy (label = key)70 of 73 (95.9%) | – | $0.25 | ChosenPassed the blind check at 95.9% (Wilson 88.6-98.6%), so the pilot rates were published with it. |
Rows T03-01 to T03-01 in the CSV.
13 Oct 2026Census codebook v3 development scoresKept: Jev, Gemma 4 26B A4B, Clef, Panel v3.1: Jev + Gemma 4 + Clef (majority; aware rule)
Answer key. 130 fresh development pairs (calib_v3), Claude blind labels, codebook v3; AI-made; used to tune the codebook, so in-sample
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| JevTypeSafe | Cloudflare AI Gateway | codebook v3, pair label = strongest passage label | panel member | pair accuracy (label = key)119 of 130 (91.5%) | – | not recorded | ChosenJoint best on development; census panel member. |
| Gemma 4 26B A4BGoogle | Cloudflare Workers AI | codebook v3, pair label = strongest passage label | panel member | pair accuracy (label = key)119 of 130 (91.5%) | – | not recorded | ChosenJoint best on development and cheap; replaced Kimi in the census panel. |
| ClefCloudflare | Cloudflare Workers AI | codebook v3, pair label = strongest passage label | panel member | pair accuracy (label = key)114 of 130 (87.7%) | – | not recorded | ChosenFastest; census panel member and tie-breaker. |
| Kimi K2.6Moonshot AI | Cloudflare Workers AI | codebook v3, pair label = strongest passage label | candidate | pair accuracy (label = key)110 of 130 (84.6%) | – | not recorded | Not usedLowest of the four and about 124 s per call, too slow for 72,000 passages. |
| Panel v3.1: Jev + Gemma 4 + Clef (majority; aware rule)TypeSafe, Google, Cloudflare | Cloudflare AI Gateway; Cloudflare Workers AI | codebook v3.1 frozen 13 Oct | panel | pair accuracy (label = key)125 of 130 (96.2%) | – | not recorded | ChosenFrozen for the census after this development score; in-sample. |
Rows T04-01 to T04-05 in the CSV.
13 Oct 2026Census tie-breaker trial on non-unanimous passagesKept: Panel v3.1 majority (Jev + Gemma 4 + Clef)
Answer key. 55 development passages where the three panel models did not all agree, Claude labels; AI-made; in-sample
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| GLM 5.2Zhipu AI (Z.ai) | Cloudflare Workers AI | codebook v3.1, tie-breaker vote on the 55 passages | candidate | passage accuracy (label = key)40 of 55 (72.7%) | – | not recorded | Not usedWorse than the panel's own majority (45/55), so no tie-breaker was added. |
| gpt-oss-120bOpenAI | Cloudflare Workers AI | codebook v3.1, tie-breaker vote on the 55 passages | candidate | passage accuracy (label = key)40 of 55 (72.7%) | – | not recorded | Not usedWorse than the panel's own majority (45/55), so no tie-breaker was added. |
| Qwen 3.8 27BAlibaba (Qwen) | Cloudflare Workers AI | codebook v3.1, tie-breaker vote on the 55 passages | candidate | passage accuracy (label = key)37 of 55 (67.3%) | – | not recorded | Not usedWorse than the panel's own majority (45/55), so no tie-breaker was added. |
| Panel v3.1 majority (Jev + Gemma 4 + Clef)TypeSafe, Google, Cloudflare | Cloudflare AI Gateway; Cloudflare Workers AI | codebook v3.1 | panel | passage accuracy (label = key)45 of 55 (81.8%) | – | not recorded | ChosenThe bar the tie-breakers had to beat; this zone is real ambiguity, reported as a band. |
Rows T05-01 to T05-04 in the CSV.
13 Oct 2026Census blind holdout (codebook v3.1)Kept: Panel v3.1: Jev + Gemma 4 + Clef
Answer key. 300 random census pairs (seed 31337, 367 passages), Claude Opus 5.5 blind labels written before any model output was read, codebook v3.1; AI-made
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Panel v3.1: Jev + Gemma 4 + ClefTypeSafe, Google, Cloudflare | Cloudflare AI Gateway; Cloudflare Workers AI | codebook v3.1 frozen | panel | pair accuracy (label = key)277 of 300 (92.3%) | – | not recorded | ChosenClean pre-registered score for v3.1; its errors led to v3.3, so this set is no longer clean for v3.3. |
Rows T06-01 to T06-01 in the CSV.
13 Oct 2026Census confirmation set (codebook v3.3)Kept: Panel v3.3: Jev + Gemma 4 + Clef (Clef breaks ties), Jev, Clef, Gemma 4 26B A4B
Answer key. 150 fresh census pairs (seed 4242, 194 passages), Claude Opus 5.5 blind labels, codebook v3.3; AI-made; this key supplies the census misclassification correction
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Panel v3.3: Jev + Gemma 4 + Clef (Clef breaks ties)TypeSafe, Google, Cloudflare | Cloudflare AI Gateway; Cloudflare Workers AI | codebook v3.3 frozen | panel | pair accuracy (label = key)142 of 150 (94.7%) | – | not recorded | ChosenClean confirmation of the frozen census panel (Wilson 89.8-97.3%); all-three-agree pairs 123/126. |
| JevTypeSafe | Cloudflare AI Gateway | codebook v3.3, each model alone | panel member | pair accuracy (label = key)142 of 150 (94.7%) | – | not recorded | ChosenBest single census model (94.7%). |
| ClefCloudflare | Cloudflare Workers AI | codebook v3.3, each model alone | panel member | pair accuracy (label = key)138 of 150 (92.0%) | – | not recorded | Chosen92.0% alone; kept as tie-breaker. |
| Gemma 4 26B A4BGoogle | Cloudflare Workers AI | codebook v3.3, each model alone | panel member | pair accuracy (label = key)136 of 150 (90.7%) | – | not recorded | Chosen90.7% alone; third panel member. |
| Two-model fallback: Gemma 4 + Clef, Clef breaks tiesGoogle, Cloudflare | Cloudflare Workers AI | codebook v3.3; used only where Jev had no vote | panel (fallback) | pair accuracy (label = key)138 of 150 (92.0%) | – | not recorded | BackupReady when Jev ran out of paid balance; in the end every census passage got a Jev vote. |
| Two-model fallback: Clef + Gemma 4, Gemma breaks tiesCloudflare, Google | Cloudflare Workers AI | codebook v3.3 | panel (fallback) | pair accuracy (label = key)136 of 150 (90.7%) | – | not recorded | Not usedLost to Clef as tie-breaker on the clean set. |
Rows T07-01 to T07-06 in the CSV.
14 Oct 2026Census test-retest (same passages asked again)Kept: Jev, Gemma 4 26B A4B, Clef, Panel v3.3: Jev + Gemma 4 + Clef
Answer key. No answer key: each model's own first label on 500 random census passages (seed 99)
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| JevTypeSafe | Cloudflare AI Gateway | codebook v3.3, second call on the same passage | panel member | same label on repeat496 of 500 (99.2%) | – | not recorded | ChosenLabels hold on a repeat call, so hosted-model drift within the run was small. |
| Gemma 4 26B A4BGoogle | Cloudflare Workers AI | codebook v3.3, second call on the same passage | panel member | same label on repeat494 of 500 (98.8%) | – | not recorded | ChosenLabels hold on a repeat call, so hosted-model drift within the run was small. |
| ClefCloudflare | Cloudflare Workers AI | codebook v3.3, second call on the same passage | panel member | same label on repeat500 of 500 (100.0%) | – | not recorded | ChosenLabels hold on a repeat call, so hosted-model drift within the run was small. |
| Panel v3.3: Jev + Gemma 4 + ClefTypeSafe, Google, Cloudflare | Cloudflare AI Gateway; Cloudflare Workers AI | codebook v3.3, second call on the same passage | panel | same label on repeat496 of 500 (99.2%) | – | not recorded | ChosenLabels hold on a repeat call, so hosted-model drift within the run was small. |
Rows T08-01 to T08-04 in the CSV.
14 Oct 2026Census 'aware' label against human labels (Hsiao and Schneider 2021)Kept: Panel v3.3: Jev + Gemma 4 + Clef ('aware' label)
Answer key. Hsiao and Schneider (QSS 2021) hand-checked labels of whether a citing paper acknowledges the retraction, on 8,944 citation pairs both studies cover; HUMAN-made; a related but not identical measure
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Panel v3.3: Jev + Gemma 4 + Clef ('aware' label)TypeSafe, Google, Cloudflare | Cloudflare AI Gateway; Cloudflare Workers AI | codebook v3.3 | panel | agreement, aware vs their acknowledgement8,737 of 8,944 (97.7%) | 0.743 (aware vs not, 2 classes) | not recorded | ChosenThe only test so far against human labels; agreement is high but the panel finds 81% of their acknowledgements. |
Rows T09-01 to T09-01 in the CSV.
29 Oct 2026Census reader calibration with Cloudflare off (confirm_v33)Kept: Panel: Gemini 3.8 Flash + Claude Sonnet 5.5 + gpt-oss-120b, Claude Haiku 5.5, Claude Opus 5.5
Answer key. 150 census pairs (194 passages), Claude Opus 5.5 blind labels, codebook v3.3; AI-made; same key as T07
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Gemini 3.8 FlashGoogle | Google Antigravity, Google Gemini API and OpenRouter (by host: 88/91, 4/4, 54/55) | thinking high, codebook v3.3 | panel member | pair accuracy (label = key)146 of 150 (97.3%) | – | not recorded | BackupBest single census reader on this key; free capacity ran out (about 125 calls a day on the free key). |
| Claude Sonnet 5.5Anthropic | Anthropic (Claude app) | Claude app, codebook v3.3 | panel member | pair accuracy (label = key)141 of 150 (94.0%) | – | no per-call charge | Not usedLowest of the Claude models on this key; same family as the key's labeller. |
| gpt-oss-120bOpenAI | Google Antigravity and OpenRouter (by host: 84/92, 53/58) | reasoning medium, codebook v3.3 | backup reader | pair accuracy (label = key)137 of 150 (91.3%) | – | not recorded | BackupLowest score here, but a different family and not a census panel member; registered fallback for census-task Claude stops. |
| Panel: Gemini 3.8 Flash + Claude Sonnet 5.5 + gpt-oss-120bGoogle, Anthropic, OpenAI | as the members | census combine rule, Gemini breaks 3-way ties | panel | pair accuracy (label = key)143 of 150 (95.3%) | – | not recorded | ChosenPassed the 89.8% bar, so it labelled the economics pilot. |
| Panel without Claude: Gemini 3.8 Flash + gpt-oss-120bGoogle, OpenAI | as the members | Gemini breaks ties | panel | pair accuracy (label = key)145 of 150 (96.7%) | – | not recorded | CandidateReported as registered; scored higher than with Claude. |
| Muse Spark (Meta; version not recorded)Meta | Meta (Muse app) | effort medium, codebook v3.3 | backup reader | pair accuracy (label = key)144 of 150 (96.0%) | – | no per-call charge | BackupPassed as a free backup census reader; labelled all 194 passages. |
| Panel: Muse + Claude Sonnet 5.5 + gpt-oss-120bMeta, Anthropic, OpenAI | as the members | Muse in Gemini's slot, Muse breaks ties | panel | pair accuracy (label = key)144 of 150 (96.0%) | – | not recorded | BackupPassed the 89.8% bar, so it can replace the Gemini panel. |
| Panel without Claude: Muse + gpt-oss-120bMeta, OpenAI | as the members | Muse breaks ties | panel | pair accuracy (label = key)143 of 150 (95.3%) | – | not recorded | CandidateReported as registered; not a pass bar. |
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort medium, JSON via output_config.format, cached system prompt | reader | pair accuracy (label = key)147 of 150 (98.0%) | 0.961 (4 labels, pair level) | $0.11 | ChosenPassed (98.0%, kappa 0.961) at about 1/60 of Opus's cost; became the bulk census reader on the API credit. |
| Claude Sonnet 5.5Anthropic | Anthropic Message Batches API | effort high | candidate | pair accuracy (label = key)143 of 150 (95.3%) | 0.910 (4 labels, pair level) | $0.93 | Not usedLower than Haiku at about eight times the cost. |
| Claude Opus 5.5Anthropic | Anthropic Message Batches API | effort medium (31 Oct) | reader | pair accuracy (label = key)140 of 147 (95.2%) | 0.908 (4 labels, pair level) | $6.96 | ChosenSelf-check, since Opus wrote this key: passed its 95% bar, so it reads Part B of the batch 7 key. |
Rows T21-01 to T21-11 in the CSV.
Citation accuracy
Does a citation say what the cited paper says? Scored mainly on serious errors: how many of the key's serious errors a setup catches (recall), how many of its calls are right (precision), and kappa.
21 Oct 2026Stage 3 pilot: citation-accuracy panel vs a blind second readKept: nothing kept
Answer key. 100 random pilot citations read blind by Claude (one AI reader, model not recorded), v1 rules; AI-made
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Gemma 4 26B A4BGoogle | Cloudflare Workers AI | v1 prompt; default settings (later found off the makers' advice) | panel member | exact verdict agreement with blind read72 of 100 (72.0%) | 0.444 (4 verdicts) | $0.19 | Not usedClosest to the blind read of the three. |
| Nemotron 3 Super 120B A12BNVIDIA | Cloudflare Workers AI | v1 prompt; default settings (later found off the makers' advice) | panel member | exact verdict agreement with blind read63 of 100 (63.0%) | 0.173 (4 verdicts) | $1.02 | Not usedRarely called major (3.9%); its JSON needed the lenient reader 1,236 times of 1,600. |
| gpt-oss-120bOpenAI | Cloudflare Workers AI | v1 prompt; default settings (later found off the makers' advice) | panel member | exact verdict agreement with blind read54 of 100 (54.0%) | 0.236 (4 verdicts) | $0.99 | Not usedCalled major on 29.5% of citations, far too often. |
| Panel: majority of gpt-oss-120b, Gemma 4, Nemotron 3OpenAI, Google, NVIDIA | Cloudflare Workers AI | v1 prompt; 2 of 3 agree, else split | panel | exact verdict agreement with blind read64 of 100 (64.0%) | 0.348 (4 verdicts) | $2.89 | Not usedMatched the blind read on the major rate (8.2% each on 85) but missed most minor errors and agreed weakly overall; replaced by the tuned checker. |
Rows T10-01 to T10-04 in the CSV.
21 Oct 2026Stage 3 overnight bake on batch 2 (single readers, settings later found wrong)Kept: nothing kept
Answer key. 200 random batch 2 citations, two blind AI readers (Claude, Muse) with a third Claude read on disagreements; AI-made; 13 majors, 13 cannot-tell
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Kimi K2.6Moonshot AI | Cloudflare Workers AI | v2 prompt; thinking off, temperatures off the makers' advice, scored by top verdict only (Clef by top choice) | candidate | exact verdict agreement129 of 200 (64.5%) | 0.576 (major vs not major) | not recorded | Not usedSuperseded: the 22 Oct audit found these settings wrong, so every model was re-run from its model card. |
| Qwen 3.8 27BAlibaba (Qwen) | Cloudflare Workers AI | v2 prompt; thinking off, temperatures off the makers' advice, scored by top verdict only (Clef by top choice) | candidate | exact verdict agreement140 of 200 (70.0%) | 0.512 (major vs not major) | not recorded | Not usedSuperseded: the 22 Oct audit found these settings wrong, so every model was re-run from its model card. |
| Gemma 4 26B A4BGoogle | Cloudflare Workers AI | v2 prompt; thinking off, temperatures off the makers' advice, scored by top verdict only (Clef by top choice) | candidate | exact verdict agreement147 of 200 (73.5%) | 0.481 (major vs not major) | not recorded | Not usedSuperseded: the 22 Oct audit found these settings wrong, so every model was re-run from its model card. |
| gpt-oss-120bOpenAI | Cloudflare Workers AI | v2 prompt; thinking off, temperatures off the makers' advice, scored by top verdict only (Clef by top choice) | candidate | exact verdict agreement134 of 200 (67.0%) | 0.433 (major vs not major) | not recorded | Not usedSuperseded: the 22 Oct audit found these settings wrong, so every model was re-run from its model card. |
| DeepSeek V4.1 FlashDeepSeek | Cloudflare Workers AI | v2 prompt; thinking off, temperatures off the makers' advice, scored by top verdict only (Clef by top choice) | candidate | exact verdict agreement148 of 200 (74.0%) | 0.422 (major vs not major) | not recorded | Not usedSuperseded: the 22 Oct audit found these settings wrong, so every model was re-run from its model card. |
| Llama 3.3 70B Instruct (fp8 fast)Meta | Cloudflare Workers AI | v2 prompt; thinking off, temperatures off the makers' advice, scored by top verdict only (Clef by top choice) | candidate | exact verdict agreement138 of 200 (69.0%) | 0.346 (major vs not major) | not recorded | Not usedSuperseded: the 22 Oct audit found these settings wrong, so every model was re-run from its model card. |
| ClefCloudflare | Cloudflare Workers AI | v2 prompt; thinking off, temperatures off the makers' advice, scored by top verdict only (Clef by top choice) | candidate | exact verdict agreement133 of 200 (66.5%) | 0.339 (major vs not major) | not recorded | Not usedSuperseded: the 22 Oct audit found these settings wrong, so every model was re-run from its model card. |
| Nemotron 3 Super 120B A12BNVIDIA | Cloudflare Workers AI | v2 prompt; thinking off, temperatures off the makers' advice, scored by top verdict only (Clef by top choice) | candidate | exact verdict agreement146 of 200 (73.0%) | 0.209 (major vs not major) | not recorded | Not usedSuperseded: the 22 Oct audit found these settings wrong, so every model was re-run from its model card. |
| Mistral Small 3.1 24BMistral AI | Cloudflare Workers AI | v2 prompt; thinking off, temperatures off the makers' advice, scored by top verdict only (Clef by top choice) | candidate | exact verdict agreement147 of 199 (73.9%) | 0.117 (major vs not major) | not recorded | Not usedSuperseded: the 22 Oct audit found these settings wrong, so every model was re-run from its model card. |
Rows T11-01 to T11-09 in the CSV.
22 Oct 2026Stage 3 overnight method tests 1-3 (cheap panels, fresh batch each)Kept: nothing kept
Answer key. 150-pair reference per test batch (b3, b4: Claude + Muse blind, adjudicated; b5: Claude only); AI-made; 5-13 majors each
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Method 1: plurality of 5 (DeepSeek V4.1 Flash, gpt-oss-120b x2 prompts, Kimi K2.6, Llama 3.3 70B)DeepSeek, OpenAI, Moonshot AI, Meta | Cloudflare Workers AI | chosen on batch 2 gold among about 6,000 panels and vote rules | panel | kappa_major on batch 30.493 | 0.493 major vs not; 0.296 error vs accurate | not recorded | Not usedFailed the fresh test (bars 0.6 and 0.5); choosing among thousands of panels on 200 items overfit. |
| Method 2: twostep vote of 5 (DeepSeek V4.1 Flash, Kimi K2.6, gpt-oss-120b, Llama 3.3 70B, Qwen 3.8)DeepSeek, Moonshot AI, OpenAI, Meta, Alibaba | Cloudflare Workers AI | ranked by worse of batch 2 and 3 scores | panel | recall of reference majors on batch 44 of 5 (80.0%) | 0.366 major vs not; 0.580 error vs accurate | not recorded | Not usedFound most majors but called about three times too many (precision 4/15). |
| Method 3: method 2 screens, Muse re-reads the flagged citationsas method 2 plus Meta | Cloudflare Workers AI; Meta (Muse app) | same v2 prompt as the reference reads; 27-34% re-read | panel | recall of reference majors on batch 54 of 10 (40.0%) | 0.471 major vs not; 0.498 error vs accurate | not recorded | Not usedFailed the fresh test; the small reference (10 majors) made kappa swing. |
Rows T12-01 to T12-03 in the CSV.
22 Oct 2026Clef as a screen, pre-registered fresh testKept: Clef
Answer key. 281 judgeable citations from batches 3 + 4 (adjudicated Claude + Muse gold), 16 majors; AI-made
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| ClefCloudflare | Cloudflare Workers AI | v2 definitions; flag when P(major) >= 0.10 (cut set on batch 2) | screen | recall of reference majors15 of 16 (93.8%) | – | $0.60 | ChosenPassed (catch >= 90%, flag <= 50%): caught 15/16 and flagged 46%; AUC 0.925. |
Rows T13-01 to T13-01 in the CSV.
22 Oct 2026Stage 3 rerun: 18 arms set from model cards (screen test, leave one batch out)Kept: Clef, DeepSeek V4.1 Flash, Qwen 3.8 27B, GLM 5.2
Answer key. 613 judgeable reference citations, batches 2-5 (b2-b4 adjudicated Claude + Muse gold, b5 Claude only); AI-made; 39 majors
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| JevTypeSafe | Cloudflare AI Gateway (route for 414 items unverified, see amendment 8) | arm jev: typed choice, 4 option orders averaged; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)36 of 39 (92.3%) | 0.508 major vs not (decide alone); 0.445 error vs accurate | not recorded | BackupPassed the screen bar with the lowest flag rate; screen of the secondary system. |
| ClefCloudflare | Cloudflare Workers AI | arm clef: typed choice, 4 option orders averaged; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)35 of 39 (89.7%) | 0.457 major vs not (decide alone); 0.341 error vs accurate | not recorded | ChosenMissed the 90% bar by one major here, but no meaningful gap to Jev (amendment 2), so Clef screens the locked system. |
| Qwen 3.8 27BAlibaba (Qwen) | Cloudflare Workers AI | arm qwen_off: thinking off, temp 0.7, top_p 0.8, presence 1.5, logprobs; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)36 of 39 (92.3%) | 0.447 major vs not (decide alone); 0.374 error vs accurate | not recorded | Not usedPassed as a screen but was not picked by the system rule. |
| gpt-oss-120bOpenAI | Cloudflare Workers AI | arm gptoss_med: reasoning medium, temp 1.0, 3 calls; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)36 of 39 (92.3%) | 0.343 major vs not (decide alone); 0.413 error vs accurate | not recorded | Not usedPassed as a screen but flagged 45%. |
| Kimi K2.6Moonshot AI | Cloudflare Workers AI | arm kimi_off: thinking off, temp 0.6, top_p 0.95, logprobs; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)35 of 39 (89.7%) | 0.468 major vs not (decide alone); 0.361 error vs accurate | not recorded | Not usedOne major short of the screen bar. |
| DeepSeek V4.1 FlashDeepSeek | Cloudflare Workers AI | arm deepseek_off: thinking off, temp 1.0, logprobs; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)35 of 39 (89.7%) | 0.424 major vs not (decide alone); 0.316 error vs accurate | not recorded | Not usedShort of the screen bar. |
| GLM 5.2Zhipu AI (Z.ai) | Cloudflare Workers AI | arm glm_off: thinking off via enable_thinking false, temp 1.0, logprobs; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)35 of 39 (89.7%) | 0.54 major vs not (decide alone); 0.394 error vs accurate | not recorded | Not usedOne major short of the screen bar. |
| Gemma 4 26B A4BGoogle | Cloudflare Workers AI | arm gemma_off: thinking off, temp 1.0, logprobs; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)35 of 39 (89.7%) | 0.519 major vs not (decide alone); 0.333 error vs accurate | not recorded | Not usedShort of the screen bar. |
| Nemotron 3 Super 120B A12BNVIDIA | Cloudflare Workers AI | arm nemotron_off: thinking off, temp 1.0, logprobs; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)36 of 39 (92.3%) | 0.104 major vs not (decide alone); 0.12 error vs accurate | not recorded | Not usedFlagged 69%; the weakest ranker. |
| DeepSeek V4.1 FlashDeepSeek | Cloudflare Workers AI | arm deepseek_on: thinking high, temp 1.0, n=3 vote; v2 prompt; json_schema output | decider | held-out majors caught at the 90% threshold (screen bar)39 of 39 (100.0%) | 0.678 major vs not (decide alone); 0.581 error vs accurate | not recorded | ChosenNot a screen (vote scores tie at 0), but a strong decider: one of the three voters in the locked system. |
| Qwen 3.8 27BAlibaba (Qwen) | Cloudflare Workers AI | arm qwen_on: thinking medium, temp 1.0, n=3 vote; v2 prompt; json_schema output | decider | held-out majors caught at the 90% threshold (screen bar)39 of 39 (100.0%) | 0.712 major vs not (decide alone); 0.455 error vs accurate | not recorded | ChosenBest single decider (kappa_major 0.712); voter in the locked system. |
| GLM 5.2Zhipu AI (Z.ai) | Cloudflare Workers AI | arm glm_on: thinking high, temp 1.0, n=3 vote; v2 prompt; json_schema output | decider | held-out majors caught at the 90% threshold (screen bar)35 of 39 (89.7%) | 0.581 major vs not (decide alone); 0.635 error vs accurate | not recorded | ChosenBest on any error (kappa 0.635); voter in the locked system. |
| Kimi K2.6Moonshot AI | Cloudflare Workers AI | arm kimi_on: thinking high, temp 1.0, 3 separate calls; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)22 of 24 (91.7%) | 0.708 major vs not (decide alone); 0.639 error vs accurate | not recorded | Not usedIncomplete at this look (332 of 613 items) because of Cloudflare timeouts; strong where scored. |
| Gemma 4 26B A4BGoogle | Cloudflare Workers AI | arm gemma_on: thinking on, temp 1.0, 3 separate calls; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)39 of 39 (100.0%) | 0.426 major vs not (decide alone); 0.56 error vs accurate | not recorded | Not usedWeak as a decider (kappa_major 0.426). |
| Nemotron 3 Super 120B A12BNVIDIA | Cloudflare Workers AI | arm nemotron_on: thinking on, low effort, n=3; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)39 of 39 (100.0%) | 0.34 major vs not (decide alone); 0.291 error vs accurate | not recorded | Not usedWeak as a decider. |
| gpt-oss-120bOpenAI | Cloudflare Workers AI | arm gptoss_high: reasoning high, temp 1.0, 3 calls; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)39 of 39 (100.0%) | 0.341 major vs not (decide alone); 0.463 error vs accurate | not recorded | Not usedWeak as a decider; high effort no better than medium. |
| Llama 3.3 70B Instruct (fp8 fast)Meta | Cloudflare Workers AI | arm llama: temp 0.6, top_p 0.9, 3 calls; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)39 of 39 (100.0%) | 0.221 major vs not (decide alone); 0.234 error vs accurate | not recorded | Not usedPoor ranker (AUC 0.63); said cannot tell often. |
| Mistral Small 3.1 24BMistral AI | Cloudflare Workers AI | arm mistral: temp 0.15, 3 calls; v2 prompt; json_schema output | screen | held-out majors caught at the 90% threshold (screen bar)39 of 39 (100.0%) | 0.095 major vs not (decide alone); 0.283 error vs accurate | not recorded | Not usedPoorest ranker (AUC 0.57). |
Rows T14-01 to T14-18 in the CSV.
22 Oct 2026Backup reference reader calibration (citation accuracy)Kept: nothing kept
Answer key. 100 random batch 2-4 gold items (seed 66); AI-made; about 10 majors, so weak evidence
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Gemini 3.8 FlashGoogle | Google Gemini API (7 reads, free tier) and OpenRouter (93 reads) | v2 rules and prompt, default thinking | backup reader | exact verdict agreement89 of 100 (89.0%) | 0.887 major vs not | $3.50 | BackupBest reader calibration we have (9 of its 10 major calls right); used for second checks and adjudication stops in batch 6. |
| Grok 4.7xAI | OpenRouter | v2 rules and prompt | backup reader | exact verdict agreement65 of 100 (65.0%) | 0.50 major vs not | no per-call charge | BackupWeak calibration (8 of 19 major calls right) but a different family; kept as the registered reader for the 66 Claude-stop items. |
Rows T15-01 to T15-02 in the CSV.
22 Oct 2026Agreement between the two blind reference readersKept: Muse Spark (Meta; version not recorded), Claude (model not recorded)
Answer key. No answer key: Claude (reader A, model not recorded) vs Muse Spark (reader B), same v2 rules; batch 6 n=915 judgeable
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Muse Spark (Meta; version not recorded)Meta | Meta (Muse app) | v2 rules and prompt, effort medium | reference reader | kappa_major with reader A0.658 | 0.658 major vs not (batch 6); 0.74-0.85 on batches 2-4 | no per-call charge | ChosenA second model family for the answer key; the readers' own agreement sets the ceiling a checker can reach. |
| Claude (model not recorded)Anthropic | Anthropic (Claude app) | v2 rules; 11-item packets | reference reader | items covered930 of 996 (93.4%) | – | no per-call charge | ChosenReader A and adjudicator for batches 2-6; stopped items went to a model from a different family, as our rules require. |
Rows T16-01 to T16-02 in the CSV.
22 Oct 2026Batch 6 pre-registered confirmatory test of the checkerKept: Checker (locked): Clef screen -> majority of DeepSeek V4.1 Flash, Qwen 3.8 27B, GLM 5.2 (thinking on), Checker (locked), tuning-set score before batch 6
Answer key. 996 fresh random citations (915 judgeable), two blind readers from different families (Claude 930 + Grok 4.7 66; Muse 996), 169 disagreements adjudicated; AI-made; 66 majors
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Checker (locked): Clef screen -> majority of DeepSeek V4.1 Flash, Qwen 3.8 27B, GLM 5.2 (thinking on)Cloudflare, DeepSeek, Alibaba, Zhipu AI | Cloudflare Workers AI | Clef flags at P(major) >= 0.1466; deciders as T14 *_on arms; unflagged = not major | screen + deciders | recall of reference majors56 of 66 (84.8%) | 0.741 major vs not | $11–$13 | ChosenPassed all three pre-registered bars (kappa 0.60, recall 0.70, precision 0.65). |
| Checker (secondary): Jev screen -> same three decidersTypeSafe, DeepSeek, Alibaba, Zhipu AI | Cloudflare AI Gateway; Cloudflare Workers AI | Jev flags at P(major) >= 0.25 | screen + deciders | recall of reference majors55 of 66 (83.3%) | 0.715 major vs not | not recorded | BackupAlso passed; kept as the secondary system. |
| Checker (locked), tuning-set score before batch 6Cloudflare, DeepSeek, Alibaba, Zhipu AI | Cloudflare Workers AI | as T17-01; scored on batches 2-5 with leave-one-batch-out thresholds | screen + deciders | recall of reference majors (batches 2-5)28 of 39 (71.8%) | 0.709 major vs not | not recorded | ChosenHighest kappa_major among Clef-screen systems flagging <= 50% (pre-registered choice rule). |
Rows T17-01 to T17-03 in the CSV.
30 Oct 2026Tuning with Cloudflare off: single readers on the tuning setKept: Claude Haiku 5.5, Composer 2.5
Answer key. 610-613 judgeable reference citations, batches 2-5; AI-made; 39 majors (37 for Haiku max after errors)
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort medium, v2 prompt, 1 per request | candidate | recall of reference majors20 of 39 (51.3%) | 0.458 major vs not | $0.28 | Not usedFailed every bar as a single reader at medium effort. |
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort high, v2 prompt | candidate | recall of reference majors27 of 39 (69.2%) | 0.671 major vs not | $0.35 | Not usedMissed the tuning recall bar by one major. |
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort high, v3 checklist prompt | candidate | recall of reference majors25 of 39 (64.1%) | 0.655 major vs not | $0.36 | Not usedWorse than the v2 prompt, so v2 stays. |
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort xhigh, v2 prompt (run 1) | decider | recall of reference majors28 of 39 (71.8%) | 0.73 major vs not | $0.44 | ChosenPassed both sets; xhigh is the sweet spot; decides in the frozen setup. |
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort xhigh, v2 prompt (run 2) | decider | recall of reference majors25 of 39 (64.1%) | 0.686 major vs not | $0.44 | ChosenRepeat run: lower tuning score, so one read is too noisy; three reads are used. |
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort xhigh, v2 prompt (run 3) | decider | recall of reference majors27 of 39 (69.2%) | 0.671 major vs not | $0.44 | ChosenRepeat run: lower tuning score, so one read is too noisy; three reads are used. |
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort max, v2 prompt, max_tokens 32000 | candidate | recall of reference majors28 of 37 (75.7%) | 0.764 major vs not | $2.24 | Not usedSame kappa as xhigh at five times the cost. |
| Composer 2.5Cursor (included usage) | Cursor (included usage) | v2 prompt, 1 citation per call | candidate | recall of reference majors32 of 39 (82.1%) | 0.624 major vs not | included usage | Not usedHigh recall but fails precision; called some error on 43% of batch 6. |
| Composer 2.5Cursor (included usage) | Cursor (included usage) | v2 prompt, 10 citations per call | screen | recall of reference majors35 of 39 (89.7%) | 0.624 major vs not | included usage | ChosenSame tuning kappa as 1 per call at about one fifth of the tokens; screens the frozen setup. |
| Grok 4.5xAI | Cursor (included usage) | effort medium, v2 prompt, 1 per call | candidate | recall of reference majors33 of 39 (84.6%) | 0.662 major vs not | included usage | Not usedVery high recall but over-calls; adds nothing in any system. |
| Grok 4.6xAI | Cursor (included usage) | effort medium, v2 prompt, 1 per call | candidate | recall of reference majors38 of 39 (97.4%) | 0.793 major vs not | included usage | Not usedBest single tuning score but fails batch 6 precision; about 27,000 tokens per citation. |
| Grok 4.6xAI | Cursor (included usage) | effort medium, v2 prompt, 10 per call | backup reader | recall of reference majors37 of 39 (94.9%) | 0.71 major vs not | included usage | BackupNamed backup screen if Composer is unavailable; packing cost it precision. |
| Grok 4.7xAI | Cursor (included usage) | effort low, v2 prompt, 1 per call | backup reader | recall of reference majors36 of 39 (92.3%) | 0.505 major vs not | included usage | BackupMost sensitive reader but too loose to decide; stays the fallback reader when Claude stops. |
| Grok 4.7xAI | Cursor (included usage) | effort medium, v2 prompt, 1 per call | candidate | recall of reference majors35 of 39 (89.7%) | 0.458 major vs not | included usage | Not usedOver-calls like 4.7 low. |
| Grok 4.7xAI | Cursor (included usage) | effort high, v2 prompt, 1 per call | candidate | recall of reference majors36 of 39 (92.3%) | 0.501 major vs not | included usage | Not usedOver-calls at every effort. |
Rows T18-01 to T18-15 in the CSV.
30 Oct 2026Tuning with Cloudflare off: single readers on batch 6 (exploratory, batch 6 already used)Kept: Claude Haiku 5.5, Composer 2.5
Answer key. Batch 6 reference, 915 judgeable (907 for Haiku max); AI-made; 66 majors
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort medium, v2 prompt, 1 per request | candidate | recall of reference majors45 of 66 (68.2%) | 0.598 major vs not | $0.28 | Not usedFailed every bar as a single reader at medium effort. |
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort high, v2 prompt | candidate | recall of reference majors52 of 66 (78.8%) | 0.74 major vs not | $0.35 | Not usedMissed the tuning recall bar by one major. |
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort high, v3 checklist prompt | candidate | recall of reference majors52 of 66 (78.8%) | 0.71 major vs not | $0.36 | Not usedWorse than the v2 prompt, so v2 stays. |
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort xhigh, v2 prompt (run 1) | decider | recall of reference majors53 of 66 (80.3%) | 0.762 major vs not | $0.44 | ChosenPassed both sets; xhigh is the sweet spot; decides in the frozen setup. |
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort xhigh, v2 prompt (run 2) | decider | recall of reference majors53 of 66 (80.3%) | 0.775 major vs not | $0.44 | ChosenRepeat run: lower tuning score, so one read is too noisy; three reads are used. |
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort xhigh, v2 prompt (run 3) | decider | recall of reference majors54 of 66 (81.8%) | 0.771 major vs not | $0.44 | ChosenRepeat run: lower tuning score, so one read is too noisy; three reads are used. |
| Claude Haiku 5.5Anthropic | Anthropic Message Batches API | effort max, v2 prompt, max_tokens 32000 | candidate | recall of reference majors55 of 65 (84.6%) | 0.762 major vs not | $2.24 | Not usedSame kappa as xhigh at five times the cost. |
| Composer 2.5Cursor (included usage) | Cursor (included usage) | v2 prompt, 1 citation per call | candidate | recall of reference majors58 of 66 (87.9%) | 0.665 major vs not | included usage | Not usedHigh recall but fails precision; called some error on 43% of batch 6. |
| Composer 2.5Cursor (included usage) | Cursor (included usage) | v2 prompt, 10 citations per call | screen | recall of reference majors61 of 66 (92.4%) | 0.583 major vs not | included usage | ChosenSame tuning kappa as 1 per call at about one fifth of the tokens; screens the frozen setup. |
| Grok 4.5xAI | Cursor (included usage) | effort medium, v2 prompt, 1 per call | candidate | recall of reference majors65 of 66 (98.5%) | 0.668 major vs not | included usage | Not usedVery high recall but over-calls; adds nothing in any system. |
| Grok 4.6xAI | Cursor (included usage) | effort medium, v2 prompt, 1 per call | candidate | recall of reference majors61 of 66 (92.4%) | 0.695 major vs not | included usage | Not usedBest single tuning score but fails batch 6 precision; about 27,000 tokens per citation. |
| Grok 4.6xAI | Cursor (included usage) | effort medium, v2 prompt, 10 per call | backup reader | recall of reference majors60 of 66 (90.9%) | 0.633 major vs not | included usage | BackupNamed backup screen if Composer is unavailable; packing cost it precision. |
| Grok 4.7xAI | Cursor (included usage) | effort low, v2 prompt, 1 per call | backup reader | recall of reference majors64 of 66 (97.0%) | 0.508 major vs not | included usage | BackupMost sensitive reader but too loose to decide; stays the fallback reader when Claude stops. |
| Grok 4.7xAI | Cursor (included usage) | effort medium, v2 prompt, 1 per call | candidate | recall of reference majors64 of 66 (97.0%) | 0.511 major vs not | included usage | Not usedOver-calls like 4.7 low. |
| Grok 4.7xAI | Cursor (included usage) | effort high, v2 prompt, 1 per call | candidate | recall of reference majors65 of 66 (98.5%) | 0.48 major vs not | included usage | Not usedOver-calls at every effort. |
Rows T19-01 to T19-15 in the CSV.
30 Oct 2026Tuning with Cloudflare off: screen -> decider systemsKept: Frozen setup: Composer 2.5 (10 per call) flags 'major' -> Claude Haiku 5.5 xhigh x3, major if any read says major
Answer key. Tuning set batches 2-5 (39 majors) and batch 6 (66 majors, exploratory); AI-made
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Frozen setup: Composer 2.5 (10 per call) flags 'major' -> Claude Haiku 5.5 xhigh x3, major if any read says majorCursor, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batches 2-5 | screen + decider | recall of reference majors (batches 2-5)28 of 39 (71.8%) | 0.73 major vs not | $0.15–$0.19 | ChosenSimplest and cheapest setup that passed both sets with two families and held across repeat runs. |
| Frozen setup: Composer 2.5 (10 per call) flags 'major' -> Claude Haiku 5.5 xhigh x3, major if any read says majorCursor, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batch 6 | screen + decider | recall of reference majors (batch 6)58 of 66 (87.9%) | 0.795 major vs not | $0.15–$0.19 | ChosenSimplest and cheapest setup that passed both sets with two families and held across repeat runs. |
| Composer 2.5 k10 [major or minor] -> Claude Haiku 5.5 xhigh x1Cursor, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batches 2-5 | screen + decider | recall of reference majors (batches 2-5)28 of 39 (71.8%) | 0.752 major vs not | not recorded | Not usedPassed, but one Haiku read is noisy across repeat runs. |
| Composer 2.5 k10 [major or minor] -> Claude Haiku 5.5 xhigh x1Cursor, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batch 6 | screen + decider | recall of reference majors (batch 6)52 of 66 (78.8%) | 0.771 major vs not | not recorded | Not usedPassed, but one Haiku read is noisy across repeat runs. |
| Composer 2.5 k10 [major] -> Claude Haiku 5.5 xhigh x1Cursor, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batches 2-5 | screen + decider | recall of reference majors (batches 2-5)27 of 39 (69.2%) | 0.758 major vs not | not recorded | Not usedMissed the tuning recall bar by one major. |
| Composer 2.5 k10 [major] -> Claude Haiku 5.5 xhigh x1Cursor, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batch 6 | screen + decider | recall of reference majors (batch 6)52 of 66 (78.8%) | 0.778 major vs not | not recorded | Not usedMissed the tuning recall bar by one major. |
| Composer 2.5 k10 [major] -> Claude Haiku 5.5 xhigh or highCursor, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batches 2-5 | screen + decider | recall of reference majors (batches 2-5)28 of 39 (71.8%) | 0.741 major vs not | not recorded | Not usedPassed; replaced by three xhigh reads, which are steadier. |
| Composer 2.5 k10 [major] -> Claude Haiku 5.5 xhigh or highCursor, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batch 6 | screen + decider | recall of reference majors (batch 6)54 of 66 (81.8%) | 0.791 major vs not | not recorded | Not usedPassed; replaced by three xhigh reads, which are steadier. |
| Composer 2.5 [major] -> Claude Haiku 5.5 xhigh or high-v3Cursor, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batches 2-5 | screen + decider | recall of reference majors (batches 2-5)28 of 39 (71.8%) | 0.776 major vs not | not recorded | Not usedTop tuning kappa but 1 call per citation costs about five times the screen tokens. |
| Composer 2.5 [major] -> Claude Haiku 5.5 xhigh or high-v3Cursor, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batch 6 | screen + decider | recall of reference majors (batch 6)53 of 66 (80.3%) | 0.775 major vs not | not recorded | Not usedTop tuning kappa but 1 call per citation costs about five times the screen tokens. |
| Grok 4.7 low screen -> Claude Haiku 5.5 xhigh x1xAI, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batches 2-5 | screen + decider | recall of reference majors (batches 2-5)27 of 39 (69.2%) | 0.746 major vs not | not recorded | Not usedMissed the tuning recall bar by one major. |
| Grok 4.7 low screen -> Claude Haiku 5.5 xhigh x1xAI, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batch 6 | screen + decider | recall of reference majors (batch 6)52 of 66 (78.8%) | 0.778 major vs not | not recorded | Not usedMissed the tuning recall bar by one major. |
| Grok 4.6 medium (1 per call) screen -> Claude Haiku 5.5 xhigh x3 anyxAI, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batches 2-5 | screen + decider | recall of reference majors (batches 2-5)30 of 39 (76.9%) | 0.823 major vs not | not recorded | Not usedBest tuning score, but about seven times the screen tokens; did not survive packing. |
| Grok 4.6 medium (1 per call) screen -> Claude Haiku 5.5 xhigh x3 anyxAI, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batch 6 | screen + decider | recall of reference majors (batch 6)56 of 66 (84.8%) | 0.79 major vs not | not recorded | Not usedBest tuning score, but about seven times the screen tokens; did not survive packing. |
| Grok 4.6 medium (10 per call) screen -> Claude Haiku 5.5 xhigh x3 anyxAI, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batches 2-5 | screen + decider | recall of reference majors (batches 2-5)29 of 39 (74.4%) | 0.737 major vs not | not recorded | BackupTies the frozen setup within one major on each set; named backup screen. |
| Grok 4.6 medium (10 per call) screen -> Claude Haiku 5.5 xhigh x3 anyxAI, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batch 6 | screen + decider | recall of reference majors (batch 6)57 of 66 (86.4%) | 0.799 major vs not | not recorded | BackupTies the frozen setup within one major on each set; named backup screen. |
| Grok 4.7 high screen -> Claude Haiku 5.5 xhigh x3 anyxAI, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batches 2-5 | screen + decider | recall of reference majors (batches 2-5)29 of 39 (74.4%) | 0.737 major vs not | not recorded | Not usedNo gain over the Composer screen and a higher flag rate. |
| Grok 4.7 high screen -> Claude Haiku 5.5 xhigh x3 anyxAI, Anthropic | Cursor (included usage); Anthropic Message Batches API | v2 prompt; scored on batch 6 | screen + decider | recall of reference majors (batch 6)58 of 66 (87.9%) | 0.777 major vs not | not recorded | Not usedNo gain over the Composer screen and a higher flag rate. |
Rows T20-01a to T20-09b in the CSV.
31 Oct 2026Claude Opus 5.5 effort calibration for the batch 7 answer key (citation accuracy)Kept: Claude Opus 5.5, Claude (batch 6 reader A, model not recorded)
Answer key. 164 batch 6 gold items, stratified: all 54 remaining majors plus 110 random non-majors (seed 7301); AI-made; gold partly built from Claude reads, so agreement is inflated for any Claude reader; majors over-sampled, so kappa is for comparing efforts only
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Claude Opus 5.5Anthropic | Anthropic Message Batches API | effort medium, v2 rules (judge_v2.SYSTEM), max_tokens 8000, 1 per request | reader | exact verdict agreement141 of 161 (87.6%) | 0.823 major vs not | $10.43 | ChosenLowest effort within 0.03 kappa and 2 points of the best (pre-set rule); reader 1 for batch 7. |
| Claude Opus 5.5Anthropic | Anthropic Message Batches API | effort high, v2 rules (judge_v2.SYSTEM), max_tokens 8000, 1 per request | reader | exact verdict agreement144 of 161 (89.4%) | 0.837 major vs not | $11.04 | ChosenBest scores; used for adjudication in batch 7. |
| Claude Opus 5.5Anthropic | Anthropic Message Batches API | effort xhigh, v2 rules (judge_v2.SYSTEM), max_tokens 8000, 1 per request | reader | exact verdict agreement143 of 161 (88.8%) | 0.807 major vs not | $9.82 | Not usedExactly 0.03 below the best kappa; more output tokens; its batch re-sent 3 refused items by mistake (excluded). |
| Claude (batch 6 reader A, model not recorded)Anthropic | Anthropic (Claude app) | v2 rules, 11-item packets | reference reader | exact verdict agreement160 of 164 (97.6%) | 0.944 major vs not (inflated: gold was built from it) | not recorded | ChosenShows the inflation any Claude reader gets on this gold. |
Rows T22-01 to T22-04 in the CSV.
31 Oct 2026Batch 7 answer key: registered design only (results sealed until scored)Kept: Claude Opus 5.5, Muse Spark (Meta; version not recorded), Gemini 3.8 Flash, Grok 4.7, gpt-oss-120b
Answer key. Batch 7 (1,320 fresh random pairs for citation accuracy; 440 targeted census pairs, internal); not yet read or scored
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Claude Opus 5.5Anthropic | Anthropic Message Batches API | effort medium for first reads, high for adjudication | reader 1 and adjudicator (Part A); reader 1 (Part B) | planned (no result)not recorded | – | not recorded | ChosenEffort fixed by T22; Part B check in T21. |
| Muse Spark (Meta; version not recorded)Meta | Meta (Muse app) | v2 prompt unchanged; census codebook for Part B | reader 2 (both parts) | planned (no result)not recorded | – | not recorded | ChosenReader B for batches 2-6; passed the census backup test (test T21). |
| Gemini 3.8 FlashGoogle | Google Gemini API, then OpenRouter (paid, only with approval) | default thinking, as calibrated | fallback for Claude stops and third blind read (Part A) | planned (no result)not recorded | – | not recorded | ChosenBest reader calibration (test T15); moved from candidate system to reference reader so it is never scored on its own reads. |
| Grok 4.7xAI | xAI (Grok app) | v2 prompt | second fallback and adjudicator for stopped items (Part A) | planned (no result)not recorded | – | not recorded | ChosenDifferent family, not a candidate system; weak calibration (test T15), so rates are also reported without its items. |
| gpt-oss-120bOpenAI | Google Antigravity | reasoning medium | fallback reader and adjudicator (Part B) | planned (no result)not recorded | – | not recorded | ChosenPassed the census check at 91.3% (test T21); not a census panel member. |
| Frozen setup: Composer 2.5 k10 -> Claude Haiku 5.5 xhigh x3Cursor, Anthropic | Cursor (included usage); Anthropic Message Batches API | as T20-01 | candidate system to be scored | planned (no result)not recorded | – | not recorded | CandidateThe checker chosen on 30 Oct with Cloudflare off (test T20). |
Rows T24-01 to T24-06 in the CSV.
Lists reading
Does each citation counted on the Lists hold up when read against both papers?
31 Oct 2026Lists wave 1 reader calibration (full-read verdicts)Kept: nothing kept
Answer key. 100 citations from the 23 Oct Lists full reads, stratified (60 HOLDS, 23 NOT_IN_PAPER, 15 PASSING, 2 FLAGGED), keyed by two blind Claude full reads with a blind third read on disagreements; AI-made
| Model | Host | Setting | Role | Score | Kappa | Per 1,000 citations | Outcome |
|---|---|---|---|---|---|---|---|
| Claude Opus 5.5Anthropic | Anthropic Message Batches API | effort medium, v1 items (no retracted-paper text), max_tokens 8000 | reader | exact verdict agreement89 of 100 (89.0%) | – | $11.06 | Not usedPassed the 85% bar but read 8 NOT_IN_PAPER items as HOLDS, which would inflate List counts; items were rebuilt with retracted-paper text. |
| Claude Opus 5.5Anthropic | Anthropic Message Batches API | effort medium, v2 items with retracted-paper excerpts and one added rule line | reader | exact verdict agreement91 of 100 (91.0%) | – | $13.12 | CandidatePasses the bar; the gap stays on items with no open retracted text (65-71% of wave 1). |
| Composer 2.5Cursor (included usage) | Cursor (included usage) | 5 citations per call, v1 items | reader | exact verdict agreement80 of 100 (80.0%) | – | included usage | Not usedBelow the 85% bar; read 10 HOLDS items as NOT_IN_PAPER, the opposite of Opus. |
| Composer 2.5Cursor (included usage) | Cursor (included usage) | 5 citations per call, v2 items with retracted-paper excerpts | reader | exact verdict agreement77 of 100 (77.0%) | – | included usage | Not usedBelow the 85% bar again (scored by this audit with the pre-written scorer); not used for wave 1. |
Rows T23-01 to T23-04 in the CSV.
What we learned
What failed, and what surprised us.
- Some papers stop the readerClaude sometimes stops partway through reading papers on medicine, drugs or disease. In our main test it stopped on 66 of 996 citations (6.6%). A model from a different family read those, and a stop is final: we never reword or re-send.
- Free tiers run out fastThe free Gemini tier gave 7 reads in 13 minutes before it began refusing calls, so the same model, with the same prompt, finished on a paid route. We record which route answered each item.
- Our checker's host went darkOur Cloudflare credit ran out, so Cloudflare AI is off. The checker that passed our main test runs there, so we tuned a new setup on what we can run: Composer 2.5 through Cursor as the screen and Claude Haiku 5.5 on the Anthropic API as the decider. It is not yet confirmed on a fresh answer key.
- Settings matter as much as the modelOur first overnight runs used the wrong settings for most models, so we re-ran every model from its maker's own settings. Some switches do not do what they say: GLM 5.2 on Cloudflare ignored the usual 'thinking off' setting and still wrote 1,339 tokens of reasoning on one item. Effort level moved Claude Haiku 5.5 most of all: it caught 20 of 39 serious errors on our tuning set at medium effort and 28 at extra-high. Max effort cost about 5 times as much as extra-high for about the same agreement.
- Reading ten at a timeGiving Composer 2.5 10 citations per call cut its tokens per citation from about 24,600 to about 4,300, with the same agreement on our tuning set (kappa 0.624). The same trick cost Grok 4.6 precision.
- Cheap models over-callModels disagree widely on what counts as a serious error. On the same 1,600 citations in our first accuracy pilot, single models called between 3.9% and 29.5% of them serious. Grok 4.7 at low effort caught 64 of 66 serious errors in our main test set, but only 64 of its 163 serious calls were right.
- Errors cluster where models splitIn the census, where all three models agreed, the panel matched the answer key on 298 of 313 passages. Where they split, it matched on only 40 of 54. Those passages are hard for any reader.
- One run is not enoughThree identical runs of Claude Haiku 5.5 at extra-high effort gave kappa 0.73, 0.686 and 0.671 on our tuning set. One read is too noisy to trust alone, so our setup reads each flagged citation three times.
Not tested yet
- A check of any of our answer keys by people: designed, not yet staffed.
- The new setup on a fresh answer key (batch 7): registered, results sealed until scored.
- The Sarol et al. human-labelled citation benchmark: registered, not yet run; the checker it names runs on Cloudflare, which is off.
- Each part of the batch 6 checker scored alone.
- NVIDIA Nemotron 3 Ultra and Thinking Machines Inkling as readers, and DeepSeek V4 Pro as a decider.
- Llama 3.3 70B with the settings that worked on an older, different task (temperature zero and a quote check).
Help us improve
Have a better idea? Tell us.
If you know a model, a setting or a scoring method that would do better, or you find a mistake in these numbers, open an issue on the census repository or write to us. We test good suggestions the same way and publish the result.