Reviewer grades made on different points or rules (T-667)

5 Oct 2026. Tool: tools/panelstale.js; output: tools/panelstale.json; test: tests/unit/panelstale.test.js.

In plain English

Each reviewer graded the game as it stood on the day they made their list. Our ledger reads the game as it stood on 30 Sep (BSData 374f505, the frozen commit). If a unit's points or datasheet changed in between, the reviewer and the ledger are grading two different units, and a "miss" on that unit isn't really the ledger's fault. So I looked up, for every panel, the BSData commit in force on the panel's date, and compared every graded unit with the frozen commit.

• Most panels are fine. Every panel from 12 Sep onwards matches the frozen data exactly (no unit's points or datasheet moved), and the August and 11 Sep ones differ by one to three units each. The stale ones are the early 11th-edition lists (late June to early August) and the four 10th-edition lists (Maelstrom, Astrategas, both Votann lists), where every unit is flagged "edition".
• For Tyranids, dropping the stale units does not close the gap. Today the ledger orders 77.7% of the reviewers' pairs the right way, while the reviewers agree with each other 90.4% of the time: a gap of 12.7 points. Taking out the units whose points or rules changed shrinks that gap by 0.3 to 1.3 points depending on how strict the flag is, and taking out the same number of units at random shrinks it by 0.1 to 0.7 on its own. What the stale units explain is about one point at most, within the noise. The ledger's misses are on units graded on today's game.
• The 10th-edition lists aren't the problem either. Maelstrom (10th edition) agrees with the other four Tyranid lists more than any of them (95.8%). Keeping only the three 11th-edition lists widens the gap (13.2 points), because those three agree with each other even more (92.7%) than the ledger agrees with them (79.6%).
• How many pairs it costs: 398 to 637 of the 4,457 pairs (9 to 14%) when the panels are read at their dates; 135 to 403 (3 to 9%) when the two early lists are read at the commit their own files name (80568cc, 2 Jul).
• Two small hygiene points for the yardstick. (1) The panels' patchDrop checked the smallest size only: Von Ryan's Leapers moved 8% at 3 models but 12.5% at 6. (2) Into the Hive Mind's patchDrop was read at the 2 Jul stand-in commit, but the video is dated 27 Jul, after the July balance update (BSData merged it on 25 Jul): at its date the Tyrannocyte's and Toxicrene's points already stood, so they needn't be dropped. Putting both back (and Pyrovores in HivemindHobbies, unchanged since 20 Jun) moves nothing that matters (gap 12.3).

Recommendation: keep the yardstick as it is. Staleness is not where the ledger's 12-point gap comes from; the next gains have to come from the model.

What was compared

• The commit at a panel's date: the last commit on BSData main's first-parent history before the start of that day (UTC). Panels without a usable date use an override in the tool (e.g. Tista Minis 22 Jul from its source line; the three Tyranid summaries take their full transcription's date). A panel dated before 1 Jun 2026, or whose edition says 10th, is 10th edition.
• Points: every squad size both commits have. A move of 10% or more at any size flags the unit ("points"); smaller moves are kept in the JSON as minor.
• Datasheet: tools/patches.js's fields() (stats, weapons by name and field, size, keywords, abilities by name): names of what changed, never the values. Before comparing, BSData's own bookkeeping is evened out (it is not a change to the game): "-" against an empty field, keyword lists' order and case, the hidden Leader and Support categories (added 22 Jun), ability names' markdown and trailing colons, the invulnerable save moved from an ability to its own profile field (from 15 Jun), the Leader and Support abilities refiled as categories. Without this, 37 of the 52 units in Into the Hive Mind looked changed; with it, 5.
• Severity, one per unit: points (10%+) over rules (a stat, weapon, size or ability changed) over keywords (keywords alone: mostly BSData's tagging catching up, e.g. the FRAME keyword or a spelling fix in a unit's own keyword) over edition. "Absent": not in that commit's data under that name (Ancient in Terminator Armour, three Space Marine panels).
• 10th-edition panels: every unit "edition". in11e says whether the unit exists in the frozen 11th-edition data: all do, because the panels' names were matched to 11th-edition datasheets when they were transcribed (the names that weren't are in each panel's own unmatched and aren't graded units).

Every panel

All 31 panels (the retired Orks panel left out): 1,235 graded units; 84 flagged points, 46 rules, 33 keywords alone, 126 edition, 3 absent. The superseded HivemindHobbies and Into the Hive Mind summaries read as their transcriptions do.

The early 11th-edition panels carry the most: Dark Angels (30 of 93 on points) and HivemindHobbies (15 of 52) were graded before BSData entered the launch points (BSData's "Tyranids MFM Updates" landed 24 Jun, three days after the HivemindHobbies video). BSData lags the official releases by days, so a panel dated on or just after a release may have been graded on points BSData didn't hold yet: that is why the Tyranid numbers below are given both ways.

The Tyranid yardstick with and without the stale units

Track B (tools/ledger-benchmarks/2026-10-05-trackB.json) on the five headline Tyranid lists; pairwise on the S-to-D pairs as tools/ledgerstep.js scores it (77.7%, the yardstick's own number). The reviewers' agreement is tools/progress.js's ceiling: each list against the mean letter of the others, on the same units. Each list's patchDrop is already out of every row. "Random" is the same number of units dropped from each list at random, 500 times (mean ± SD), so the flagged drop can be read against what dropping that many units does by itself.

Panels read at their dates (HivemindHobbies at 20 Jun, Into the Hive Mind at 27 Jul):

The two early lists read at their files' own commit (80568cc, 2 Jul, the stand-in their patchDrop was read at):

Read across both tables: the gap moves by −1.4 to −0.3 points with the five lists; the random drops move it by −0.7 to −0.1. The best case (any change out) beats chance by about 0.7 points, one to one and a half standard deviations. With only the 11th-edition lists the gap gets wider, not narrower.

On the dropped pairs themselves. Read at 2 Jul, the ledger orders the pairs touching the stale units worse (72 to 77%) than the rest (about 80%), and the reviewers agree a little less there too (83 to 94%): the stale units are somewhat harder for both. Read at the panels' dates, the reverse: the 12 HivemindHobbies units whose points moved before the launch points are ones the ledger orders well (84% of their 398 pairs), so dropping them lowers its score. Either way they are too few pairs to move the mean.

The reviewers between themselves (each pair of lists, mean): 90.2% today; 90.0 to 90.7% with stale units out. The pairs that disagree most are Astrategas against the 11th-edition lists (75 to 85%), and Astrategas against Maelstrom, the other 10th-edition list, is 95%: the two 10th-edition reviewers agree with each other, and Maelstrom agrees with everyone, so the disagreement is Astrategas's view, not the edition.

Units flagged in the Tyranid yardstick lists

Field names only (never values). Already out by the panel's own patchDrop: HivemindHobbies and Into the Hive Mind: Pyrovores, Neurotyrant, Tyrannocyte, Toxicrene.

• HivemindHobbies, at 20 Jun (by date): points 10%+: Biovores, Neurolictor, Tyrannofex, Tyranid Prime with Lash Whip, Zoanthropes, Von Ryan's Leapers, Neurotyrant, Parasite of Mortrex, Hive Guard, Tyrannocyte, Ripper Swarms, Toxicrene, Harpy, Hive Crone, Venomthropes (the launch points, entered by BSData on 24 Jun); rules (abilities): The Red Terror, Norn Assimilator, Norn Emissary, Tyranid Warriors with Melee Bio-Weapons; keywords alone: Maleceptor (a spelling fix), Harridan.
• Into the Hive Mind, at 27 Jul (by date): points: Pyrovores (−11% at 1 model); rules: Tyrannofex (Rupture cannon: AP), Neurotyrant (abilities); keywords alone: Harridan, Maleceptor.
• Both, at 2 Jul (their files' commit): points: Zoanthropes (−10% at 3), Von Ryan's Leapers (−12.5% at 6, missed by the smallest-size check), plus the four already dropped; rules: The Red Terror (keywords, abilities), Tyranid Warriors with Melee Bio-Weapons (abilities); keywords alone: Harridan, Maleceptor.
• Auspex (12 Sep): nothing; the frozen commit holds the same Tyranid data.
• Maelstrom and Astrategas: every unit, "edition"; all 48 of each exist in 11th edition.

Doubts and limits

• The commit by date can't see BSData's lag behind the official release, nor which points a reviewer actually had in mind (a video can be recorded days before it's posted). Hence the two readings; neither closes the gap.
• The field diff knows names, not meaning: a changed ability name may be a renamed rule with the same effect, and a rule's wording can change without its name changing (invisible here). The bookkeeping list was read off the diffs between these commits; a new BSData refactor would need adding to it.
• Only the Tyranid lists were scored. The Space Marine Tobias panel (8 Jul) carries 11 points flags, 9 rules and 13 keyword-only flags; worth the same check if the Space Marine gap is questioned.