Different models, not more dials: statistical learning on the reviewer letters (T-736)

6 Oct 2026, Pacific. tools/statlearn.js; results in build/statlearn/ (git-ignored: features.json and .csv, results.json). Snapshot 8dcd24032d, inputs hash dcd477d8a78c. Base: Track A (tools/ledger-benchmarks/2026-10-05-step34b-baseline.json), wCarry 0. A report: nothing in the ledger moves, no slider, setting or letter changes, and nothing is adopted.

In plain English

The question. For days we have tuned dials on one model: Value per 100 points, about 85–91% of it Kill. It sits near 62% agreement with the reviewers across 16 armies. Jordan asked for structurally different models instead, tested cheaply and honestly. So I built one table of 75 facts about each of 733 units (595 graded), and fitted 34 models of different kinds on it, plus every fact alone.

How it was judged.

• Every model was fitted on 15 armies and scored on the 16th, for each army in turn. No army's letters ever tune its own score.
• The score is the share of the held-out army's reviewer pairs put in the right order (cascade2's consensus pairs; ties count wrong).
• Every model was also refitted on 20 sets of shuffled letters. All 34 beat at least 19 of the 20 shuffles.
• Each is compared with the ledger by a bootstrap over armies.
• The ledger was tuned on Tyranids, so its Tyranid score (89.5) is in sample. The fairest column is 15 armies, no Tyranids, where the ledger scores 60.0.

The leaderboard (the main ones; the full table is below).

What this says.

1. Re-shaping the ledger's own information gets nowhere. Logistic (L1 and L2), ordinal, boosted trees, nearest neighbours, value over replacement and leaders at their joined form all land within about 2 points of the ledger. So does Value with every fact added. The ledger is about as good as its inputs allow. The Kill line alone does as well as the whole ledger (62.6).
2. The gain comes from information the ledger does not use. A model that never sees the ledger beats it by about 3 points on 15 armies. It sees only the datasheet, keywords and support rules. Averaging its rank with the ledger's is the one model whose interval clears 0 on the human letters: +3.9 [+0.4, +7.3] on 15 armies, better in 9 of 16. It was declared before the run.
3. A second, independent target agrees. The models were trained on human letters only. On Gemini's letters for the held-out army, the datasheet model beats the ledger by +7.4 [+1.3, +13.3] and the ensemble by +6.2 [+3.0, +9.8]. Caveat: Gemini says it draws on tournament results, and list share does just as well there. So some of that agreement may be popularity.
4. Ranks beat raw values. The same every-fact model on each fact's rank within its army scores 65.2 against 61.4. Our facts are heavy-tailed, and a few outliers dominate a raw fit. Ranks are robust to that.
5. Within-role scoring is worse, not better. Within role, within archetype, value over replacement and replaceability all lose 3 to 14 points. Reviewers rank across roles. A unit that is the best of a weak role is not graded up for it. (Our roles come from simple rules, not T-732's measured roles. A better role assignment might do better.)
6. Multiplicity. I tried 34 models. One interval clearing 0 on the human letters is suggestive, not proof. The Gemini agreement is the stronger evidence, because Gemini's letters played no part in any fit. Nothing should be adopted before a confirmation run on lists none of these fits saw.

What reviewers seem to value. These signs hold in all 16 leave-one-out fits, across the datasheet model, the rank model, the ordinal model and the L1 model. Each is "one army-SD more of this, within an army".

• Up: Move (the largest and steadiest); Epic Hero; Psyker; Lone Operative; Infantry (hideable in 11e terrain); Soak (durability); Toughness and Wounds; Deep Strike; a command-point rule; an army-rule enabler; Kill into tanks, elites and hordes; points (dearer units are graded higher within an army, all else equal).
• Down: Fly (aircraft, skimmers and jump packs, all else equal); a debuff rule; Transport; Vehicle; long range; and a pile of support-rule kinds once the rest is known.
• Lines. A lines-only refit gives Soak and Kill about equal weight (31% and 28% of its weight), then Score 15% and Hold 11%. The ledger gives Kill 85% and Soak 3%. The scales differ (the fit is per army-SD of log(1 + line)), so read the order, not the sizes. Still, durability is badly under-weighted against killing.
• Cheapness does not help on its own. OC per point, OC kept, holding and screening alone are at chance (49–50, ties as half). Cheap activations are slightly negative.
• Gemini as training data. Human + 0.5 × Gemini pairs moves held-out human accuracy by +0.7 (every-fact logistic), −0.3 (lines) and +1.2 (trees). Small, as T-731's half weight suggests.

Prediction (written before any fit), checked.

• Wrong that nothing clears the ledger by 2 with an interval clear of 0: the ensemble does, by +3.6 to +3.9.
• Right that durability gets far more weight than the ledger gives it. Wrong on cost: dearer units are graded higher, and cheapness and OC per point carry nothing.
• Wrong that ranks would do about as well as raw values: ranks are better by 3.8.
• Wrong on within-role scoring: it is worse by 4 to 5, not even.
• Wrong on leaders at their joined form: −1.8, not +0.5 to +1.5.
• Right that trees do no better than linear models.
• Right that real models beat scrambled ones: 7 to 16 points above the scrambled mean.
• Right that Gemini training moves things little (+0.7 for the every-fact logistic; +1.2 for trees, just over the bound).

Recommendation

1. Stop tuning dials on the present lines. Every reshaping of the ledger's own information, linear, ordinal, trees, neighbours, roles, lands within about 2 points of it.
2. The missing signal is on the datasheet. Speed, named characters and psykers, Lone Operative, Infantry (Hidden), durability. A ledger-free datasheet model matches or beats the ledger.
3. Run one confirmation of the declared ensemble (ledger rank + datasheet-model rank) on human lists none of these fits saw, such as new reviewer lists as they arrive. Adopt nothing before that.
4. In the ledger itself, Soak deserves weight comparable to Kill. Speed and character or psyker status are mechanisms worth pricing as lines.
5. Drop within-role, value-over-replacement and replaceability as letter models. Use ranks rather than raw values in any pooled model.

Method notes

• Features (75; build/statlearn/features.csv lists every value): each line's contribution per 100 points; Kill by target class; the datasheet (points, models, T, Sv, invulnerable save, W, wounds per 100 points, Move, OC, OC per 100 points, Ld, Feel No Pain, range); keywords (Character, Epic Hero, Vehicle, Monster, Infantry, Fly, Transport, Battleline, Psyker, Synapse, Deep Strike, Infiltrators, Scouts, Lone Operative, Stealth, Fights First, Leader); leading (the best joined form as leader and as squad, readable.js's package and lift); support (aura, CP, Battle-shock and debuff rules, how many kinds, enabler, transport capacity); and design/jobs.md's 22 job measures. The datasheet-only model uses the datasheet, keywords, support and Leader columns: no line, Kill class or job.
• Training pairs: each army's consensus pairs, weighted to total its graded units (halved for a single-list army). Features are standardised within each army. λ is chosen by an inner leave-army-group-out (4 groups of the training armies). Trees: depth 2, 100 rounds, learning rate 0.1, at least 20 units a leaf; fixed in advance, not tuned.
• Roles (Jordan's eight) come from simple rules in priority order: transport; character leader or support with low Burst; debuff, shock or character-hunter; cheap low-Kill bodies; action-doer at 110 points or less; anti-tank shooter; durable holder; else Hammer. Counts per army are at the end of the tables.
• Post hoc: three models were added after the first run had shown the datasheet and rank models on top. They are labelled POST HOC and are not counted as evidence: datasheet facts on ranks (64.0), datasheet + log Value (63.5), and the ledger plus the every-fact rank model (63.5).
• Ties: single flags tie on most pairs, so their strict score sits far below 50. The single-feature table sorts on ties as half a pair instead.
• The Actions line's coefficient in the lines-only fit comes out at about 0 (0.0001), so it rounds out of that table.
• Reproduce: node --max-old-space-size=3000 tools/statlearn.js --features (about 10 s), then node tools/statlearn.js --fit --scrambles 20 --boot 2000 --md <file> (about 14 min on one thread); --only <prefixes> refits some models into the same results file; --md-only <file> rewrites the tables.

The tables (tools/statlearn.js --md)

The leaderboard (held out, leave one army out)

Pooled = mean over the 16 armies of the held-out accuracy on each army's human consensus pairs (cascade2's definition); every pair = all armies' pairs together; lists = each human 11e list's own pairs (army mean). Scrambled: the identical pipeline on 20 letter sets permuted within each army. vs ledger: pooled difference from Track A's Value with a 95% bootstrap interval over armies (2000 draws); armies = how many of 16 it beats the ledger in. 15 (no Tyranids): the same without the army the ledger was tuned on (its Tyranid score is in sample), with its own bootstrap over the 15. Gemini = held-out accuracy on the held-out army's GEMINI AGGREGATE pairs (15 armies; Emperor's Children has none). Family: a ledger, c pairwise logistic, d ordinal, e within role, f value over replacement, g ranks, h boosting, i replaceability, j leaders joined, k diagnostic, plus/knn/ens other structural tries, gem trained with Gemini.

Each feature alone (family b; its sign chosen on the training armies)

Sign: + higher is better, as most training folds chose it (folds agreeing of 16). All 75 features listed, sorted by the score with ties as half a pair (flags and whole-number stats tie on most pairs, so the strict score, ties wrong, sits far below 50 for them).

What the fits value: coefficients

Pairwise logistic, every feature, L2

Fitted on all 16 armies at λ 300 (λ in the held-out folds: 300, 100, 30). Coefficient per army-SD of the feature (on the score's logit scale); folds = how many of the 16 leave-one-out fits share its sign; range over the folds.

Pairwise logistic, every feature, L1

Fitted on all 16 armies at λ 10 (λ in the held-out folds: 3, 10). Coefficient per army-SD of the feature (on the score's logit scale); folds = how many of the 16 leave-one-out fits share its sign; range over the folds.

Pairwise logistic, the seven lines only, L2

Fitted on all 16 armies at λ 100 (λ in the held-out folds: 100, 3, 10, 300, 30, 1000). Coefficient per army-SD of the feature (on the score's logit scale); folds = how many of the 16 leave-one-out fits share its sign; range over the folds.

Pairwise logistic, datasheet + keywords + support only (no ledger), L2

Fitted on all 16 armies at λ 10 (λ in the held-out folds: 3, 30, 10). Coefficient per army-SD of the feature (on the score's logit scale); folds = how many of the 16 leave-one-out fits share its sign; range over the folds.

Pairwise logistic, design/jobs.md's jobs only, L2

Fitted on all 16 armies at λ 1000 (λ in the held-out folds: 1000, 300, 30, 100). Coefficient per army-SD of the feature (on the score's logit scale); folds = how many of the 16 leave-one-out fits share its sign; range over the folds.

Pairwise logistic, every feature + log Value, L2

Fitted on all 16 armies at λ 100 (λ in the held-out folds: 300, 100, 30). Coefficient per army-SD of the feature (on the score's logit scale); folds = how many of the 16 leave-one-out fits share its sign; range over the folds.

Ordinal logistic on letters, every feature, L2

Fitted on all 16 armies at λ 30 (λ in the held-out folds: 30, 300, 3). Coefficient per army-SD of the feature (on the score's logit scale); folds = how many of the 16 leave-one-out fits share its sign; range over the folds.

Pairwise logistic on within-army ranks, every feature, L2

Fitted on all 16 armies at λ 3 (λ in the held-out folds: 3). Coefficient per army-SD of the feature (on the score's logit scale); folds = how many of the 16 leave-one-out fits share its sign; range over the folds.

Role intercepts + Value within role (pairwise fit)

Fitted on all 16 armies at λ 100 (λ in the held-out folds: 100, 30, 10, 1). Coefficient per army-SD of the feature (on the score's logit scale); folds = how many of the 16 leave-one-out fits share its sign; range over the folds.

Archetype intercepts + Value within archetype (pairwise fit)

Fitted on all 16 armies at λ 3 (λ in the held-out folds: 1, 100, 3). Coefficient per army-SD of the feature (on the score's logit scale); folds = how many of the 16 leave-one-out fits share its sign; range over the folds.

Value + value over replacement (pairwise fit)

Fitted on all 16 armies at λ 3 (λ in the held-out folds: 10, 1, 30, 3). Coefficient per army-SD of the feature (on the score's logit scale); folds = how many of the 16 leave-one-out fits share its sign; range over the folds.

The ledger's line weights against the fitted ones

Ledger: each line's mean share of a unit's Value (Track A). Fitted: the lines-only pairwise model's coefficients as a share of their absolute sum (a different scale: per army-SD of log(1 + the line), so read the order and sign, not the size).

Boosting: split-gain importance (all 16 armies)

By army (held out)

Gemini (GEMINI AGGREGATE letters, reference only)

Every model above was scored on the held-out army's Gemini pairs (the Gemini column; human letters only in training, so Gemini's letters are a second, independent target). Against the ledger on those pairs (15 armies, bootstrap over armies):

Trained on human + 0.5 × Gemini pairs (Gemini at half a single human list's weight), scored on held-out human pairs only:

The eight roles as the rules assign them (counts per army)