Target-Local Verifier Choice in Best-of-K Reasoning Selection

Part 01

One verifier, many generators

Verifier-guided selection is now standard infrastructure. Sample K candidate solutions, score each with a reward model, return the top-ranked answer. It is cheap relative to training, it composes with any generator, and it reliably converts sampling compute into accuracy.

A production service, though, rarely serves one generator. It serves a fleet — different sizes, different families, base and instruct, reasoning-tuned and not — and it almost always reuses one verifier across all of them. That is an operational assumption, not a measured one: it holds if the best verifier is a property of the verifier pool, and fails if the best verifier depends on the candidate distribution being served.

Most verifier work asks how to build a stronger verifier, or how much test-time compute to spend on a prompt. Compute-allocation work predicts per-prompt difficulty and chooses how much decoding or search effort to spend. Router work sends prompts to generators under quality/cost tradeoffs. We ask a different question, and it changes what is being allocated: we hold the candidate set fixed and allocate verifier identity.

Part 02

The panel: 34 targets, 7 verifiers, one frozen eval mix

Everything downstream depends on the comparison being matched: the same problems, the same candidates, the same extraction, with only the verifier changing. Every generator produces K=64 candidates per problem at T=0.7, top-p=0.95, max_new=4096, seed 47. Every verifier consumes the full candidate trace, and selection is argmax. The metric is top-1 pick@1, averaged over the problem set.

Part 03

The best fixed verifier is not the best local verifier

PRM-7B wins the aggregate cleanly. Averaged over the 22 first-pass targets it is the strongest single fixed verifier — 43.02% pick@1, with the next best fixed choice (V4-9B) trailing at 38.91%. If the question were "which one verifier should I train and ship?", the answer would be settled.

It is not settled per target. On the 22-target panel the local winners split 13 PRM / 8 V4 / 1 ORM-boundary; on the 12 second-pass targets, 8 / 3 / 1. Because winners land in all three classes rather than clustering on one weaker family, the effect is a genuine allocation problem and not an artifact of a soft baseline.

Scale does not explain it three families, three directions

The obvious hypothesis is that bigger, better generators simply outgrow the PRM. Three family-scale views inside the panel move in three different directions, which rules that out:

• Qwen3.5-Base (0.8B → 9B) moves PRM → V4: PRM-class at 0.8B (McNemar p=0.018), then V4 winners at 4B and 9B beating fixed PRM by +6.00 and +8.33 pp, both surviving the direct winner-vs-PRM Holm comparison.
• Qwen3 (0.6B → 32B) moves the opposite way: V4-class from 1.7B to 4B, a crossover near 8B, and PRM at 32B by +8.33 pp (p=0.0015).
• Phi (3-mini → 3-medium → 4) never crosses over at all: all three stay PRM-class with p<0.05, but the margin shrinks steadily from +13.67 pp to +5.33 pp.

Two families of similar capability moving in opposite directions is the cleanest evidence that the driver is the candidate distribution, not raw ability. The next section makes that concrete: the two trajectories are visible in the candidate texts themselves.

Part 04

Reading the regime off the candidates, before scoring

A mismatch you can only detect by running all seven verifiers is not operationally useful — you would have already paid for the answer. So we ask whether the regime is visible in the candidate set alone. For each routable target we compute four statistics over its K=64 candidate texts: boxed-answer rate, log candidate length, answer-cluster entropy, and max-cluster share. No labels, no verifier scores.

How well does it predict? 19/21 first pass · 27/32 combined

An L2-regularised classifier on those four features, evaluated under target leave-one-out, recovers the point-estimate PRM/V4 regime on 19 of 21 first-pass routable targets (90.5%, Wilson CI95 [71.1, 97.3]%) and 27 of 32 on the combined panel. A permutation null on the first-pass panel gives p=0.0003; the combined-panel V4 prior is 12/32 = 37.5%, so the combined number is well above base rate. Adding the second-pass targets costs roughly nine points of accuracy — the honest cost of extrapolating a fitted diagnostic to generators it never saw.

The signal is not one lucky number. It survives being taken apart:

• Family and architecture holdouts: 11/12 and 20/21 correct when whole families or whole architectures are held out of training.
• Feature ablations: dropping boxed-rate or dropping length individually still leaves 20/21 — no single surface statistic is carrying the classifier.
• Parser and style controls: 19–20/21 under style-stripped extraction.
• The obvious cheap baseline fails: a metadata-only router keyed on model size and training tags collapses to 13/21, and loses −0.67 pp against fixed PRM. Knowing the model card is not enough; you have to look at what it actually generated.
• Re-fit on an expanded 9-verifier pool (adding two non-Qwen verifiers, see Part 06): 29/34 = 85.3%, against a 52.9% base rate.

Part 05

A budgeted response menu, not a new verifier

The diagnostic is only worth having if it changes what you ship. We turn it into a menu of policies indexed by two budgets a practitioner actually controls: how many gold labels you can afford on probe problems, and how much online verifier scoring you can afford per request. Every policy is evaluated as a panel-average delta against fixed PRM-7B, with bootstrap CIs over targets.

How many labels do you actually need? m=30 starts paying

Part 06

Where it holds, and where it breaks

A result like this is only as good as the checks that could have killed it. We ran five, and report them in both directions.

The held-out slice AIME 2026 · +3.88 pp

In the strict split, the 270 non-AIME-2026 problems supply calibration probes and the 30 AIME 2026 problems form a deterministic test set for all 22 targets. PVA-Adapt full retains +3.88 pp (CI95 [+1.56, +6.15]) and PVA-Calibrate +2.45 pp ([+0.77, +4.37]) — the selected policies match their within-pool run exactly, while oracle headroom drops from +9.55 to +5.30 pp. This is a same-panel leakage check, not new-generator or private-benchmark transfer.

The seed replay the router is the casualty

Verifier lineage 9 of 34 targets go to a non-Qwen verifier

The sharpest objection to the original panel is that all seven verifiers share a Qwen lineage, so the local winners might be tracking a style match rather than a real regime. We scored the frozen candidate banks with two verifiers from a different lineage under their official protocols — Skywork-Reward-V2-Llama-3.1-8B as a sequence classifier and RLHFlow Llama3.1-8B-PRM-Mistral-Data with mean-of-step-rewards — and repeated the scoring on the second candidate bank.

• The global ranking survives. PRM-7B remains the point-estimate best fixed verifier on both banks (43.5% / 43.3% mean pick@1), with Skywork-V2 second (42.7% / 42.6%) and RLHFlow well back (37.6% / 37.3%).
• Local winners still move. The new verifiers are strict local winners on 9 of 34 targets and tie for the lead on 2 more; the max-over-9 per-target oracle is +2.1 pp on the 34-target panel, against +1.6 pp over the original seven.
• And it replicates. Five of the seven Skywork strict wins recur as strict wins on the second bank, and a sixth becomes a tie — 6/7 non-losses.

Prompt and decoding stress flips live on thin margins

If the regime flipped every time someone edited a system prompt, the diagnostic would certify nothing. Under two alternative prompts (open chain-of-thought and structured steps), the winner class is preserved on 10/12 and 10/11 evaluable second-pass targets, and every observed flip sits on a cell whose original winner margin is ≤1.67 pp. Regenerating all 22 first-pass banks at T=1.0 preserves the winner class on 19/22 targets (20/22 on the deployment-relevant PRM-vs-non-PRM split) and the exact winner on 18/22; all three flips sit at margins ≤2.0 pp, while every one of the 11 cells with a margin ≥3.67 pp preserves its class.

The most informative case runs the other way. For Mistral-Nemo, the frozen diagnostic predicted PRM (p_V4 ≈ 1e−4) while the observed bank-1 winner was V4 by one point — and both alternative templates moved the observed winner to PRM, toward the frozen prediction.

The negative result selection quality ≠ training-data quality

Everything above measures selector correctness. A separate and tempting question is whether better-selected traces make better SFT data. We fine-tuned a Qwen3-8B Instruct student on full PVA-Adapt-selected traces (m=75) and evaluated pick@1 on 210 non-floor problems. No variant showed positive transfer, and the full-7 arbiter's data was significantly negative.

Part 07

What to actually deploy

The paper's recommendation is deliberately conservative: keep the strong PRM default, and spend extra scoring or labels only on the targets the diagnostic flags. The table below is the decision procedure, with each row's honest downside attached — gains are reported on both candidate banks, because a policy that only works on one seed is not a policy.

Part 08

Takeaways

• Verifier choice in Best-of-K is a target-local allocation problem, not a fixed configuration. The strongest average verifier is not the best verifier for every generator you serve, and the local winners split across process, outcome, and scalar reward models.
• You can see the regime before you pay for it. Four label-free statistics of the candidate texts predict the coarse PRM/outcome regime at 19/21 and 27/32 under target leave-one-out — while a metadata-only router keyed on model size and training tags does worse than nothing.
• The deployable answer is a menu, not a replacement. Score fusion and a small-label arbiter buy roughly two points over the fixed default on two independent candidate banks; the no-label router buys less and is seed-sensitive.
• The boundaries are as informative as the gains: gains are modest, the router carries a −3.33 pp worst target, the evidence is conditioned on one decoding setup and one verifier lineage per class, and better-selected traces produced no detectable SFT utility.

The useful reframing is small but load-bearing: stop asking which verifier is best, and start asking which verifier is best for the distribution you are about to serve — then decide, honestly, whether the two points are worth the switch.