Claim Verification

Reasoning-relevant experts specialize early in training, before step 12k.

4 linked sources

Routing entropy correlates with downstream reasoning accuracy across all three model scales tested.

2 linked sources

The probe predicts benchmark performance before fine-tuning completes.

0 linked sources

Sparse MoE models outperform dense models of equal active-parameter count on reasoning benchmarks.

3 linked sources

Reasoning specialization is detectable via linear probing of routing statistics alone.

5 linked sources

Expert utilization plateaus by the midpoint of training regardless of model scale.

1 linked source

Routing entropy is a stronger predictor of reasoning accuracy than raw parameter count.

4 linked sources

Reasoning capability transfers zero-shot across unrelated benchmark domains.

0 linked sources

Probe-based specialization signals generalize to dense transformer baselines.

2 linked sources