MACHINE LEARNING · NEURIPS 2026

Emergent Reasoning Chains in Sparse Mixture Models

Alaya, R. · Ferro, D. · Whitcombe, N. — University of Halden Institute for AI

Abstract

We study how multi-step reasoning behavior arises in sparsely-activated mixture-of-experts models without explicit chain-of-thought supervision. Across three model scales, we find that reasoning-relevant experts specialize early in training and that routing entropy correlates with downstream reasoning accuracy. We introduce a lightweight probe for detecting this specialization and show it predicts benchmark performance before fine-tuning completes.

1. Introduction

Sparse mixture-of-experts (MoE) architectures have become a standard tool for scaling model capacity without proportional increases in compute. Yet the internal mechanisms by which such models come to support multi-step reasoning remain poorly understood. In this section we motivate a probe-based approach to studying expert specialization over the course of training, building on prior routing-analysis literature.

2. Related Work

Prior work on routing analysis has largely focused on load balancing and expert utilization rather than the emergence of reasoning-specific circuits. We build on this literature by introducing a probe that tracks routing entropy per expert cluster across training, rather than aggregate utilization statistics alone.

3. Method

We train three MoE model scales (1B, 8B, and 32B active parameters) on an identical data mixture and checkpoint routing statistics every 500 steps. A lightweight linear probe is fit to routing entropy at each checkpoint to predict held-out reasoning-benchmark accuracy, without access to the benchmark labels during pretraining.