Mixture of Experts, explained for builders

The short version

A mixture-of-experts (MoE) model is a neural network that doesn't use its whole brain for every token. Instead, a small routing network picks a handful of "expert" sub-networks to handle each token, and the rest stay dormant. The result: you get the capacity of a giant model while only paying the compute cost of a much smaller one.

If that sounds like a free lunch, it isn't — but it is the closest thing the field has found.

How routing actually works

Every MoE layer contains N expert feed-forward networks (typically 8 to 64) plus a gating network. For each token, the gate computes a score per expert, keeps the top-k (usually 1 or 2), and the token's representation becomes a weighted combination of just those experts' outputs.

Two details matter more than anything else in practice:

• Load balancing. Without intervention, the gate collapses and routes everything to two or three experts. Training adds an auxiliary loss that penalizes imbalance, effectively bribing the router to spread work around.
• Capacity factor. Each expert can only process a fixed number of tokens per batch. Overflow tokens get dropped or passed through unmodified. Set the factor too low and you silently lose information; too high and you waste the sparsity you built the whole thing for.

Why builders should care

The headline benefit is inference economics. A 7B-active MoE can match a 13B dense model on many benchmarks while costing roughly half per token to serve. For anyone running agents that burn thousands of tokens per task, that arithmetic decides whether a feature ships.

The trade-offs are real, though:

1. Memory, not compute, is the bottleneck. All experts live in VRAM even though only a few fire per token. A 47B-total MoE needs ~90GB of weights in fp16 — you quantize, or you don't serve it.
2. Fine-tuning is fussier. Experts specialize during pretraining; naive fine-tuning can wreck the routing balance. Most teams fine-tune with the router frozen, or use expert-parallel-aware recipes.
3. Batching behavior differs. Throughput scales beautifully with large batches (experts stay busy), but single-stream latency looks worse than the active parameter count suggests.

When dense still wins

Small models, latency-critical paths, and edge deployment still favor dense architectures. MoE pays off when you're serving at scale and your workload is diverse enough that specialization actually helps — which, for general assistants and coding agents, is most of the time now.

Rule of thumb: compare models by active parameters for speed and total parameters for capability. Anyone quoting only one number is selling you something.