Models & Architectures
Mixture of Experts (MoE)
An AI architecture where only a fraction of the model "wakes up" per request.
Also known as: MoE,sparse model
Mixture of Experts (MoE) is an AI architecture where the model is internally split into many "expert" sub-networks, and only 2–3 of them activate for any given input. A small router network decides which experts handle each request. The result: a model can have hundreds of billions of parameters but only compute like a much smaller one for any single call. MoE is the trick that broke the old "bigger = slower" trade-off. In 2026, almost every frontier model uses MoE — GPT-5, Claude, Gemini, DeepSeek V3, Llama 4, Mistral. DeepSeek V3 has 671B total parameters but only activates 37B per token. That is why it can be smart AND fast.