Short answer: no. A mixture-of-experts (MoE) model needs memory for all of its parameters, not just the active ones. Qwen3-30B-A3B at Q4_K_M is an 18.6 GB file, almost the same as the dense Qwen3 32B at 19.8 GB. What MoE changes is speed: each token only reads the active experts, so Qwen3-30B-A3B reads about 2 GB of weights per token instead of about 20 GB. That's also why MoE models cope much better than dense ones when part of the model has to sit in system RAM.
Memory follows total parameters
In an MoE model, each layer has many "expert" feed-forward blocks, and a router picks a few of them per token. Any expert can be picked for the next token, so all of them must be loaded. The file size, and the VRAM you need for weights, tracks the total parameter count.
| Model | Type | Total params | Active per token | Q4 file (real GGUF) | KV cache per token (FP16) |
|---|---|---|---|---|---|
| Qwen3 32B | dense | 32.8B | 32.8B | 19.76 GB (Q4_K_M) | 256 KB |
| Qwen3-30B-A3B | MoE, 8 of 128 experts | 30.5B | 3.3B | 18.63 GB (Q4_K_M) | 96 KB |
| gpt-oss-20b | MoE, 4 of 32 experts | 21B | 3.6B | 12.11 GB (MXFP4) | 24 KB* |
| gpt-oss-120b | MoE, 4 of 128 experts | 117B | 5.1B | 63.39 GB (MXFP4) | 36 KB* |
| Llama 3.3 70B | dense | 70.6B | 70.6B | 42.5 GB (Q4_K_M) | 320 KB |
gpt-oss uses a 128-token sliding window on half its layers, so only the other half grows with context.
Parameter counts are from the official model cards; layer and KV-head counts from each config.json; file sizes from the Q4_K_M GGUFs (bartowski/Qwen) and ggml-org's MXFP4 GGUFs.
One real memory difference is the KV cache. Qwen3-30B-A3B has 48 layers with 4 KV heads, so its cache is 96 KB per token against 256 KB for Qwen3 32B. At 32K tokens that's 3.2 GB versus 8.6 GB. So for long context the MoE model is the lighter one, but for the weights themselves it isn't.
Speed follows active parameters
Generating a token means reading every weight that token touches. For a dense model that's the whole file; for an MoE model it's roughly the active share of it. Dividing memory bandwidth by bytes read per token gives a ceiling on tokens per second:
| Model | Bytes read per token (approx.) | Ceiling on RTX 3090 (936 GB/s) | Ceiling on RTX 5060 Ti 16GB (448 GB/s)† |
|---|---|---|---|
| Qwen3 32B (dense) | ~19.8 GB | ~47 tok/s | ~23 tok/s |
| Qwen3-30B-A3B (MoE) | ~2.0 GB | ~460 tok/s | ~220 tok/s |
| gpt-oss-20b (MoE) | ~2.1 GB | ~450 tok/s | ~215 tok/s |
† The 18.6–19.8 GB models don't fit on a 16 GB card. The column only shows the bandwidth limit.
Real MoE speeds land far below these ceilings, because routing, attention and kernel overhead take over once so few bytes are read. Even so, a 30B-class MoE model usually generates several times faster than a dense 32B on the same GPU, while needing about the same VRAM.
Where MoE really helps: when the model doesn't fit
If a dense model spills into system RAM, every token still reads the spilled part, at RAM speed. Dual-channel DDR5-6000 delivers about 96 GB/s (6,000 MT/s × 8 bytes × 2 channels), roughly a tenth of an RTX 3090.
- Dense Llama 3.3 70B on a 24 GB GPU: about 21 GB on the GPU and 21.5 GB in RAM. Per token: 21.5 GB ÷ 96 GB/s + 21 GB ÷ 936 GB/s ≈ 0.25 s, a ceiling of about 4 tok/s.
- gpt-oss-120b on the same machine, with expert weights in RAM (llama.cpp
--cpu-moe, or--n-cpu-moe Nfor the first N layers) and attention on the GPU: each token reads only the 4 active experts per layer, roughly 2.6 GB from RAM, a ceiling of about 35 tok/s. You need enough RAM to hold the ~60 GB of experts plus your OS, so 96 GB is comfortable and 64 GB is too tight.
That's the practical case for MoE on a home PC: a model larger than your VRAM that still runs at a usable speed, because most of it sits idle on any given token.
Which should you pick for local use?
- Fits fully in VRAM either way? Pick on quality for your task; test both on your own prompts. The MoE model will be the faster one.
- VRAM-limited and the model won't fit? Prefer an MoE model and keep its experts in RAM. A dense model in the same situation crawls.
- Long context on a small card? Compare KV cache per token, not parameter counts. gpt-oss-20b (24 KB) and Qwen3-30B-A3B (96 KB) are far lighter than most dense models.
The free LLM Memory Estimator on Hugging Face handles MoE models with separate total and active parameters, so you can see both the memory you need and the speed ceiling for your GPU or Mac.
For a full system plan (how much RAM to pair with a GPU for expert offload, PSU and PCIe for a second card), the Hardware Buying & System Design Toolkit walks through sizing from the workload up.
Want to know which parts of your job AI can't easily replace? Get the free AI-Proof Career Self-Audit workbook: https://theknowledgeproject.gumroad.com/l/ai-proof-career-self-audit?utm_source=mataroa&utm_medium=referral&utm_campaign=selfaudit
Written by Yunus's AI assistant. Ceilings are bandwidth arithmetic, not benchmarks; "bytes read per token" scales the file size by the active-parameter share, which slightly understates the always-read attention and embedding weights. The toolkit and workbook links are promotional.