98 tokens/s on an AMD RX 7900 XT (PCIe 3.0, Ryzen 7 3800XT, 64 GB DDR4) for Qwen3.8-Flash-Next (125B MoE).
16 GB of RAM is enough: the 89 GiB model runs at ~49 tokens/s with 16 GB and ~88 with 32 GB, thanks to Avalon's own RAM cache of experts (below).
Avalon is a fork of llama.cpp built for one model, Qwen3.8-Flash-Next (512 experts, 10 active, 48 layers), on one consumer GPU with the experts in system RAM. It is not tuned on a flagship machine: every number below comes from a desktop most people would call old.
| part | what we have | why it matters |
|---|---|---|
| CPU | AMD Ryzen 7 3800XT (Zen 2, Ryzen 3000 family, 2019), 8 cores | computes the experts that are not on the GPU |
| RAM | DDR4-3200, dual channel (33 – 35 GB/s measured), 64 GB (16 GB works) | every expert missing from the GPU is read from here |
| GPU | AMD Radeon RX 7900 XT 20 GB (RDNA3, 2022), ROCm, no NVIDIA, no CUDA | holds the dense weights, the KV cache and a cache of the hottest experts |
| bus | PCIe 3.0 x16 (~14 GB/s), on a B450 board | every expert copied to the GPU goes through it |
The model is ~89 GiB; the GPU holds ~4.6 GiB of dense weights plus a cache of 9 – 14 GiB of experts, the rest stays in RAM, and with less RAM than the experts take (~49 GiB) Avalon reads the missing ones from the disk itself. The disk here is an NVMe SSD, a Kingston FURY Renegade (PCIe 4.0) running at PCIe 3.0, ~3.7 GB/s; a SATA SSD was not measured, and with less RAM the speed depends on it. Numbers with less RAM: The RAM cache of experts.
Squeeze every last drop out of the hardware. RAM, VRAM, CPU, PCIe link and GPU should all be busy, all the time, at their limit. No byte of VRAM sits idle: whatever a phase does not need goes back to the expert cache. Compute buffers are lent to the cache between phases and handed back when a graph needs them, the slots of the prefill prefetch are borrowed from it, the end of the KV cache past the current context belongs to it until the context grows there, and the reserve kept outside it is calculated, not guessed. The CPU computes the missing experts while the GPU computes the cached ones. During prefill the PCIe link prefetches the next experts so the GPU hardly ever waits for them, while during generation it carries the experts the dispatcher moves to the GPU and the cache fills.
Optimization in partnership with an AI. The macro optimizations, the architecture and the way the work flows between RAM, CPU, PCIe and GPU, come from me and my 30 years of optimization work. The AI first served to build the simulators, benchmarks and instrumented measurements that made each decision precise. Then every micro optimization, the GPU kernels and CPU kernels above all, was written entirely by Claude Opus 5.5, with me steering. I am only the driver; the AI is the engine, and the rest of the car.
IQ3_KS hybrid (2 – 4-bit experts, perplexity close to Unsloth's UD-IQ4_XS), greedy decoding, 512 tokens, MTP speculative decoding with 2 draft tokens; best case: mean of 3 answers, the other rows: median of 2 runs (Nov 1):
| case | generation, no MTP | generation, MTP | prefill, no MTP / MTP |
|---|---|---|---|
| 32k context, short prompt, warm cache | 66.1 t/s | 98.4 t/s | — |
| 32k context, 32k filled | 62.3 t/s | 88.0 t/s | 1,854 / 1,680 t/s |
| 256k context, 131k filled | 60.5 t/s | 88.0 t/s | 1,693 / 1,521 t/s |
| 256k context, 250k filled | 57.3 t/s | 81.1 t/s | 1,491 / 1,342 t/s |
Missed experts that every token of a verification routes only at rank 7 or lower are skipped, as Strata does (on by
default; GGML_CUDA_MOE_TAIL_SKIP=0 turns it off). Without it: 62.8 / 90.7 t/s on a short prompt, 51.2 / 71.0 t/s at
250k filled, prefill within 3%. GSM8K is the same with and without (see Quality).
The full 256k context fits with the MTP draft model, the KV cache in q8_0 / q4_0 and the expert cache, under 19.7 GiB of VRAM.
The experts take ~49 GiB. llama.cpp maps the model file and leaves them to the kernel's page cache, and Avalon registers them with the GPU driver so that the expert cache copies them without a staging buffer. Two things go wrong with that when RAM gets short:
- the page cache is shared: any other program (a browser, a build) can push experts out, and they come back from the disk at the next page fault, in the middle of a token;
- the GPU driver limits the memory registered by all processes to RAM − RAM/64 − 1.5 GiB. On a 16 or 32 GB machine most experts cannot be registered; the GPU then copies them from plain memory, which the driver locks on the fly under the same limit, and a long prompt ends in a crash.
So Avalon keeps its experts in RAM it owns, instead of the page cache, and the two are never mixed. At start it looks at the RAM it can take (and the driver's limit):
- every expert fits (64 GB here): they are read once with O_DIRECT into Avalon's own memory, at the same addresses, and registered in place. Nothing else can evict them, and the server is at full speed ~20 s after a start from a cold disk;
- they do not fit: an arena of slots, registered once, as large as the RAM allows (~23 GiB at 32 GB, ~9 GiB at 16 GB), filled at start with the hottest experts of the shipped ranking and kept by LRU. An expert the CPU or a prompt needs that the arena does not hold is read into a slot from the disk with O_DIRECT; the GPU's expert cache is filled from the arena only. During a prompt, part of the arena is lent to the prefill's reads and restored before the answer starts, as the GPU cache lends VRAM to the compute buffers. The idea of a RAM tier of hot experts under the GPU cache comes from buun's expert store.
GGML_CUDA_MOE_ARENA_MB=0 restores the old behavior (Linux; on Windows it is still the default).
Measured here with less RAM simulated: the server in a memory cgroup of 14 or 28 GB, and the driver's registration limit lowered to the one of a 16 or 32 GB machine (MTP, 32k context, short prompt, 5 answers of 512 tokens per server):
| RAM | short prompt | after a 19k prompt | prefill of a new 19k prompt | before (page cache) |
|---|---|---|---|---|
| 64 GB | 99.1 t/s | 93.9 t/s | 1,735 t/s | same speed, ready ~3 s later after a start |
| 32 GB | 85.5 – 88.6 t/s (9 servers) | 40.6 – 49.5 t/s | 1,062 – 1,092 t/s | crash at the 19k prompt |
| 16 GB | 48.8 – 50.3 t/s (9 servers) | 30.1 – 38.1 t/s | 475 – 501 t/s | 13 – 17 t/s, crash at the 19k prompt |
The numbers this README gave before for 16 and 32 GB (90 and 89 t/s) were measured with the memory cgroup only: the driver's limit was still the one of the 64 GB machine, so every expert could be registered, which a real 16 or 32 GB machine refuses.
With MTP, Avalon generates 44% faster than Strata on a short prompt and 30% faster at 250k filled:
| engine | short prompt, no MTP / MTP | 256k context, 250k filled, no MTP / MTP | max prefill |
|---|---|---|---|
| Avalon | 66.1 / 98.4 t/s | 57.3 / 81.1 t/s | 1,854 t/s |
| Strata v0.2.9 (HIP) | 51.1 / 68.4 t/s | 48.3 / 62.6 t/s | 1,453 t/s |
Same day, same protocol, both with the tail skip on; Strata serves ISTA's IQ3_S model, the one our experts come from, with its own drafters on.
The same code builds and runs with CUDA. Measured on one rented machine, both engines one after the other: an RTX 4090 24 GB in PCIe 4.0 x16, an AMD EPYC 7443 (24 cores), MTP with 2 draft tokens (Strata: its own MTP and suffix drafters), greedy, 512 tokens (Nov 1):
| case | Avalon | Strata v0.2.9 |
|---|---|---|
| 32k context, short prompt, MTP | 141.9 t/s | 141.4 t/s |
| 32k context, short prompt, no MTP | 98.0 t/s | 101.5 t/s (suffix drafter only) |
| 32k context, after a 19k prompt, MTP / prefill | 137.7 t/s / 3,113 t/s | 130.8 t/s / 3,422 t/s |
| 256k context, 128k filled, MTP / prefill | 132.6 t/s / 3,130 t/s | 123.6 t/s / 4,312 t/s |
| 256k context, 250k filled, MTP / prefill | 123.8 t/s / 2,986 t/s | 128.5 t/s / 4,319 t/s |
VRAM peak ~23.5 GiB of 24 for both. The kernels are still tuned for RDNA3; Strata's prefill is faster on this card.
Numbers posted by people running Avalon on their own machines, with the conditions they gave; their protocols differ from the tables above.
| machine | what they measured |
|---|---|
| Radeon AI PRO R9700 32 GB (RDNA4, gfx1201), an older 6-core Xeon, 64 GB RAM, Linux | 262k context, MTP: Avalon 86.6 t/s (85 – 93), prefill 4k / 16k / 32k 1,522 / 1,906 / 1,803 t/s; Strata on the same machine 31.0 t/s (UD-IQ4_XS; 33.8 with IQ3), prefill 988 / 1,837 / 1,872 t/s |
| RX 7900 XTX 24 GB, Ryzen 9 5900X, PCIe 4.0 (X570), 64 GB DDR4-3200, Linux | 131k context, -ub 19072, GGML_SCHED_PREFETCH_EXPERTS=6: prefill 2,024 – 2,043 t/s, generation 112.9 – 122.8 t/s (median 116.5); a later version with --moe-cache-cpu-overlap 2: prefill 2,113 – 2,130 t/s |
| RX 7900 XTX 24 GB, Ryzen 7 5800X3D, 64 GB RAM, Windows 11 (ppkrol's port) | 32k context, MTP, the file cache warm: 83.6 – 92.2 t/s; the first requests after a start are slower while the 64 GB fill |
| Radeon Pro W7800 48 GB (RDNA3, 70 CUs) | one card 58 t/s without MTP, 87 with; two cards (all experts in VRAM) 46 and 70: compute-bound once the experts no longer come from RAM, not tuned for it |
The R9700 row is the first one on RDNA4: Strata's own R9700 figures come from its maintainers' machine (DDR5, PCIe 5.0, 125 – 251 GB of RAM), a different desk from this Xeon.
Speed does not come from a smaller model. Perplexity, 8 × 4,096 tokens of external texts (War and Peace, Les Misérables, SQLite and CPython sources; lower is better):
| quantization | English | French | code |
|---|---|---|---|
| Unsloth UD-Q4_K_XL (104 GB) | 1.316 | 3.581 | 1.325 |
| Unsloth UD-IQ4_XS | 1.401 | 3.841 | 1.343 |
| Avalon IQ3_KS hybrid (the model above) | 1.443 | 3.857 | 1.331 |
| ISTA GSQ-RCO IQ3_S (served by Strata above) | 1.469 | 3.932 | 1.338 |
| AtomicChat AD-4.27bpw | 1.509 | 4.073 | 1.347 |
| ISTA GSQ-RCO IQ3_XXS | 1.596 | 4.294 | 1.364 |
The hybrid keeps ISTA's experts and requantizes the up/gate tensors of 27 layers from the original BF16 weights with an importance matrix: it beats the IQ3_S model Strata runs on all three texts and comes close to Unsloth's UD-IQ4_XS. The KV cache in q8_0 / q4_0 adds 0.016 of KL divergence over f16 at 32 – 64k of context (same English text, measured on an earlier quantization of the model). The MTP draft head agrees with the model on 80.7% of the positions of a fixed text (teacher-forced); in greedy generation its acceptance follows the text: 0.55 – 0.65 on prose, up to ~0.95 on code or copied passages.
GSM8K (the first 200 problems of the test set, greedy, 4,096 tokens at most): Avalon 196 / 200 (98.0%), Strata 196 / 200 (98.0%) on ISTA's IQ3_S.
- GPU expert cache (from buun-llama-cpp), grown with the VRAM nobody else uses: compute buffers are lent to the cache between phases, the end of the KV cache past the current context too, and the reserve outside the cache is calculated (400 MiB).
- Hosted MoE: the GPU keeps each MoE block, computes the cached experts while the CPU computes the missing ones, and moves one or two missing experts to the GPU when that finishes sooner (all inside the HIP graph).
- Expert prefetch during prefill on 4 slots borrowed from the cache, cached experts read in place.
- A RAM cache of experts of Avalon's own instead of the page cache (above): all the experts loaded with O_DIRECT when they fit, else an arena of registered slots (buun's idea of a RAM tier), part of it lent to the prefill.
- Sparse attention (QSA) for long contexts: masks built on the GPU, gathered cells for generation, split kept across speculative steps.
- MTP speculative decoding with a small draft head (cut LM head over 65k frequent tokens) that shares the main model's expert cache.
- Fused GPU kernels for generation (a linear-attention layer went from 53 to 39 launches), AVX2 kernels for the experts on the CPU, IQ3_K / IQ3_KS types.
curl -fsSL https://raw.githubusercontent.com/freriadoc/Avalon/main/install.sh | bashThe installer checks the machine (Linux, AVX2, ROCm 7.x, an RDNA3 GPU, or RDNA4: reported working on a Radeon AI PRO
R9700, not measured here; 16 GB of RAM or more, ~98 GiB of disk), builds
llama-server for the detected GPU, downloads the model and its MTP head (~89 GiB, resumable), verifies every file
against its SHA-256 on Hugging Face and installs an avalon-server launcher in ~/.local/bin with the command of
Run. Settings (environment): AVALON_DIR, MODEL_DIR, BIN_DIR, AMDGPU_TARGETS, SKIP_MODEL=1 (build
only), FORCE=1; see the top of install.sh. Then:
avalon-server # OpenAI-compatible API on http://127.0.0.1:8080; extra arguments go to llama-serverBy hand, without the installer. Tested with ROCm 7.2.4 on gfx1100 (RX 7900 XT / XTX).
git clone https://github.com/freriadoc/Avalon && cd Avalon
cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1100 -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=/opt/rocm/bin/amdclang -DCMAKE_CXX_COMPILER=/opt/rocm/bin/amdclang++
cmake --build build -j$(nproc) --target llama-serverTested with CUDA 12.4 on an RTX 4090 (89; set your GPU's architecture). The installer is ROCm only for now.
git clone https://github.com/freriadoc/Avalon && cd Avalon
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc) --target llama-serverhf download freriadoc/Qwen3.8-Flash-Next-Avalon-GGUF --local-dir Qwen3.8-Flash-Next-Avalon-GGUF
cd Qwen3.8-Flash-Next-Avalon-GGUF
../Avalon/build/bin/llama-server -m Qwen3.8-Flash-Next-Avalon-IQ3KS-00001-of-00033.gguf \
-c 262144 -b 9536 -ub 9536 -fa on -ctk q8_0 -ctv q4_0 --jinja --parallel 1 \
-md MTP/mtp-Qwen3.8-Flash-Next-shared-IQ3K-IQ4NL-FR65k.gguf --spec-type draft-mtp \
--spec-draft-n-max 2 -ubd 256 -ctkd q8_0 -ctvd q8_0Nothing else to set: --fit (on by default) places the dense weights on the GPU and the experts in RAM, the expert
cache takes the rest of the VRAM, and the cache is prefilled from moe-cache-ranking.bin next to the model.
--parallel 1 matters: other slots would keep their part of the KV cache.
On NVIDIA, backend sampling is on by default: the target's argmax runs on the GPU instead of over 248k logits on the
CPU, 6% faster per step on the RTX 4090 (neutral on the RX 7900 XT, where it stays off; -bs / -no-bs force it).
Lemonade can run Avalon's llama-server in place of its own ROCm build:
start lemond with LEMONADE_LLAMACPP_ROCM_BIN=/path/to/Avalon/build/bin/llama-server (for the systemd service, in
its environment), list the model where it is with lemonade config set extra_models_dir=/path/to/models (the 33 parts
are one model; lemonade list gives its name), and give it Avalon's settings in recipe_options.json: "ctx_size": 262144, "llamacpp_backend": "rocm" and "llamacpp_args": "-b 9536 -ub 9536 -fa on -ctk q8_0 -ctv q4_0 -md /path/to/models/Qwen3.8-Flash-Next-Avalon-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-IQ3K-IQ4NL-FR65k.gguf --spec-type draft-mtp --spec-draft-n-max 2 -ubd 256 -ctkd q8_0 -ctvd q8_0". Lemonade adds the model, the context, the port,
--jinja and --parallel 1 itself and no -ngl, so --fit keeps placing the model. Not tried yet: if you do, please
open an issue with what worked or not.
GGML_MOE_CACHE_LAYER_DEVICE=1 avalon-server # or the llama-server command aboveThe second GPU becomes more expert cache, it does not split the model: the dense weights and the KV stay on the
first GPU (--fit puts them there), and the MoE layers are shared between the GPUs' expert caches in proportion to
their free VRAM. In generation, the MoE block of a layer runs on the GPU that caches it (only tens of KB of activations
cross per layer); the prefill and its expert prefetch stay on the first GPU. With one GPU the option changes nothing
(measured: same speed, same text).
It has never run on two GPUs: I have only one. If you have two RDNA3 cards, please try it and open an
issue with the server log (-lv 4), your GPUs and how they are connected (x16 + x4, x8 / x8...). Known gap: the
prefill only uses the first GPU's PCIe link, so with two cards in x8 / x8 slots it is probably slower than one card in x16
(a per-GPU prefetch would fix it). Without the option, several GPUs are untested too.
- One model. Avalon is tuned for Qwen3.8-Flash-Next; the GGUF files above use types and a layout stock llama.cpp does not have. Other models run as in llama.cpp, without the expert cache work.
- One GPU measured. Two or more: see above, experimental and untested.
- Tuned on RDNA3 with ROCm 7.2. CUDA builds and was measured on rented RTX 4090s (see NVIDIA); its kernels are not tuned for NVIDIA yet, and the installer is ROCm only. RDNA4 runs (reported on an R9700), not measured here; the chunked prompt GDN kernel is RDNA3-only.
- Windows: ported by ppkrol (HIP SDK 7.2, measured on a 7900 XTX, see above) and merged here; a Windows installer is being tested.
- Base: llama.cpp master up to Oct 31, 2026 (merged that day, a few commits left out for speed or overlap).
- llama.cpp and the ggml authors: the base.
- buun-llama-cpp (spiritbuun): Avalon started as a branch of buun and
is built on its GPU expert cache and LFU admission; buun's expert heat profile recorded the shipped
moe-cache-ranking.bin, and the expert arena in RAM follows buun's RAM tier of hot experts. Thank you. - thecodacus: the expert prefetch.
- ik_llama.cpp (Iwan Kawrakow): the IQ3_K and IQ3_KS types, the shared MTP head support.
- Strata (Niko1221): the expert ranking format and the preload idea.
- Unsloth: the MTP head we requantized.
- ISTA-DASLab, AtomicChat, drluoto / Vitronia: the quantizations and the FR-Spec token list our GGUF files are built from.
- Qwen: the model.
MIT, as llama.cpp. The model files are under the Qwen Community License 1.0.