Blog ·
Two NInfer results, two different questions
The generation comparison and cache-placement experiment address different behaviors. Their figures, hardware and conclusions should be read separately.
1. How quickly did the tested configuration generate output?
| Item | Baseline | NInfer-based candidate |
|---|---|---|
| Hardware | RTX 3090, 24 GiB | Same GPU |
| Workload | Matched short-summary task, model already loaded | Same task and warm-model condition |
| Client-estimated generation rate | 40.57 tokens/s | 133.19 tokens/s with MTP3 |
| Model family | Qwen3.8-27B | Qwen3.8-27B |
| Model representation | UD-Q4_K_XL GGUF | Official NInfer model artifact |
| Runtime | llama.cpp under llama-swap | NInfer-based candidate |
The recorded ratio is 133.19 / 40.57 ≈ 3.28×, rounded to 3.3×. These are short-summary generation-rate estimates, not sustained fleet throughput or coding-task completion rates.
The candidate changed model quantization/representation and the key-value cache format as well as runtime. This is a comparison between configurations; attributing the entire difference to the runtime would exceed the experiment.
The record also reports 48.03 tokens/s for the NInfer reasoning configuration without MTP3. That helps explain why the 133.19 result must name its speculative configuration. It is not a default NInfer speed guarantee.
Multi-token prediction proposes several pieces of output together for checking. How much this helps depends on the workload and acceptance of those proposals.
Reproduction and quality scope
The candidate record identifies:
- Revision:
e7b7e669c47703d2c0489e4a01c5ef4dd0f87f81 - Image index:
sha256:39bf9ea1badfc6f6d285e03b4b1736a709515364243bcb08c3617517ce3ddcba
Workflow linkage was recorded, but builder identity was absent from the tested artifact’s attestation. The tested GGUF could not be imported directly, so the comparison used the official NInfer artifact.
Short usefulness, tool-call and JSON probes passed in the recorded variants. These are not the complete quality evaluation: separate 27B follow-up checks are recorded in the experiment log. This speed comparison does not establish long-context or universal quality equivalence.
2. Could a long conversation retain its cached state after overflow?
| Item | Reported setup or result |
|---|---|
| Hardware/model | One RTX 5090; Qwen3.8-27B NVFP4 |
| Context and lanes | 240K-token context; two lanes; MTP speculative decoding |
| Host-memory cache | Approximately 50 GiB; reproduction uses --host-context-mib 51497 |
| Stock base | 68c54356 |
| Fixed bases | 68c54356 plus fix; 81c8ce09 plus fix |
| Workload | Seeded synthetic prompts around 200K tokens; 16-token replies |
| Stock | 47.5 s, mean of 47.4 and 47.6 s; 0 of 199,965 prompt tokens cached |
| Fixed | 0.42 s, one reported latest-build run; 199,906 of 199,913 cached |
| Metric | Client wall-clock duration of a complete non-streaming request |
Seven conversations filled the host tier. Two more arrived, then the first newcomer was requested again. Stock had lost the saved state; the fixed build reused almost all of the prompt computation.
The runs were unpaired and their prompt lengths differed. Earlier fixed-build timings are not repeated measurements of the latest build. These durations are not time to first token.
Why placement failed
The engine used total free bytes to approve a move into an arena requiring contiguous allocation ranges. In one trace, 35.0 MB was free for a 33.8 MB move, but the available gaps were too small.
The placement check simulates the incoming allocations and the ranges candidate evictions would return. It selects useful evictions and avoids evicting state when no admissible selection would make placement possible.
Capture and pause changes are preventive; the reported investigation did not reproduce failures in those paths.
Fragmentation remains. In a separate cycling test, both builds served cached turns below 0.5 seconds before overflow. The fixed build later retained a newcomer by evicting two earlier conversations.
This does not demonstrate strict recency-based eviction or performance equivalence outside the tested conditions.
What these results establish
The RTX 3090 comparison provides evidence that the tested NInfer-based configuration can generate substantially faster on that workload.
The RTX 5090 experiment provides evidence that a placement-aware cache decision can avoid repeat prompt computation in the reported overflow scenario.
They answer different questions. The cache fix did not produce the 3.3× generation result, and the two gains must not be multiplied. Neither experiment measures Saylek network performance.
Recorded configuration excerpt
Transcribed from Saylek’s October 13, 2026 Linux Qwen runtime migration record for the RTX 3090 comparison above. Host names, network addresses, account and session identifiers and internal file paths are removed. Nothing here was remeasured.
| Item | Baseline | NInfer-based candidate |
|---|---|---|
| GPU and driver | RTX 3090, 24 GiB; NVIDIA driver 610.57.04 | Same GPU and driver |
| Operating system | Debian 13, Linux kernel 6.12 | Same host |
| Model | Qwen3.8-27B, UD-Q4_K_XL GGUF | Official qwen3_8_27b.ninfer artifact, 20.4 GB; the GGUF importer rejected 96 Q4_K tensors |
| Runtime | llama.cpp under llama-swap | NInfer-based image |
| Image | sha256:5f245844a3244287bf69d4964670a021acf6f336467ae1feaa0024b4665c018d | Index sha256:39bf9ea1badfc6f6d285e03b4b1736a709515364243bcb08c3617517ce3ddcba, revision label e7b7e669c47703d2c0489e4a01c5ef4dd0f87f81 |
| Context | 262,144 tokens | 262,144 tokens configured; fit in memory |
| Key-value cache | q4_0 K/V | NInfer format; differs from baseline |
| Other settings | Full GPU layers, flash attention, one slot, medium reasoning template | Reasoning configuration; MTP3 speculative decoding for the 133.19 row |
| Variant | Generation rate | First token | First answer content | GPU memory |
|---|---|---|---|---|
| llama.cpp, reasoning lane | 40.57 tokens/s | 0.25 s | 6.78 s | 22,576 MiB |
| llama.cpp, fast lane | 41.16 tokens/s | 0.20 s | not recorded | 22,578 MiB |
| NInfer, reasoning | 48.03 tokens/s | 0.195 s | 4.98 s | 21,554 MiB |
| NInfer, reasoning with MTP3 | 133.19 tokens/s | 0.209 s | 2.04 s | 22,654 MiB |
| NInfer, fast | 48.60 tokens/s | 0.198 s | not recorded | 21,554 MiB |
Generation rate is client-estimated: completion tokens divided by the time after the first streamed delta. Requests went directly to the local model server, not through Saylek. All variants passed short synthetic usefulness, tool-call and JSON probes with no recorded errors.
In the separate 27B follow-up coding checks, the reasoning configuration scored 61 of 62 hidden checks on both baseline and candidate; the fast configuration scored 60 of 62 on baseline and 61 of 62 on candidate. Quantization differs, so these checks do not isolate the runtime either. All of these results cover the recorded 27B configuration only.
Sources and attribution
- Generation comparison: Saylek’s October 13 Linux Qwen runtime migration record and its baseline/candidate evidence; see the recorded configuration excerpt.
- Cache reproduction and proposed change: Neroued/ninfer issue 378.
- Related arena proposal: issue 379.
- Common parsing change referenced by the cache experiment: issue 244.
- Upstream: Neroued/ninfer.
- Community hardware extension: iamwavecut/ninfer-all, derived through Don-Chad/ninfer-3090.
- Saylek’s public fork: Saylek-ai/ninfer.
Applicable licenses, copyright notices and contributor attribution remain with the software. This companion does not assert a current upstream merge or universal artifact compatibility.