More AI from the GPU you already own

You may already own a computer capable of running an AI assistant. The question is whether it can do useful work at a speed you want to use.

Many people access AI through a hosted service. Running a model on hardware you or your organization owns is another option. It gives you direct control over that machine and its software, but it also makes software choices consequential. The same GPU can deliver very different experiences depending on how the model is run.

That is why NInfer matters to us. It is an open-source inference engine: the software that executes a model and generates its answers. NInfer and its community forks pursue fast inference on NVIDIA GPUs, including hardware people already own.

In one of our RTX 3090 comparisons, the estimated generation rate was approximately 133 tokens per second with a NInfer-based configuration, against 41 on our llama.cpp baseline: roughly 3.3×.

Tokens are pieces of text; more per second means faster output during generation. For someone waiting as an assistant writes an answer, that can shorten this part of the wait.

This was a short-summary test with the model already loaded. Several software settings also differed, so the result compares two setups on the same GPU, not just two engines. It does not establish three times the completed work. The companion records those differences.

Why the promise is worth pursuing

For someone working on owned hardware, a gain like that is a reason to investigate before assuming a faster experience requires another GPU.

One technique behind NInfer’s performance is multi-token prediction: generating and checking several candidate pieces of text together. Its benefit depends on how often those proposals are accepted and the work being performed. Community forks also extend the engine to additional GPU generations.

The appeal is practical: potentially more useful performance from existing equipment.

Saylek is built around making owned AI compute useful across people, machines and locations. Improvements in the software running on those machines are directly relevant to that purpose. NInfer is upstream technology we evaluate and contribute to; Saylek did not originate the engine.

A fast answer is only one part of the experience

People switch projects, return to earlier conversations and ask assistants to use tools. A runtime that generates quickly must also handle those transitions well.

A separate NInfer investigation examined one of them: returning to a long conversation after other conversations had filled the memory used for saved computation.

An engine can retain work it has already done processing a conversation. That cache makes later requests cheaper to process. Losing it does not erase the conversation’s text; it makes the engine process that text again.

In a synthetic cache-overflow test on one RTX 5090, the reported time to complete a short reply was 47.5 seconds without a placement fix and 0.42 seconds with it.

The stock time is a two-run mean; the fixed time is one run. The runs were unpaired, used slightly different prompts of about 200,000 tokens, and requested 16-token replies.

This is a different result from the 3.3× generation comparison. It measures avoided repeat computation after cache loss, on different hardware. The cache fix did not cause the 3.3× gain.

The engineering behind the wait

The engine had checked whether enough memory was free. But the available memory was split into gaps too small for the allocations it needed to place.

The fix checks whether the state will actually fit. It can then release other retained state when doing so creates usable space.

That improves the decision; it does not create unlimited capacity. In a separate test, keeping a new conversation meant evicting two older ones. Explaining that cost is part of understanding whether a change suits a workload.

What to take from this

If you are evaluating AI on your own hardware, test three things:

• How quickly it generates.
• What happens when you return to earlier work.
• Whether it completes the tasks you need correctly.

Our measurements make NInfer worth exploring. They do not establish a universal replacement for every runtime or equivalent quality across every model and configuration.

For Saylek, the opportunity is to make capable machines more useful as people’s work changes. Evaluating promising runtimes, diagnosing specific failures and contributing improvements are concrete steps toward that goal.

To explore the product, start with the Saylek quickstart. For the configurations, measurements and cache reproduction, continue to the technical companion.