Meet Hanoi, our ultra-fast inference engine

The lore

On Monday at 4:13 PM, we typed this prompt into Claude Code:

Host Gemma 4 26B and write the inference engine and optimization code from first principles in C++. Do not use vLLM. Optimize for time-to-first-token and not tokens-per-second. Make no mistakes.

Well, it was not exactly this, but you get the drift. The exact prompt is in the footnotes.

What happened over the next 48 hours was genuinely mind-boggling to us.

By Tuesday afternoon, a from-scratch inference engine — our own tokenizer, weight format, CUDA kernels, scheduler, and HTTP server — was serving 400 concurrent voice calls at 71 ms median TTFT, against vLLM's 105 ms, on the exact same GPU with the exact same checkpoint and benchmark flags.

We did not touch a line of it by hand.

Why did we do it?

Our customer builds voice agents. The inference metric they care about is not how quickly token number 50 arrives. It is how long the line stays silent before token number one does.

The text-to-speech model consumes roughly 3.3 tokens per second. That is the floor we need to beat on throughput — anything faster does not affect the agent's performance.

We wanted to see how much we could beat vLLM by if we specialized everything for one model, Gemma 4 26B-A4B; one GPU, a single NVIDIA L40S; and one workload: voice calls with roughly 2K-token prompts, short responses, and substantial prefix reuse between turns.

What did the agent build?

Fable generated roughly 36,000 lines of C++ and CUDA across 83 files.

It did not use vLLM, TensorRT-LLM, SGLang, or FlashInfer, but it did use cuBLASLt and CUTLASS as low-level matrix-multiplication primitives. It wrote the model execution, attention, KV management, scheduling, tokenization, and serving layers around them.

To make sure inference optimization was not trading off model quality, we tested against Hugging Face's greedy-first-token output. Hanoi passed on 20 of 20 prompts and matched 16 of 20 complete 64-token rollouts, with 94.7% token-level agreement.

Then we ran the same voice-call simulator against both engines:

How does the magic work?

The first version was terrible. At 400 calls, its p95 TTFT was 64,344 ms. vLLM took 188 ms. Then the iterations came, and the result speaks for itself — our company is called Autoloops for a reason.

So how did we beat vLLM at its own game?

1. Leverage the model architecture. We exploit Gemma's sliding-window attention and vLLM does not — that is the biggest edge. Twenty-five of Gemma's 30 layers use sliding-window attention with a 1,024-token window. For those layers, Hanoi uses a different KV cache, so the attention step reads 40% less KV data.
2. Sacrifice decode throughput for better TTFT. vLLM's scheduler shares each GPU step between decode batches and a chunked-prefill budget because it is built to maximize aggregate throughput for arbitrary workloads. Hanoi loads incoming prefills immediately and keeps decode lean. Fast decode does not matter for a voice bot speaking at human speed — a trade-off we can make but vLLM cannot.
3. Remove framework-induced overhead. One C++ process, with no Python scheduler loop between GPU steps. The future of inference engineering is bespoke.

When can I get this?

If you are an enterprise building agents for yourself or your customers and hear that they are not fast enough, contact us and we will optimize the hell out of your model.

We are rolling out serverless APIs for individual models based on demand, so check out our website for updates.

Who is behind this?

If you read this far, you might as well reach out and talk to us at Autoloops — your friendly neighborhood inference optimization platform. @LatentKush, the magician behind this, is a Conductor user. The inference engine got the name Hanoi because the Conductor session was called Hanoi.

Footnotes

1. Exact start-of-session prompt:

   Lets try and host Gemma 4 26B from 1st principles. We will not use any vLLM inference backend. We will optimize inference only for this particular model. No other model, just this one.

   Go through what architecture this model is, and what we can do to actually improve its speed to be lightning fast on this hardware, then we can take it from there. Explain the architecture to me, what all will you write, in what language, ideally we should be using something like C++ to write these kernels and make it extremely fast and ensure that we are optimising for time to first token. And you need to just tweak it in such a way that the response comes lightning fast. That is what we're trying to do. So we need to think about it from first principles on how this inference engine should look like. We are not going to derive anything from vLLM or anything else and like build our own inference engine and then maybe we can have a library of items that we can reuse for other optimisations next time.

2. Benchmark methodology: N simulated phone calls, eight caller turns each, with growing transcripts and realistic turn cadence: reply playout at roughly 150 wpm, think time, caller speech, and 800 ms endpointing. Mean prompt length was approximately 1,980 tokens; replies were capped at 64 tokens; the prefix cache was reset before each level; and calls were ramped in over 30 seconds. Both engines ran the same RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic checkpoint on the same g6e.4xlarge, one L40S 48 GB GPU, with the same simulator flags. The vLLM comparison used our tuned configuration — FP8 KV cache and a prefill cap of 8,192 — not an untuned default.
3. Multiple AI models contributed to the article, but the author reviewed every word.
4. The reason for naming the inference engine Hanoi is in the Who is behind this section.