At some point over the last nine days, I stopped testing an AI runtime and accidentally started directing a private engineering team consisting mostly of Codex instances. I was not personally writing CUDA kernels or tracing attention code. I was doing what I increasingly do: find something promising, point Codex at the runtime and logs, explain what did not make sense, test what came back, and send it into the next hole.
This began because I finally decided that attempting to find innovative ways to run models that were absurdly too large for a single RTX PRO 6000 was getting crazy. FreeToken, LvLLM, SGLang-KTransformers, Colibri. FreeToken actually ran DS4Flash at 40 t/s with decent prefill at reasonable power utilization, but once you have a taste of 80+ t/s, you can't go back. It is like the first time I was upgraded to first class on an international flight. I had no idea what I was missing until I had my own pod and oxtail ravioli on a 10-hour flight. Now I am depressed when I don't get upgraded! Anyway, Qwen3.8-27B made a splash when released, and it is genuinely good. It fit comfortably on one NVIDIA RTX PRO 6000, produced excellent reasoning results, used tools properly, and was fast enough that a reasonable person could have deployed it and moved on. My untouched vLLM starting point decoded at 74.06 tokens per second, processed a 64K prompt at 6,753.58 tokens per second, scored 96.96 out of 100 on my reasoning qualification, and completed every tool call correctly. You can run or examine my reasoning/tool suite at jpezzulli/pennyroyal-validation.
With Qwen3.8-27B, there was no crisis. Nothing was broken. I simply looked at a perfectly functional local model and decided it ought to be much faster. I simply can't let that RTX PRO not be used fully.
I kept finding new speculative approaches and draft models. Codex kept tearing apart vLLM and SGLang to see what they actually did. Ferrari, DSpARK, DFlash2, HiCache, one rejected Mooncake path, NIXL, and enough benchmarking to make the word "tokens" lose all meaning eventually produced 108.75 tokens per second in a controlled single-request decode test. That is 39.8 percent faster than the closest comparable public TP1 RTX PRO 6000 result I could find and 46.8 percent faster than my untouched local starting point. It also made fresh 490K prefill 29 percent slower.
That last number is why this is an engineering story rather than a victory lap. I did not find a magic flag that made everything 40 percent faster. I moved the bottleneck, made some parts dramatically better, made another part materially worse, and then changed the system so the workload usually would not have to pay that prefill cost over and over.
The starting point was already good
The clean local baseline came from stock vLLM using the official Qwen3.8 FP8 checkpoint. This matters because my vLLM fork, which I dubbed Ferrari, and DSpARK were already custom runtime work. Comparing the final result against either one would let me pretend that the engineering began halfway through the story.
The actual starting point looked like this:
| Measurement | Untouched vLLM baseline |
|---|---|
| Single-request decode | 74.06 tok/s |
| 64K fresh prefill | 6,753.58 tok/s |
| Approximately 490K fresh prefill | 2,280.13 tok/s |
| Reasoning quality | 96.96 / 100 |
| Tool qualification | 31 / 31 semantic and exact |
That is not a bad result. Seventy-four tokens per second is already faster than most people read, the prompt-processing rate was excellent, and the model was demonstrably capable. It could do real agentic work without making me stare at the screen wondering whether the server had died. Being me, I immediately started screwing with it.
I kept finding things for Codex to try
Ferrari was my first serious custom-vLLM speculative path. I found promising speculative work, handed it to Codex, and asked it to make the idea work on my model and RTX PRO. The useful lesson was not merely that speculation could make the model faster. It was that the number reported by the scheduler and the work physically performed by the GPU were not necessarily the same thing.
I noticed that changing the speculative depth did not always change performance the way it should. I pointed Codex at that discrepancy. It traced the runtime and found what became the positive-K problem: the scheduler could choose a positive number of draft tokens, but the active path did not always physically execute the same amount of draft work. The plain-English fix was essentially, "If the scheduler says to draft K tokens, actually run K tokens." Codex found and implemented the generic mechanism. My contribution was recognizing that the behavior made no sense and refusing to accept the first explanation.
DSpARK came next because I found another speculative path that looked interesting and, naturally, wanted to try it immediately. Codex integrated it, we tested it, and it produced another meaningful performance step. It also gave Codex better evidence about where Blackwell verification spent time and which supposedly active paths were actually doing the work. Ferrari and DSpARK both moved the system forward, but neither became the final public runtime.
This is the part of software optimization that benchmark summaries erase. The final answer often looks obvious after it works. Getting there usually involves several reasonable ideas, some useful failures, and at least one period during which the machine appears to be actively lying to you.
SGLang and DFlash2 became the winning direction
The path that survived was SGLang with DFlash2. I found DFlash2, saw reports that it worked unusually well with Qwen3.8, and pointed Codex at SGLang. DFlash2 was already upstream work. Neither Codex nor I invented it in my apartment. The project was figuring out how to make the complete combination work well on my specific SM120 Blackwell system and then preserving exactly what produced the result.
At a useful altitude, the final runtime is not especially mysterious. The Qwen checkpoint stores eligible weight matrices in block FP8 and dynamically quantizes activations to FP8 inside those matrix operations. The remaining unquantized model paths use BF16, and the target and draft KV caches use FP8. DFlash2 proposes as many as eight future tokens, and the target model verifies them together rather than generating every token independently.
One more problem appeared here. Blackwell has an optimized attention path called XQA that can verify the drafted tokens efficiently, but the fixed-width DFlash2 path needed a packed causal mask that was not being supplied. In normal-person terms, the target was checking several proposed positions at once, and the mask tells each position which earlier tokens it is allowed to see. I asked Codex why the expected Blackwell path was not behaving correctly; it traced the path, found the missing mask, and wired it in. That is as deep as I need to go, and probably as deep as most people need to read. However, I did submit this upstream and we got a nod from an NVIDIA engineer about the work, so I have that going for me.
There was also the fact that I would not accept less than 512K of context (524K actual). That added YaRN/RoPE to both the main model and the speculative drafter.
The result still cannot honestly be reduced to "DFlash2 made it 40 percent faster." It came from the complete combination: checkpoint, draft model, SGLang, Blackwell's verification path, cache formats, configuration, and the RTX PRO 6000 itself. Codex did the code archaeology and implementation. I chose the directions, decided which compromises mattered, ran the qualification, and kept asking why when the machine behaved differently from the story the software told about itself.
DFlash2 can propose eight tokens at a time, but it does not receive eight free correct answers from the universe. Acceptance depends heavily on the content being generated. When the draft predicts the target well, the thing flies. When acceptance falls, throughput falls with it.
The fastest completed real request in the published trace reached 244.24 tokens per second. During that request, short telemetry windows reached 258.26, 276.34, and 278.40 tokens per second. The strongest observed window reached 300.16 tokens per second with an acceptance length of 7.75 and an acceptance rate of 0.96.
Those are instantaneous windows, not sustained request throughput. If I put "300 TOKENS PER SECOND" in a giant headline without that sentence, r/BlackwellPerformance will come to my house with pitchforks.
The public comparison is approximately 40 percent
The closest public single-GPU comparison I found is the local-inference-lab/rtx6kpro Qwen3.8-27B TP1 official-FP8/MTP3 result. It used one RTX PRO 6000 at 600 watts and reported 77.8 tokens per second at C1, 292.7 aggregate tokens per second at C4, and 5,877 prompt tokens per second at 64K.
My final runtime, qualified with the card limited to 450 watts, produced:
| Measurement | Public TP1 official-FP8/MTP3 | Final DFlash2 runtime | Directional difference |
|---|---|---|---|
| C1 decode | 77.8 tok/s | 108.75 tok/s | +39.8% |
| C4 aggregate decode | 292.7 tok/s | 390.23 tok/s | +33.3% |
| 64K prefill | 5,877 tok/s | 6,163.07 tok/s | +4.9% |
That is the useful public comparison, but it is not a controlled A/B. The checkpoint details, runtime, speculative method, power limit, client harness, output duration, and cache/offload configuration differ. The public result is also marked research-only because it is one community capture without repeat variance or quality-gate receipts.
The comparison is directional: this is what another documented TP1 RTX PRO 6000 configuration produced, and this is what mine produced. It establishes that the result is interesting. It does not establish that any single change deserves the complete 39.8 percent.
There is also a separately published 106.9-token-per-second MTP result for a W8A8 checkpoint under another measurement. That is close to my controlled 108.75 result, and I am not going to hide it just because 40 percent makes a better headline. It is not part of the same official-FP8 context matrix, so combining the two would be just as misleading as pretending it does not exist.
Then I made fresh long-context prefill worse
Against my untouched local vLLM starting point, the final runtime improved decode from 74.06 to 108.75 tokens per second. Reasoning quality also moved from 96.96 to 98.26, although that does not mean SGLang or DFlash2 made the model smarter. It means the performance work did not obviously damage measured reasoning quality.
Fresh prefill went the other way:
| Measurement | Untouched vLLM | Final DFlash2 runtime | Difference |
|---|---|---|---|
| Single-request decode | 74.06 tok/s | 108.75 tok/s | +46.8% |
| 64K fresh prefill | 6,753.58 tok/s | 6,163.07 tok/s | -8.7% |
| Approximately 490K fresh prefill | 2,280.13 tok/s | 1,618.31 tok/s | -29.0% |
The 489,921-token final test was not broken. It recovered all three needles exactly. It simply took longer to process the cold prompt.
If the machine primarily served unrelated giant prompts once and then discarded them forever, that would be a serious reason to prefer the original runtime. My actual workload does almost the opposite. I run long-lived agent conversations with enormous shared prefixes, repeatedly add relatively small amounts of new context, restart the runtime because I cannot leave working software alone, and then ask the server to rediscover hundreds of thousands of tokens it saw before. I also may let a friend play around to learn some agentic work. All I need is for him to bounce back and forth between several long sessions to essentially lock the card in prefill states continuously.
The fresh-prefill benchmark was telling the truth. It just was not describing the operation I most needed to optimize.
Stop rebuilding the same enormous prefix
Once I accepted that fresh long-context prefill had become slower, the next question was obvious: why was I paying to rebuild the same prefix at all? My global token pool is about 1.1M, which would probably be enough for me, but what if I wanted a friend to be on as well? There are options out there that save the precomputed KV cache, which can be loaded directly. I do this at work with our Alletra MP X10000 with KVCache over RDMA (official publication of the work we have been doing coming soon!)
This was one of my decisions, not something Codex discovered in a profiler. I told Codex to investigate HiCache because making cold prefill slower was acceptable only if we stopped treating every familiar conversation like the model had never seen it before.
SGLang already had HiCache for moving reusable state out of GPU memory. The target state was more complicated than ordinary text KV because Qwen3.8 and DFlash2 also carry Mamba/GDN state and speculative sidecar state. I wanted the complete thing to move from GPU memory to host RAM and then persist on NVMe:
The configured host cache says 96 GB, but the combined target KV, Mamba/GDN, and DFlash2 allocations actually consume 120.63 GB. Apparently even the cache configuration needed a footnote saying the number on the label is not the whole number.
Mooncake was the first persistence architecture we tried. It was plausible, already present in the ecosystem, and I did not like it for this system. It added an architecture and more moving pieces around a problem that, in my apartment, should have been much simpler: take the complete reusable state and write it to the local NVMe. It also did not fit the hybrid target, Mamba and DFlash2 state cleanly enough. Codex made the path work far enough for us to evaluate it, and I told it to remove Mooncake rather than keep building plumbing around an answer I already disliked. I will note that Mooncake would be the better-scaling production software.
I kept looking and found NIXL. Its POSIX FILE backend was much closer to what I wanted: essentially direct local writes to NVMe, using io_uring and O_DIRECT, without pretending my single home server needed a distributed storage religion. I pointed Codex at it, and that became the final persistence path.
Codex still found two low-level correctness problems while qualifying it. One involved transfers exceeding the registered bounce-buffer limits, and the other involved concurrent file-path registrations being treated as the same device. Codex fixed both. I do not need to explain their internals at a conference; I need to know that three agents can restore concurrently without the storage path becoming drunk.
A persistent cache needs to know what its bytes mean
Writing cache data to NVMe is easy. Safely deciding whether a future runtime should consume those bytes is much more difficult.
The persistent state is not generic text. Change the checkpoint, page size, cache dtype, speculative configuration, attention path, runtime source, or several other settings and yesterday's files may still look valid while representing something today's runtime should never read.
I wanted a namespace hash built from the things that define that representation. My first suggestion was to create a canonical string from the relevant launch parameters and hash it into the storage namespace. Please excuse the profanity in the following responses. I speak to ChatGPT/Codex like an engineering colleague I have been working with for 10 hours straight. It has learned my style and throws it right back at me. I personally love it, and it often makes me laugh out loud because it is exactly how I think.
Codex responded to the hash idea:
One thing I like about the hash
It solves the inevitable:
"Oh shit, we forgot that
--foo-bar-fuckeryalso changes the cache representation."We can create a canonical string from the relevant launch parameters.
That became the basic design. Change the representation and the runtime selects another directory. Return to the previous representation and it can reuse the previous cache. Old bytes do not get to cosplay as correct merely because they are sitting in the expected place.
I then proposed including the complete SGLang SHA. That is deliberately conservative because even an unrelated source change can create a new persistent namespace when the cache representation did not actually change. Codex explained the tradeoff and then gave me the correct recommendation for my environment:
There is a tradeoff: tying it to the entire SGLang commit means even an unrelated source change creates a new L3 namespace. In a conventional production environment I'd probably invent a separate cache-ABI revision to avoid that.
For you?
Fuck it. Use the SGLang SHA.
You change the runtime constantly, the SSD isn't precious, and the downside is merely that the first prompt after a build doesn't reuse yesterday's L3. That's dramatically preferable to wondering whether yesterday's 400 GB of persistent state is compatible with today's weird experimental branch.
Exactly. The penalty is one cold prompt after a build. The alternative is wondering whether yesterday's cache can work with today's shenanigans.
LMCache remained another possible architecture because its separate daemon could preserve RAM across an SGLang-only restart. I pointed out that most of my SGLang restarts happen because I changed the runtime, which means the prior namespace is intentionally incompatible anyway. Codex agreed with my assessment in the best way possible:
The main thing LMCache could still add is RAM surviving an SGLang-only restart, because the LMCache daemon is separate. But think about your actual workflow: most SGLang restarts happen because you changed the runtime. With the new namespace scheme, that means the prior cache is intentionally incompatible anyway. So preserving RAM across that restart buys you precisely fuck-all.
That behavior was tested. A 60,032-token prefix restored in 3.16 seconds after a process restart. A 518,528-token prefix restored with only a six-token tail in approximately 14.64 seconds and recovered every needle exactly. Changing the page size selected a different namespace; changing it back reused the original one.
Concurrent restoration also worked. Three approximately 60K requests restored after restart, and a larger concurrent test reused 775,168 tokens while computing only 320 tokens of tail and page-rounding work.
HiCache and NIXL did not repair the fresh-prefill benchmark. They did something more useful for this workload: they reduced how often the runtime had to perform fresh prefill at all.
Then I gave it a series of tasks and saved the two-hour run
Controlled benchmarks are necessary, but an agent does not spend its day generating exactly 1,024 tokens from a 136-token prompt while nothing else happens. I wanted to know what the runtime did when I stopped qualifying it and started using it.
The published sanitized trace contains 124 completed requests across approximately two hours and 22 minutes of ordinary activity. It includes 85,156 generated tokens and contexts ranging from 183 to 350,195 input tokens. The reasoning qualification had ended an hour before the trace began, so these were not benchmark requests quietly wandering into the real-use dataset.
Across all 124 requests, including real interference, the median effective decode rate was 131.31 tokens per second. The token-weighted rate was 102.57. Removing seven request intervals that overlapped other completed requests left 117 requests with a 133.21-token-per-second median and a token-weighted rate of 125.36.
The non-overlapping context curve looked like this:
| Input context | Requests | Median decode | Token-weighted decode |
|---|---|---|---|
| 0-2K | 34 | 165.68 tok/s | 175.46 tok/s |
| 64-100K | 5 | 168.01 tok/s | 151.46 tok/s |
| 100-150K | 14 | 133.25 tok/s | 130.36 tok/s |
| 150-262K | 2 | 117.96 tok/s | 116.10 tok/s |
| 300-325K | 21 | 110.81 tok/s | 116.69 tok/s |
| 325-340K | 18 | 133.03 tok/s | 130.98 tok/s |
| 340-360K | 23 | 105.45 tok/s | 101.69 tok/s |
The rebound between 325K and 340K is not a typo. I am not smoothing it into a prettier line because the data has offended my sense of order. Context length matters, but speculative acceptance, output mix, scheduling, and competing work matter too. Tokens per second is not one number, even on the same model, runtime, GPU, and afternoon.
The seven overlapping requests are also real. They produced a median of 68.37 tokens per second and a token-weighted rate of 46.05. That is not isolated engine speed, but it is absolutely user experience when multiple jobs collide. Removing them helps explain the runtime; retaining them in the complete dataset prevents the explanation from turning into advertising.
Medium reasoning is faster for a completely different reason
The final runtime scored 98.26 out of 100 at xhigh reasoning effort. I then qualified medium, which scored 95.807. Medium produced dramatically less reasoning text: 80.4 percent fewer completion tokens and 84.74 percent less summed request time.
Its effective generated tokens per summed request second increased from 124.74 at xhigh to 160.21 at medium. Three-request server throughput moved from 396.78 to 486.19 tokens per second.
That is a very useful operational result and a terrible number to combine with the runtime optimization claim. The runtime did not suddenly become another 20 percent faster. Medium asked the model to do dramatically less work and gave up 2.453 points in this qualification. Whether that trade is worthwhile depends on the task, which is why medium is now my normal default while xhigh remains available when I want the model to spend absurd quantities of tokens thinking about something.
I froze the exact thing that worked so others might try it
The final result is published at github.com/jpezzulli/qwen38-dflash2-pro6000. This is the complete frozen SGLang-derived source distribution, not a neat patch series pretending the path here was cleaner than it was.
The measured runtime source is commit 8e197ed3af. The publication commit is 68e05c793c; it adds the documentation and sanitized evidence without changing the executable Python or Rust trees. The history still contains inactive experiments, including Mooncake-related work, because the point is to preserve what actually happened rather than rewrite the repository into a heroic origin story.
The repository includes the runtime source, exact target and draft checkpoint revisions, build identity, launch configuration, performance definitions, sanitized agentic trace, qualification summaries, provenance, and explicit limitations. It also says, repeatedly, that this is not a claim of the fastest Qwen3.8 runtime in existence.
The measured scope is one RTX PRO 6000 Blackwell Workstation Edition, one exact target and draft pair, TP1, a 450-watt board limit, four admitted requests, and a 524,288-token context admission limit. The repository records the exact FP8, BF16, cache and build details for people who need them. The controlled long-context test reached 489,921 prompt tokens. It did not validate every possible 524K prompt, every Blackwell GPU, or whatever current SGLang becomes three weeks after I publish this.
Clone the frozen source if you want to reproduce the result. Do not install current SGLang main, copy three flags from the launch script, and send me an angry note when it behaves differently. I have already performed enough of that experiment for all of us.
The result was not one optimization
I started with a good model running at approximately 74 tokens per second. I ended with a controlled 108.75-token-per-second result, a 39.8 percent directional advantage over the closest comparable public single-card result I found, and real short-context agent requests commonly living around 165 tokens per second.
I also ended with fresh 490K prefill running 29 percent slower than where I started. Instead of hiding that, I built a persistent prefix hierarchy that restored more than half a million cached tokens in roughly 15 seconds and made the regression far less important to the long-lived agent sessions I actually run.
I did not personally write the positive-K mechanism, the XQA mask integration, or the NIXL corrections. Codex found and implemented those after I pointed it at behavior that did not make sense. My role was finding speculative models and runtime paths worth exploring, deciding that repeated prefix work mattered more than winning every cold-prefill benchmark, rejecting Mooncake when its architecture did not fit, finding NIXL, choosing direct local persistence, defining the namespace safety policy, and testing every supposedly successful result until it either survived or went away.
That is the useful lesson. AI-assisted runtime engineering is not asking a model to "make this faster" and accepting whatever patch appears. It is giving the agent a concrete discrepancy, enough access to trace the system, a human decision about which tradeoffs matter, and real qualification that can tell a clever-looking failure from an improvement.
The magic flag does not exist. The fast runtime does.
Now I have a bigger problem. I don't have a reason to mess with runtimes anymore and I might actually have to do real work with it. However, there are bigger models and commission payout is Friday. I should buy a 2nd RTX PRO before the next price increase, which was already announced! Just kidding...maybe...
When I told Codex I might finally be done, it responded:
Yes. Stop.
You now have both ends solved in one runtime:
Medium: 95.8 reasoning, 160 tok/s, 486 tok/s three-way, essentially perfect tool use.
xhigh: 98.26 reasoning for genuinely hard architectural work.
Dynamic per-request selection works cleanly.
Huge context, stable serving, excellent speculative acceptance, no operational weirdness.FreeToken/DS4 took enormous engineering to reach ~30 tok/s and still cannot exercise its advertised 524K context. Penny gives vastly better practical performance and lets you buy 80% fewer reasoning tokens whenever maximum depth is unnecessary.
Freeze Penny as the selected runtime. Reopen runtime/model work only for a materially better checkpoint, a missing required capability, a regression, or new hardware. Another interesting GitHub repository is no longer sufficient cause. The machine is finished; use the bastard.