On August 24, I published Pennyroyal, my custom SGLang runtime for Qwen3.8 on the RTX PRO 6000 Blackwell.
Since then, I have benchmarked and run the hell out of it.
Decode speed. Prefill. Long context. Concurrency. Speculative decoding. Cache persistence. Reasoning. Tools. Vision. Restart recovery.
Every time I changed something, I tried very hard to make sure I was not making one number look better while quietly breaking three other things. That has a bad habit of happening to me.
That work is necessary, but there is an obvious problem with it: I built the runtime, I built the test harness, and I know how it is supposed to behave. I am also testing it on the exact card I have been beating on through development.
This week, somebody else came back with a different kind of test.
Reddit user u/kazimirek had been running the public Pennyroyal runtime for more than a week on an RTX PRO 6000 Blackwell Max-Q, the 300 W version of the card, with Claude Code doing actual work.
His opening line was pretty much everything I wanted to hear:
Your runtime has been running a real workload for me for over a week now and it just works...
The numbers he sent me were a little ridiculous:
4.8 billion prompt tokens
55 million generated tokens
96% prefix-cache hit rate
about 200 tok/s aggregate in his scheduler logs
peaks around 520 tok/s aggregate
no server errors reported
That last one got my attention.
The cache number matters too. His agents repeatedly reuse enormous chunks of context, and the runtime can reuse the work it has already done. That is how you make billions of prompt tokens practical without processing every repeated word from scratch.
Read u/kazimirek's original report.
This system is quite different
His machine has one Max-Q card and 125 GB of system RAM. Claude Code connects through LiteLLM to Pennyroyal, with one orchestrator and up to three subagents. Contexts normally run between 150K and 236K tokens, with images mixed into the traffic.
Basically, the kind of workload that eventually finds out whether all the things you tested separately actually work together.
He was not even running the latest release.
The deployment was based on v2.1.1, using the RadixArk Qwen3.8-Flash-Next-NVFP4 model. He started with my public recipe and adjusted it for his environment: a 524K GPU cache pool, another 20 GB of cache in system memory, a 262K context limit, and up to four simultaneous requests. Native speculative decoding was enabled, and image preprocessing used the PIL path.
His scheduler-log results looked like this:
| Workload | Reported throughput |
|---|---|
| Single request | 118 tok/s median |
| Two concurrent requests | 207 tok/s aggregate median |
| Three concurrent requests | 272 tok/s aggregate median |
| Four concurrent requests | 326 tok/s aggregate median |
| Overall scheduler-log rate | ~200 tok/s aggregate |
| Peak | ~520 tok/s aggregate |
Aggregate means the combined output of the requests running at the same time. Four agents sharing the card are doing more total work, even though each one is not getting the single-request speed.
Those are pretty damn respectable numbers for the 300 W card, especially with that much context in play.
My development card is the 600 W Workstation Edition, which I run with a 450 W power ceiling. His hardware and workload are different, and that is what makes this interesting.
The runtime left my machine, somebody configured it for theirs, and it kept running.
This thing has only been public for 19 days
That is the other part of this report that caught me a little off guard.
I released jpezzulli/sglang-rtxpro6000 on August 24. As I write this on September 12, the repository has been public for all of 19 days.
The traffic since launch:
| Since launch | Total |
|---|---|
| Views | 3,960 |
| Clones | 4,473 |
| Sum of daily unique cloners | 960 |
| Sum of daily unique visitors | 1,770 |
| GitHub stars | 66 |
The daily unique counts are added across days, so repeat visitors and cloners can appear more than once.
Still, for a very specific runtime aimed at people running Qwen3.8 on a 96 GB Blackwell workstation GPU, that is considerably more interest than I expected.
The clones have actually exceeded the page views, which is funny. Apparently this is an audience that appreciates getting straight to breaking things.
More importantly, somebody is already running billions of tokens through it.
Pennyroyal is already a little different
The original public goal was fairly narrow: make Qwen3.8 work properly on a single RTX PRO 6000 Blackwell with the combinations of features I wanted to use.
Nineteen days and several releases later, the scope has grown.
Pennyroyal now supports two Qwen3.8 configurations in the same runtime:
Flash-Next NVFP4, using its native speculative decoding with FR-Spec.
27B FP8, using DFlash2 to draft tokens ahead of the main model.
Both keep the things I actually care about: large context, caching into system memory and NVMe, restart restoration, tools, reasoning, images, and multiple agents sharing the card.
Flash-Next is where most of the recent development has been happening.
By v2.4, processing a fresh prompt on my development system had reached about 14.8K tok/s at 64K context and 8.8K tok/s at roughly 490K context. A single request generated at 181.72 tok/s, while four simultaneous requests reached 446.49 tok/s combined.
Then v2.5 brought another useful jump.
An optional online FP8 path uses lower-precision computation for selected parts of the model. I was interested in the memory savings. The improvement in generation speed turned out to be considerably more interesting.
Here is the v2.5 comparison on my system, retaining the same 524K context limit and 824K GPU cache pool:
| Single-request generation | Standard path | Online FP8 | Change |
|---|---|---|---|
| Short context | 161.47 tok/s | 207.12 tok/s | +28.3% |
| 128K context | 154.70 tok/s | 195.63 tok/s | +26.5% |
| ~490K context | 149.14 tok/s | 172.64 tok/s | +15.8% |
It also freed about 3.86 GiB of GPU memory after startup. On a card I regularly pack almost to the last megabyte, I will absolutely take that.
The repository has the test details and configuration for anyone who wants to dig into the numbers.
The other major v2.5 addition is an optional NVMe path for Flash-Next's roughly 47.7 GiB PLE table, a large model component normally kept in system RAM. Pennyroyal can now prepare that table on local NVMe and read it from there instead.
That trades some speed for a substantial reduction in the RAM requirement. If you have the memory, keeping it in RAM remains the default. If you do not, there is another option.
In less than three weeks, this has acquired two model configurations, several releases, and actual choices about how to fit the workload to the machine. With my HPE role and an attempt to keep some sort of social life, sleep has been lacking and some of my other hobbies have taken a back seat.
I did not really expect that when I put the repo up.
V2.1.1 was still cooking out there
Which brings me back to u/kazimirek's Max-Q system.
While I was changing prefill paths, testing online FP8, moving a nearly 48 GiB table between RAM and NVMe, and generally finding new ways to abuse the RTX PRO 6000, he was using an older build to get work done.
He did find two integration problems: a streaming tool-parser issue that could produce a nameless tool call, and image preprocessing running out of memory when the GPU was already packed tightly.
Later Pennyroyal releases already included changes addressing both areas, which he pointed out in his report.
That is useful feedback. Someone is exercising paths I care about, finding specific problems, and checking whether newer releases address them.
He was still able to report more than a week of use, 55 million generated tokens, and no server errors in the run he described.
That is a pretty good reason to keep going.
Then another report showed up
While I was taking all of this in, u/WonderRico posted results from the updated runtime using Qwen3.8-27B FP8.
On GSM8K, a math benchmark with short prompts, he was seeing around 800 tok/s combined across four concurrent requests, with FP8 KV caching enabled.
Then he moved to Automation Bench for more agentic work and came back with 500–600 tok/s.
Different prompts, different demands on the model, and still a lot of useful output. He included the scheduler logs in his initial comment and follow-up.
My response was:
Badass - thanks buddy!
There was not much else to say at that moment.
One person has Flash-Next doing sustained work on a Max-Q. Another is trying the 27B configuration and coming back with results as he changes workloads.
I am starting to get feedback on both sides of the runtime from people who did not build it with me.
Excuse me, but that is pretty fucking cool.
Benchmarks still matter
I am definitely not going to stop benchmarking and developing this thing as my use cases require.
If I change something and single-request generation drops from 180 to 165 tok/s, I want to know. If processing a 64K prompt gets faster but a 490K prompt suddenly takes five minutes, I want to know that too.
The same applies to reasoning, tools, images, and whether the cache survives a restart. I have personally broken almost every one of those things at some point while optimizing something else. All the maintenance fixes and the movement of image processing (I can even offload image processing to the A4000 now) came from me actually using it for work and personal items.
The tests and use help me catch those mistakes. Reports like these show what happens when somebody else takes the whole thing home and drives it.
I need both.
Can I claim victory?
I wanted a reproducible SGLang runtime for a single Blackwell workstation GPU that I could actually use.
Long contexts. Caches that can spill into system memory and survive restarts. Images. Tools. Multiple agents. Enough documentation that another reasonably competent person can clone the thing without needing three months of my chat history to understand why a random patch exists.
Apparently somebody did.
u/kazimirek took Pennyroyal, adjusted it for a different RTX PRO 6000, put Claude Code on top, and ran 4.8 billion prompt tokens through it. Now u/WonderRico is putting the other model configuration through its paces too.
That is probably the most encouraging thing to happen to this project so far.
There is, however, now one problem.
That Max-Q report came from v2.1.1.
I really want to see what that card does with v2.5.
PS - if you made it this far, the image at the top was generated locally. Penny Royal is an AI from Neal Asher’s Polity series, which is where the name came from. This is a rendition of the creation of the AI in the War Factory Room 101 (also the name of the VM the harness runs on).