Running High-Quality DeepSeek V4 Flash on One 96 GB GPU

I know I said I was going to go back and reconstruct the earlier history of this project. I will. This was too interesting to wait.

As I have stated before, one of the recurring problems with local inference is figuring out what you are actually testing. Is the model bad? Is the prompt bad? Did the runtime execute it poorly? Did the quantization remove something important? Did you leave too little memory for context?

Once you start squeezing genuinely large models into limited memory, all of those questions start playing a game of Twister. A model can technically load and still be a terrible system. It might be fast but stupid, intelligent but unusably slow, or excellent until the context window fills and everything falls apart. It could take 1,400 watts to run. That is a good story for another time.

I bought a stupidly expensive 96 GB RTX PRO 6000 because I wanted fewer of those compromises although with mental gymnastics, it is for work! Naturally, as will be the recurring theme, something came up that required either compromise or a One Flew Over the Cuckoo's Nest level idea.

Why DeepSeek V4 Flash was worth the trouble

DeepSeek-V4-Flash-0731 is a 159-billion-parameter mixture-of-experts model. Its dense and attention weights are FP8, while its routed expert weights start from the checkpoint's native FP4 representation. That makes it unusually compact for a model near this capability class. Because the experts start as native FP4, the source model does not suffer the same potential quality loss as taking some other model and quantizing it down to MXFP4 or NVFP4.

"Compact" is relative. This is not a model most people are going to casually load onto a 24 GB gaming card without using a very aggressive quantization that risks removing much of the reason to run it.

The interesting starting point was Kacper's vLLM-MoET project. Kacper had already done the difficult foundational work: the SM120 runtime, custom 2-bit expert execution, an FP4 recovery system, DeepGEMM kernels, and a ton of work in general.

The deployed runtime does not leave all of those expert weights entirely in FP4. Its base expert representation uses a sign-symmetric 2-bit codebook while retaining the checkpoint's scales. A separate FP4 correction tier keeps frequently used experts closer to their original representation. The result is compact enough to run on enthusiast hardware without treating intelligence as an optional feature.

For practical purposes, it preserves the model's native intelligence. I am not submitting that sentence for peer review. I care whether the model still reasons properly, follows instructions, and uses tools reliably. It does.

My RTX PRO 6000 fork of vLLM-MoET started from that foundation.

The first version worked, which was not good enough

The initial build loaded the model and ran well. That should have been the end of the story.

Unfortunately, DeepSeek had also added a new DSpark draft head. DSpark is used for speculative decoding. Instead of generating and accepting only one token at a time, a smaller draft path proposes several tokens and the main model verifies them. When the guesses are good, the model generates faster without changing the final output. Who doesn't want faster?

Leaving that performance unused was obviously unacceptable because I am a madman.

The first obstacle was the runtime path. The DSpark implementation required vLLM's Model Runner V2, while the working configuration had come through the older runner path. Codex and I worked through the Runner V2 integration and the additional startup problems until the model reached the point where DSpark could actually initialize. I added another reason Python is off my Christmas list: it is a greedy jerk that likes to hold on to memory for too long.

If you did not already catch where this was heading, it did not fit anymore.

The checkpoint contains three W2 layers for the DSpark draft model. Those layers, plus the associated graph captures and workspace, consumed several additional gigabytes of VRAM. The original model fit. The complete system I wanted did not.

Any normal person would have accepted the 30 or so tokens per second of decode and been thrilled. Not I. I was not trying to demonstrate that a checkpoint could be loaded. I wanted the model, the FP4 correction tier, speculative decoding, useful context, and enough runtime headroom to serve it reliably. I have a pathological need to make it work with all the bells and whistles.

The obvious alternatives were not attractive

I had already tested more conventional CPU and GPU split execution through lvLLM. It worked surprisingly well, but it pushed the complete inference system toward 1,300 watts. That is one way to heat an apartment in the winter. It is a less compelling plan in August. I will write about that experience too, because there are good lessons there.

Runtimes such as llama.cpp can also offload complete layers into system RAM and execute portions of the model on the CPU. That is enormously useful, especially when the goal is simply to run a model that otherwise would not fit. It was not the trade I wanted here. It would probably be slower than even non-MTP DS4Flash, and llama.cpp is not great for concurrency and my eventual agent horde, if I ever stop playing with runtimes and models.

I had a very fast GPU sitting beside more than 200 GB of system RAM. I did not want the CPUs executing whole transformer layers if I could avoid it. I wanted the GPU to continue performing the computation while selectively reading a small, predictable set of weights from host memory.

The idea was to create an escape hatch, not turn the server into a CPU inference system.

Five layers made the difference

I came across this article about the open-source GreenBoost project, and boy, was the poor developer getting beaten up on Reddit. However, it looked like a perfect fix when used minimally, rather than in the heavy way he intended. We dug through the code to find the correct functions, and we were off to the races.

My first instinct was to move some of the KV cache into host memory. That would have been ambitious, invasive, and probably a week-long project. Codex correctly suggested that I calm down. It turns out there are active projects and an RFC on this topic, so I was not crazy...this time.

Instead, we looked for large static allocations that could live outside VRAM without changing the execution structure of the model, while staying within my PCIe 4.0 x16 bandwidth as much as possible.

After many iterations, the answer was five complete target-model W2 layers.

W2 is one of the large weight planes inside the model's routed expert feed-forward path. In the final layout, target W2 layers 0 through 37 remain in GPU memory. Layers 38 through 42 live in pinned, NUMA-local host memory. The three DSpark W2 layers, 43 through 45, stay in VRAM because the speculative decoding path uses them repeatedly.

Each mapped layer is approximately 1.69 GB. Moving five of them recovered about 8.44 GB of VRAM.

It uses CUDA-mapped host memory and Unified Virtual Addressing. The runtime allocates pinned host memory, obtains a CUDA-visible device pointer, and installs that stable pointer into the existing Runner V2 W2 path.

The GPU still performs the math. When it needs one of those expert weight planes, its existing SM120 kernel reads the data directly from host RAM across PCIe.

There is no redundant full copy of those five layers in VRAM. There is no CPU replay of the layer, no upload before every inference step, and no separate GPU staging cache. The mapped host allocation is the canonical copy used by the kernel.

That sounds simple when reduced to a paragraph. Making it survive model construction, DeepGEMM warm-up, full and piecewise CUDA graph capture, DSpark capture, serving, and cleanup was less simple.

The host pages also need to be physically located on the NUMA node local to the GPU. This machine has two Xeon sockets. Accidentally placing the mapped weights behind the wrong CPU would send a substantial amount of traffic across the socket interconnect before it ever reached PCIe.

The final implementation resolves the selected GPU's PCI address, identifies its local NUMA node, places and audits the host pages, verifies the CUDA pointers, and fails startup if the expected geometry is wrong. It is narrow and deliberately boring once it starts. That is exactly what I wanted.

Where the recovered VRAM went

I did not move five layers into host memory merely to make the out-of-memory error disappear.

The recovered space allowed the final configuration to retain:

• A 6 GB FP4 correction tier with 512 hot-expert slots
• All three GPU-resident DSpark draft layers
• DSpark proposing four speculative tokens
• FP8 MLA KV cache
• A configured one-million-token admission limit
• Normal full and piecewise CUDA graph capture

The complete system now works around one 96 GB GPU, even though five of its W2 layers are directly mapped from system RAM.

At one point during qualification, sampled free VRAM dropped to 7 MB. That is not a typo. I call that getting every penny's worth. It is why I am in the sand traps and water when I golf. I have to experience everything the course offers.

Did the compromise damage the model?

This was the part that mattered most.

A clever memory layout is useless if the model becomes noticeably worse. I ran a frozen reasoning suite and a separate tool-use suite against the final DSpark-4 configuration.

The controlled suite is intentionally harder and slower than ordinary interactive use. In real work, I generally see generation much closer to 80 tokens per second, with prompt prefill around 6,000 tokens per second.

The reasoning score was excellent. Every one of the 30 tool tests selected the correct tool with the correct arguments. There were no parser failures, loops, out-of-memory errors, or runtime crashes during qualification.

That is native intelligence for the purpose I care about. Any remaining theoretical delta is substantially smaller than the difference between a model I can actually run properly and one I cannot.

All eight reasoning cases passed. The implementation review still scored the banking case at 81.52 percent because it lost points on an extremely complex idempotency problem. If you do not know what that word means, do not feel bad. I did not either, so the distinction does not affect me much.

Because I am crazy, I tested the million-token window

DeepSeek V4 Flash supports a one-million-token context window. Once the runtime reported enough KV capacity, I obviously had to try it.

Because I am crazy, I actually tested it. The needle test passed at 994,987 rendered input tokens. Prefill cratered to roughly 1,000 tokens per second, and it took 16 minutes, but it worked.

The test recovered the exact needle placed near the beginning. Afterward, the same process correctly answered an immediate follow-up, so the server was still healthy rather than merely surviving long enough to print one result.

I tried the true one million afterward, and it OOM'd while attempting to send. Close enough. I am taking the context-millionaire title.

Also, if you ever find yourself prefilling one million tokens from scratch, I am genuinely curious what the hell you are doing. If it is a continuation, use LMCache or something, for the love of God.

This is an edge case. It is not representative of normal prefill performance, and I would not describe a 16-minute time to first token as interactive. The point was to find out whether a practical near-million-token working envelope was real. It was.

Why sparse MoE changes the PCIe calculation

The immediate reaction to mapped host weights is usually correct: PCIe is dramatically slower than VRAM.

For a dense model, every generated token has to work through essentially the entire model. If a meaningful portion of those weights lives in host memory, the GPU must repeatedly reach across PCIe for all of it. Very quickly, PCIe becomes the system bottleneck.

Sparse mixture-of-experts models change that calculation. Every token is routed to only a small subset of the available experts. Most expert weights are not touched for that token.

That does not make PCIe free. The validation runs reached roughly 30 GB/s of PCIe receive traffic, and mapped-host execution costs some short-context throughput. It does mean that carefully selected expert weight planes can live in host memory without imposing the same penalty that a dense model would.

The model's sparse MLA attention is a separate advantage. It keeps the KV-cache requirements for very long contexts manageable. Sparse MoE makes selective host-mapped expert weights plausible. Sparse MLA makes the enormous context window plausible. Both matter, but they solve different problems.

This is why I think the technique has value beyond this exact model. Someone running a smaller MoE checkpoint on a smaller card may be able to recover several gigabytes for context, correction weights, or runtime overhead by mapping a few carefully chosen expert layers instead of falling back to conventional CPU execution.

That still requires runtime-specific work. This is not a universal flag that makes every GPU appear to have more VRAM. I have directly validated one checkpoint on one RTX PRO 6000. The architectural opportunity is broader than that result, but the repository does not pretend that every possible combination has already been tested. If you have an RTX PRO 6000 Blackwell, go to my GitHub and try it. The build and run instructions are there.

The direction models are moving

As I write this, Alibaba has just announced Qwen3.8-Max, a 2.4-trillion-parameter sparse MoE model that activates approximately 95 billion parameters at a time. It also supports a one-million-token context window.

That scale is not practical as a dense model. Even large enterprises would have trouble treating 2.4 trillion dense parameters as an ordinary inference workload.

Dense models are not disappearing, especially for smaller specialist workloads. At the frontier, however, sparse architectures increasingly appear to be the only economically sane way to keep scaling. That makes memory-placement techniques designed around sparse execution more interesting than they might appear if evaluated only against yesterday's dense models.

This particular implementation is not the final answer. It is a useful escape hatch that turned a model that almost fit into the complete system I actually wanted.

The model reasons well. Tool use is perfect across my suite. Normal interaction is fast. A practical near-million-token context works. The GPU is doing the computation, and the CPUs are no longer trying to turn my apartment into a sauna.

I have finally settled my runtime and model selection...for a few days.