Beyond DeepInfer: DeepSeek-V4.1-Flash on a 12 GB GPU

On 23 October 2026, ISTA-DASLab published three GGUF builds of DeepSeek-V4.1-Flash-1017 that its model card says run on a single 12 GB GPU with the Strata runner. Two days later, llama.cpp merged support for the V4.1-Flash architecture. For a model that DeepInfer serves from clusters of SM100 GPUs, that sounds like a contradiction. It is not. It is a different product, with a different bill and a different job.

The short answer

If you want to try V4.1-Flash on one desktop, this is a documented path that does not start with a cluster. Budget 12 GB of VRAM, 46–64 GB of system RAM, at least 275 GB of free NVMe space, and the patience for roughly 4–11 tokens per second on the cards the project tested.

If you need to serve users, nothing here replaces DeepInfer. DeepInfer is still DeepSeek's offline cluster engine for SM100 and Ascend 950, still limited to one model, and still missing the KV cache system, prefill/decode disaggregation, and replica load balancing that a production service needs. A desktop answering one request at 5 tokens per second is not a smaller version of a 64+64-GPU deployment. It is a different product.

If you want the llama.cpp ecosystem, the architecture support is now upstream, but that is not the same as this particular GGUF package loading in mainline llama.cpp. The package keeps its Engram table in a separate 110.6 GB file; upstream's converter writes Engram into the model GGUF. Check the layout before you plan a 250 GB download.

The one sentence to keep: a 763B-parameter model did not shrink to 12 GB. It was quantized to 1.75–2.25 bits per weight on the backbone, and the runner streams the rest from RAM and SSD. The 12 GB number is the GPU floor, not the total memory budget.

What shipped this week

The local path did not arrive in one release. It arrived as a cluster of releases across four days.

Dates are from the Hugging Face and GitHub APIs. Project release notes may use local dates.

That sequence matters. The quantization made the model small enough to be interesting; the runner made it usable without a build chain; llama.cpp made it part of one of the largest local-inference ecosystems. Each piece is necessary, and none of them is sufficient on its own.

What GSQ-RCO actually changes

GSQ-RCO is not a new model. It is a quantization method applied to the released V4.1-Flash-1017 checkpoint. GSQ handles the low-bit scalar quantization of each tensor; RCO assigns a quantization type to every tensor under a total size budget. The result is a standard GGUF file, not a custom format that only one runner can read.

The important word is per-tensor. A uniform 2-bit quantization applies one type everywhere. This release searches across types: the dense tensors (attention, shared experts, embeddings, output head) choose among Q5_K, Q6_K, and Q8_0, while the routed-expert matrices choose among types from IQ1_S to IQ3_S. The search spends most of its freedom on the routed experts because they dominate the parameter count.

The Strata figure is higher because it adds the roughly 8 GB DSpark drafter. The Engram file is shared by all three builds; download it once.

The Engram table is the part people miss. It holds 196.6B parameters at IQ4_NL (4.50 bits per weight), which is 110.60 GB of lookup tables. It is not a matrix-multiply weight. It is read sparsely, one row per token, so it can stay memory-mapped on disk instead of occupying VRAM or RAM. That is why the package can advertise a 12 GB GPU at all — and why an NVMe drive is part of the requirement, not a nice-to-have.

The base model is a sparse mixture-of-experts model: 763.2B parameters, 384 routed experts per layer, six active per token, and a 1M-token context window. The quantization card describes the released weights as FP8 dense and FP4 routed. The 2-bit builds are a separate, much smaller operating point, not a lossless copy.

The 12 GB claim, unpacked

“Runs on a 12 GB GPU” is technically accurate and practically incomplete. The model card says the backbone does not need to fit in VRAM: with Strata, the rest of the model is held in system memory. The runner keeps hot experts in RAM and streams the rest, while the Engram table stays on disk. The GPU is one part of the memory hierarchy, not the whole thing.

That table is the real buying guide. If you have a 12 GB gaming card and 16 GB of system RAM, this is not your path. If you have a 12 GB card, 64 GB of RAM, and a fast 2 TB NVMe drive, you are in the configuration the project actually tested. The difference between those two machines is larger than the difference between the three quantization builds.

There is also a versioning detail worth pinning: the speed figures come from Strata's own release notes and model tables, not from ISTA-DASLab. Strata 0.2.6 says machines that can hold all CPU-side experts in RAM with 8 GiB to spare now copy them in at load and skip the demand store. That improves speed on big-RAM machines; it does not change the 46 GB floor.

One more setting changes the comparison: Strata 0.2.5 added a Faster mode that uses five of the model's six experts per token, and it is the default on graphics cards with 16 GB of memory or less. The published quality numbers do not say whether Faster mode was on. If you run the default on a 12 GB card, check that before comparing your output with the benchmarks.

Two quality stories that do not say the same thing

ISTA-DASLab reports a clean result: the 2.00-bit build matches the base model on all three benchmarks, reaching 100.0% of the base model's task average. That is a strong headline. It is also narrower than it looks.

ISTA-DASLab measurements at reasoning effort 50. Author-reported; no independent replication. AIME25 is saturated at 100.00 for every row, so it cannot separate the builds.

Read the columns, not just the average. AIME25 is 100.00 for the base model and every quantized build, which means it is not discriminating at this difficulty. The real signal is in GPQA-Diamond (88.38 for IQ1_M vs 90.91 for the base) and LiveCodeBench v6 (88.57 vs 89.71). The 2.00-bit build matches both; the 1.75-bit build loses about 1.2 points of task average; the 2.25-bit build loses about 0.2 points. Those are small differences on a three-benchmark average, and they are the kind of differences that can disappear or grow on your own workload.

Strata tells a second story. On a fixed set of 4,995 positions, it measures perplexity 15.5% higher than the original for the 2.00-bit build, 7.3% higher for 2.25, and 21% higher for 1.75. It also reports code and agent-text degradation: 8.6% and 5.3% for 2.00, 3.2% and 2.2% for 2.25, 8.9% and 10.2% for 1.75. The 1.75 build is especially weak on plain English prose (+53% perplexity) and documentation (+24%).

Both sets of numbers can be true at the same time. A three-task benchmark average is coarse and can saturate; perplexity and KL divergence see distribution shift that a benchmark average hides. “100.0% recovery” means the 2.00-bit build matched the base on those three tasks at that reasoning effort. It does not mean the quantized model is indistinguishable from the original on every kind of text. If your work is long-form writing, multilingual content, or unusual tool-call traces, the task average is not enough evidence.

llama.cpp support landed. That is not the same as this package running.

On 25 October, llama.cpp merged PR #31086, “model: DSV4.1 flash.” The PR adds the deepseek41 architecture, a converter, the DSpark draft model, the vision projector, and DSV4 KV-cache extensions. A follow-up PR, #31137, merged on 26 October with review fixes. That is real upstream support, and it is the moment V4.1-Flash entered upstream llama.cpp.

But “llama.cpp supports V4.1-Flash” and “the ISTA-DASLab GGUFs load in mainline llama.cpp” are two different checkboxes. The ISTA package ships the Engram table as a separate 110.60 GB file. Upstream's converter writes the Engram tables into the model GGUF as one per-layer token-embedding tensor, and the runtime loads them from there. Those are different layouts.

The model card, as checked on 26 October, still says: “As of October 23, 2026, llama.cpp does not support DeepSeek-V4.1, so these GGUFs do not run in llama.cpp yet. They are supported in Strata.” Its llama.cpp commands are written as a future step, after support arrives. That sentence may age quickly — the architecture support is already merged — but the package itself has not documented a mainline llama.cpp build that loads its separate Engram file.

So the practical advice is boring and important: if you want llama.cpp, start from a GGUF that embeds the Engram table, or convert the HF checkpoint with the merged converter. If you want the ISTA 2-bit builds, use Strata until the model card documents an upstream path. Do not download 250 GB on the assumption that “support merged” means “this package runs.”

One more caveat: the merged PR adds two attention paths, an optimized sparse gather by default and a dense reference path behind LLAMA_DSV41_SPARSE=0. The dense path is a correctness reference, not a speed recommendation. The upstream support is also days old; follow-up fixes are still landing. Treat it as fresh code, not a settled runtime. For the broader scope comparison between the cluster engine and the local runtime, see our DeepInfer vs llama.cpp page.

Where DeepInfer still wins

DeepInfer is an offline inference engine, not a hosted API or a model; it supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, and it serves one model, DeepSeek-V4.1-Flash. It also leaves out the KV cache system, VE/prefill/decode disaggregation, and load balancing across data-parallel replicas. Those are not minor omissions. They are the difference between an engine and a service.

DeepSeek says the smallest efficient production unit is 64 prefill GPUs plus 64 decode GPUs. That number is the clearest statement of what DeepInfer is for. A desktop running 5 tokens per second can be useful for a private draft, a prompt experiment, or a tool-call prototype. It cannot absorb a multi-user workload with a latency SLO, and it cannot amortize a 110 GB Engram table across hundreds of concurrent sequences.

It is also not a fair speed comparison. DeepInfer's published decode chart uses 64 SM100 GPUs at EP64DP64, a 131,072-token context, and a 75% draft-acceptance assumption. A single RTX 5070 at 8K context is a different experiment. There is no public head-to-head between the two, and this note does not invent one.

The useful split is workload-shaped. Use the local stack when the bottleneck is access: you want the model on a machine you control, with no per-token bill, and you can tolerate minutes for a long answer. Use DeepInfer when the bottleneck is throughput: many users, cache reuse, long context, and a serving layer you are prepared to build around the missing pieces. Use llama.cpp when the bottleneck is portability: you want one runtime across laptops, desktops, and small servers, and you are willing to track a fast-moving upstream.

How to check the claims yourself

The release notes are unusually detailed, which makes them testable. If you are about to spend a day downloading 250 GB, spend ten minutes pinning these first.

If you only do one thing, do number two. The most expensive mistake in this release is downloading the right model for the wrong runtime.

FAQ

Sources

• ISTA-DASLab, “DeepSeek-V4.1-Flash-1017 · GSQ-RCO GGUFs” — builds, sizes, benchmarks, quantization procedure, runtime note. Checked 26 October 2026.
• IST-DASLab/GSQ and IST-DASLab/RCO — reference code, Apache-2.0.
• Strata and its DeepSeek V4.1 documentation — MIT license, requirements, speed and quality measurements.
• Strata releases — v0.2.4 (24 Oct), v0.2.5 and v0.2.6 (25 Oct).
• llama.cpp PR #31086 and PR #31137 — architecture support and follow-up fixes.
• llama.cpp conversion/deepseek41.py — Engram handling in the upstream converter.
• DeepInfer repository and deployment Q&A — scope, hardware, missing components, 64+64 GPU unit.
• DeepSeek-V4.1-Flash-1017 model card — 763.2B parameters, 384 experts, 1M context, MIT weights.

Access date for all sources: 2026-10-26. Vendor and project figures are labeled in the text. We found no independent benchmark of these builds.