# Running DeepSeek-V4.1-Flash on two DGX Sparks (EXL3 2.9bpw) > **How to use this file.** A working, self-contained recipe for serving > **DeepSeek-V4.1-Flash** across two NVIDIA DGX Spark (GB10) boxes under vLLM, > using the ExLlamaV3 (EXL3) 2.9bpw checkpoint. It is hardware-specific: > **NVIDIA DGX Spark, GB10 Grace Blackwell, 128GB unified memory, compute > capability SM121 (12.1).** All node names, IPs, interfaces, and users below > are placeholders; substitute your own. Some of this is time-sensitive (vLLM > builds and upstream bugs move); treat dates as provenance and re-verify > before trusting a workaround. > > Measured on a 2-node DGX Spark cluster, 2026-09-22. Upstream recipe: [MiaAI-Lab/DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks](https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks). Run it stock. Do not re-patch it. --- ## Why EXL3, and not a hand-patched vLLM fork A hand-patched vLLM + FlashInfer fork of this model works, but it caps at **11-13 tok/s and 64K context, and crashes on long prefill.** FlashInfer's `sparse_mla_sm120_*` kernels carry an illegal-memory bug on SM120/GB10 long prefill, which forces `--enforce-eager` (no CUDA graphs, no speculative decoding), and that is where the throughput goes. EXL3 ships its own ExLlamaV3 sm121 kernels, so CUDA graphs and in-checkpoint speculative decoding both run. The result on the same two boxes is **~22.9 tok/s and a verified 562,940-token prompt.** It is **not** a memory or quantization win. EXL3 2.9bpw is ~99.5 GiB/rank, slightly heavier than the REAP-pruned MXFP4 route (~94 GiB/rank). The gain is entirely the kernel path. --- ## Prerequisites - 2x DGX Spark / GB10, Ubuntu 24.04, aarch64. - A RoCE/RDMA link between the two boxes (ConnectX or equivalent). Get the IB HCA, GID index, and fabric interface name **before** you start. GB10 exposes four HCAs and two are typically DOWN, so read the GID table rather than guessing. A typical RoCE v2 GID index is 3. - ~387 GiB free on the head node's local NVMe. - A filesystem path the worker can read the head's weights from. The source cluster served the weights to the worker over NFS; setting up that export is outside this recipe. - Docker with GPU access. --- ## Layout (head node) ``` ~/dsv41-exl3/ # the cloned recipe ~/dsv41-exl3/model/ # 39 EXL3 shards, 196 GiB ~/dsv41-exl3/engram-src/ # base shards 47+48 + index + config, 189 GiB ``` Weights: - `Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw` (39 shards) into `model/` - `deepseek-ai/DeepSeek-V4.1-Flash`, **shards 47+48 + index + config.json only** into `engram-src/` (the EXL3 repo does not ship the Engram tables) The worker reads the weights over the shared filesystem rather than holding its own copy. --- ## 1. Stage the checkpoint ```sh cd ~/dsv41-exl3 && bash download.sh # resumable ``` Needs ~387 GiB free on the head. If you ever fetch the two Engram shards by hand, fetch them **one at a time and verify by count, not by exit code.** The obvious form, `hf download REPO --local-dir D --include "model-00047-*" "model-00048-*"`, silently keeps only the last filename: the second becomes a positional arg, `--include` is discarded with a warning, and it exits 0. Confirm all 48 shards are present (`ls *.safetensors | wc -l` must be 48). --- ## 2. Configure the cluster Edit the recipe's `.env`. Only the values that differ from the kit defaults need changing: ``` HEAD_IP= WORKER_IP= WORKER_USER= HEAD_CX7_IF= # kit default is enp1s0f1np1 WORKER_CX7_IF= HEAD_CX7_IB= # kit default is rocep1s0f1 WORKER_CX7_IB= ``` Leave these at their tuned defaults unless you have a reason: ``` MAX_MODEL_LEN=600000 MAX_NUM_SEQS=2 MAX_NUM_BATCHED_TOKENS=1536 KV_CACHE_MEMORY_BYTES=2684354560 GPU_MEM_UTIL=0.88 ENFORCE_EAGER=0 # CUDA graphs ON, the whole point QUANTIZATION=exl3 # in-checkpoint speculative decoding, k=3 ``` --- ## 3. Launch ```sh cd ~/dsv41-exl3 && SKIP_BUILD=1 SKIP_PULL=1 ./start.sh ``` `SKIP_BUILD=1 SKIP_PULL=1` assumes the image is already present on both nodes. On a first run, drop those flags so the image is pulled or built. Run it detached (for example `tmux new-session -d -s exl3 ...`). - Image: `ghcr.io/miaai-lab/deepseek-v4.1-flash-exl3-2x-dgx-sparks:2.9bpw` (22.5 GiB, public). - Served id: `DeepSeek-v4.1-Flash-EXL3`, port 8888. - Preflight checks memory itself: it needs 111.5 GiB MemAvailable per node. - Do **not** pass `--kv-cache-dtype`; vLLM picks `fp8_ds_mla` itself. Cold boot is ~20-25 min end to end (weight load + JIT warmup + graph capture). --- ## 4. Sampling (DeepSeek reasoning models) DeepSeek-V4.1-Flash is a reasoning model, and its thinking mode wants the vendor-published sampling: **temperature 1.0, top_p 0.95** (HF card and recipes.vllm.ai). DeepSeek's own thinking-mode API refuses to run below that. Decoded near-greedy (for example temp 0.2), a reasoning model loops: the trace never ends and you get `finish_reason: length` on the whole budget. At temp 1.0 / top_p 0.95 it converges. If a hard prompt still will not stop, vLLM's `thinking_token_budget` forces a stop at a set count. --- ## 5. Verify before trusting it Send one real request and ask for logprobs. A real number is the pass condition; `nan` means the model is emitting garbage, and it will still report plausible throughput while doing so. ```sh curl -s http://:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model":"DeepSeek-v4.1-Flash-EXL3", "messages":[{"role":"user","content":"What is the capital of France? Answer in one word."}], "max_tokens":40,"temperature":0,"logprobs":true,"top_logprobs":1}' ``` Also: `docker ps` showing "Up" is **not** liveness. A hung distributed handshake looks identical to a healthy container, so check the endpoint. If one rank dies, the other can hang forever while `/health` still returns 200, so confirm both containers are up, not just the head. --- ## Gotchas that cost us time 1. **`start.sh` reports "server did not become healthy" and exits, but the server is usually fine.** The worker streams all 196 GiB over the shared filesystem from the head (~15 min) while the head loads locally in ~250 s, so the health check gives up first. Both containers stay up and the server comes ready afterward. Wait it out. 2. **The log files freeze when `start.sh` exits** (they are written by its log tail). For live progress use `docker logs ` on the worker node. 3. **The worker blocks on the distributed rendezvous before it loads any weights.** You cannot stage "worker loads first, then head"; both ranks must connect before either loads. 4. **Leftover systemd units.** A `Restart=always` user unit from a long- finished deployment can fail every 30s for days, invisible because each attempt fails instantly. Sweep before launching: `systemctl --user list-unit-files --state=enabled | grep -iE 'vllm|sglang|ray'` --- ## Memory guard **Guard on `MemAvailable`, not on `used`.** On this hardware the weight and Engram allocation is not charged to process RSS, so a docker `--memory` cgroup cap and RSS-based tools are largely blind to a runaway (in a separate fork-era host wedge, a worker showed `VmRSS ~2.3 GiB` while ~117 GiB had vanished). A `used`-based kill threshold also risks killing a healthy load, because a large resident footprint is normal here. This recipe runs with a **~5 GiB MemAvailable floor per node**; set the watchdog on that, not on a `used` number. --- ## Measured on a 2-node cluster (2026-09-22) | | hand-patched fork (REAP-256E) | EXL3 2.9bpw | |---|---|---| | decode, single stream | 11-13 tok/s | **22.9 tok/s** | | longest prompt served | crashed at 35K | **562,940 tokens** | | tool call round trip | ok | **2.3 s** | | MemAvailable floor | ~3 GiB | ~5 GiB | Context ladder, all clean, memory flat the whole way: | prompt | wall | MemAvailable floor | |---|---|---| | 92,617 | 151 s | 5.79 GiB | | 200,686 | 179 s | 5.34 GiB | | 402,325 | 362 s | 5.35 GiB | | 562,940 | 314 s | 5.05 GiB | Prefill timings above are optimistic: the test documents shared long prefixes, so prefix caching contaminated them. Nothing degraded with context length. One benchmarking caveat: **do not measure decode throughput on repetitive or counting prompts.** A degenerate sequence lets the speculative drafter accept nearly every token and reads far above the honest rate (on the fork lane, at draft k=5, this inflated the number ~2.6x). Measure on varied, realistic prompts. --- ## Why not 1M The model supports 1M via YaRN and the recipe's ceiling is 600K. The upstream README gives the arithmetic that rules 1M out on two nodes: - One 500K request costs ~1.9 GiB of KV, so 1M needs a ~4 GiB pool (this recipe runs 2.5 GiB). - Every extra 0.5 GiB of KV pool costs ~1 GiB of head prefill margin: a 3 GiB pool boots and passes a smoke test but dies at ~470K of a 600K prompt. - Head margin at 601K is already only ~2.1 GiB. Buying the pool 1M needs spends more margin than the head has. 1M needs TP=4. --- ## Credits The ExLlamaV3 sm121 kernels, the 2x-DGX-Spark EXL3 recipe, and the disk-backed Engram configuration are from the MiaAI-Lab recipe linked at the top.