# DGX Spark + vLLM Deployment Playbook (LLM-ready) > **How to use this file.** Hand it to a coding/ops LLM or agent as context and > say: *"I'm deploying an open-weight model on an NVIDIA DGX Spark with vLLM. > Use this playbook to choose settings, write the launch command, and diagnose > failures."* It is written as decision rules + symptom→cause→fix tables, not > prose, so a model can act on it directly. It is hardware-specific: **NVIDIA > DGX Spark, GB10 Grace Blackwell, 128GB unified memory, compute capability > SM121 (12.1).** Some of this is time-sensitive (vLLM builds and upstream bugs > move); treat dates as provenance, re-verify before trusting a workaround. > > Source: field notes from benchmarking 30+ open-weight models on a 2-node DGX > Spark cluster. Published at betheadversary.com. All node names, IPs, and > interfaces below are placeholders — substitute your own. --- ## 0. Hardware facts that drive every decision - **The bottleneck is memory bandwidth, not compute or capacity.** A dense model streams its entire weight set per token; a sparse MoE streams only its active experts. Consequence: throughput tracks *active* parameter count and sparsity, not total size. - **Rule: prefer MoE checkpoints over dense ones**, always, on this hardware. Observed: dense models ~13–14 tok/s regardless of size (a 70B dense model hit ~2.9 tok/s — the slowest of anything tested); MoE checkpoints ~27–230 tok/s. - **Marketed "efficiency" architectures (SSM/Mamba/linear-attention hybrids) do not reliably deliver their promised throughput on this vLLM build.** Measure, don't trust the datasheet. - **~85% of the unified pool is reserved for the KV cache at `--gpu-memory-utilization 0.85` regardless of weight size.** Practical model: **one model per node at a time, full resources, tear down to swap.** Do not try to co-resident two models on one node. --- ## 1. Choose the serving image | Situation | Choice | |---|---| | Any model released after ~early 2026 | Community `vllm/vllm-openai:cu130-nightly` (default; most architecture support) | | Model needs an arch only in newer vLLM | A newer versioned image, e.g. `vllm/vllm-openai:v0.25.1-aarch64-cu129` | | Older/very standard model | NVIDIA NGC `nvcr.io/nvidia/vllm:...` is fine, but see caveats | **NGC image caveats (why it's not the default):** - Its ModelOpt integration only accepts a top-level quant_algo of exactly `FP8` or `NVFP4` — checkpoints tagged **`MIXED_PRECISION` per-layer fail to load.** - It lags on newer architectures (failed to recognize newer MoE arch names). **Warning:** a tag named `nightly` is not necessarily rebuilt nightly. Check the image's actual build date (`docker inspect`) before assuming an upstream fix is present. If a bug should be fixed upstream but isn't in your image, the stale image is a prime suspect. --- ## 2. Single-node launch template (MoE, the common case) ```bash docker rm -f 2>/dev/null || true docker run -d \ --name \ --gpus all \ --ipc=host \ -p 8000:8000 \ -v :/root/.cache/huggingface \ vllm/vllm-openai:cu130-nightly \ --model / \ --served-model-name \ --trust-remote-code \ --max-model-len 32768 \ --gpu-memory-utilization 0.85 \ --max-num-seqs 4 \ --reasoning-parser # see §4 ``` - Add `--enable-auto-tool-choice --tool-call-parser ` if you need tool calling. The correct tool-call parser is model-specific — verify with a real round-trip, don't assume (e.g. a Qwen3 model may need `qwen3_xml`, **not** `hermes`, despite `hermes` being a common default). - Short-context models: set `--max-model-len` to the model's real native context or the server won't start; then cap generation with the client's `max_tokens` so `max_tokens <= max_model_len`. --- ## 3. Quantization: if it won't load, swap the quant - **Not every quant export of a model is equally supported.** An NVFP4 export failing with an obscure loader `KeyError` (e.g. `...experts.w2_input_scale`) while the **FP8 export of the same model loads cleanly** is a known pattern. - **Rule:** on an obscure loader `KeyError`, try a different quantization of the same model **before** debugging vLLM's loader internals. - Community NVFP4 mirrors sometimes ship a stale tokenizer file — if a mirror won't import against current `transformers`, point at the model author's official repo instead. --- ## 4. The reasoning-parser trap (most common "it's broken" cause) Diagnose empty/ugly output by the *signature* first: | Symptom | Cause | Fix | |---|---|---| | `content: null`, `finish_reason: "stop"`, non-thinking model | A `--reasoning-parser` was set on a model that has no `` mode; vLLM treats the whole reply as unclosed reasoning | **Remove** `--reasoning-parser` | | Raw `...` text leaking into `content` | No parser, or wrong parser, for a thinking model | Add `--reasoning-parser deepseek_r1` (generic `` fallback that works for most models using that convention) | | `content: null`, `finish_reason: "length"` | Budget consumed mid-thought; reasoning is in the `reasoning` field | Raise `max_tokens` (2000+; reasoning models need room) | | Bundled parser file in the HF repo, still leaking | Wrong registered name | Open the parser `.py`, read its `@ReasoningParserManager.register_module(...)` decorator, use that exact name via `--reasoning-parser-plugin --reasoning-parser ` | **Always check `completion_tokens` against the budget before trusting a transcript.** At/near the cap = truncated; rerun with more room, don't score it. --- ## 5. Kernel limits on SM121 (some models simply won't run) - **MLA (Multi-head Latent Attention) and linear/lightning-attention kernels** can request more Triton shared memory per block than GB10/SM121 allows (hardware limit **101,376 bytes**). Symptoms: `triton...OutOfResources: out of resource: shared memory, Required: N, Hardware limit: 101376`, or an MLA JIT `make_shape_compatible` error on the **first** inference request, or a crash during KV-cache profiling. - **No CLI flag or `--enforce-eager` fixes this** — the block size is baked into the kernel. It needs an upstream vLLM kernel patch. Timebox and move on. - **But it is model/config-specific, not universal:** some MLA models run cleanly on the same build. **Always smoke-test one real inference request per MLA model; never blanket-ban the architecture.** - **MXFP4 (e.g. gpt-oss):** the default `MARLIN` kernel **silently produces wrong logits** on SM121 (no crash — corrupted output, often content-dependent and non-deterministic). Fix: `--moe-backend emulation` (hardware-independent reference path; correct anywhere, but ~60× slower; needs the `amd-quark` package layered into the image). Do **not** trust blog claims that a Marlin env-var flag fixes SM121 correctness — verify against your exact model/quant/hardware. --- ## 6. Cross-node (two boxes, for models too big for one) Only reach for this when a checkpoint genuinely won't fit on one node. **Topology (the #1 mistake):** - 1 GPU per node → `--tensor-parallel-size 1 --pipeline-parallel-size 2`. **Not** TP=2. TP all-reduces every layer and is brutal over a node link; PP passes activations once per stage boundary and tolerates it. **Required env (set at container launch on BOTH nodes):** ``` VLLM_HOST_IP= # unique per node; do not let it auto-detect the LAN IP GLOO_SOCKET_IFNAME= # pin to the fast inter-node NIC, not loopback NCCL_SOCKET_IFNAME= VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE=nccl # cross-node data channel VLLM_USE_RAY_V2_EXECUTOR_BACKEND=1 # see deadlock note below ``` Launch vLLM with `--distributed-executor-backend ray` on top of a real 2-node Ray cluster (`ray start --head` on one node, `ray start --address=:6379` on the other). vLLM's default multiprocessing executor uses a single-host shared-memory queue and fails across machines. **Symptom → cause → fix:** | Symptom | Cause | Fix | |---|---|---| | `Every node should have a unique IP address` | `VLLM_HOST_IP` unset; nodes auto-detect the same-looking LAN IP | Set `VLLM_HOST_IP` explicitly per node | | `Gloo connectFullMesh ... Connection refused` | Gloo tried loopback across nodes | Pin `GLOO_SOCKET_IFNAME`/`NCCL_SOCKET_IFNAME` | | Loads and serves, then **hangs mid-generation** (a few tokens, then silence, no error) | Ray Compiled Graph deadlock (assumes in-order arrival; jitter reorders) | `VLLM_USE_RAY_V2_EXECUTOR_BACKEND=1` | | `RayChannelTimeoutError` / `EngineDeadError` early in a request | Compiled-DAG channel defaulting to shm across nodes | `VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE=nccl` (+ the V2 executor above) | | OOM at load despite the math "fitting" | Transient loading overhead eats nominal headroom | Size to a **per-node PP=2 footprint ≤ ~90–95GiB**, not just ≤128GB | | `TimeoutError: RPC call to sample_tokens timed out` under sustained load | Unfixed upstream `shm_broadcast` `max_chunks=10` deadlock at **any TP>1** | Use **single-node** for sustained/high-volume workloads; cross-node is fine for short/interactive use | **Sizing heuristic:** the working cross-node deployments all sat around ~65GiB/node; the one that OOM'd was ~110GiB/node. Aim well under the wall. **Reasoning models cross-node:** they burn budget on `` before answering — give a generous completion budget (8000+) or responses get cut off mid-thought. --- ## 7. Tokenizer / detokenizer artifacts - **Symptom:** literal `Ġ` for spaces and `Ċ` for newlines in every response (e.g. `"Hello,ĠI'mĠan..."`), confirmed in the raw JSON (not a console artifact). Common on DeepSeek-Coder-tokenizer-family models. - **Cause:** a `transformers` v5 regression — `LlamaTokenizer`/`...Fast` force-installs a `Metaspace` decoder that discards the checkpoint's own byte-level (`ByteLevel`, GPT2-style) decoder. - **Fix:** download the repo's tokenizer files, change `tokenizer_config.json`'s `"tokenizer_class"` from `"LlamaTokenizer"` to `"PreTrainedTokenizerFast"`, and point `--tokenizer` at the patched local copy. `--tokenizer-mode slow` does **not** fix it. --- ## 8. Loading is slow but not stuck - When a checkpoint's size exceeds ~90% of what vLLM computes as available RAM at that moment, **auto-prefetch is disabled** and loading crawls (shard-by-shard). A ~75GB checkpoint took ~10 minutes in this state. Not an error — give it time before assuming a hang. Large checkpoints can also take 20–30 min just to download. --- ## 9. Persistence (long-lived endpoints) - Use **`systemd --user` units + `loginctl enable-linger `** so they start at boot without an interactive login and without `sudo`. - **Gotcha:** a long-running `systemd --user` manager keeps the group set it was spawned with. If the user was added to the `docker` group *after* that manager started, units fail every restart with `permission denied ... docker.sock` even though `docker` works in an interactive shell. Fix: restart the user manager (`loginctl terminate-user `; it respawns because linger is on). Not needed after a full reboot. - **Recreate the container on each start** (`docker rm -f` + `docker run`), don't `docker start` — re-running the entrypoint against stale `/tmp/ray` state leaves the unit restart-looping with no container. --- ## 10. Provenance check (before running ANY third-party image or quant) Community builds are fine — the useful hardware patches often only exist as community work. The bar is **verification, not origin.** Before running: 1. **Who published it** — established org/account with a track record, or an anonymous one-off? 2. **Is the build inspectable** — public Dockerfile/patch overlay against a pinned base, or an opaque binary? 3. **Are the weights the model author's official org** — check the HF org page directly; don't trust a lookalike repo path. 4. **Blast radius** — anything run `--privileged` or with device passthrough (RDMA/GPU) deserves more scrutiny. 5. **Is there a better official path, and is it actually better?** Sometimes "official" is an unmerged PR with its own bugs — compare on merits. --- ## Quick decision flow 1. MoE available? Prefer it. Dense only if nothing else fits the need. 2. Fits on one node? Use the §2 single-node template. Else §6 cross-node. 3. Empty/ugly output? → §4 (reasoning parser) before anything else. 4. Won't load with a weird `KeyError`? → §3 (swap the quant). 5. Crashes on first inference with a Triton shared-memory/`make_shape_compatible` error? → §5 (kernel limit; likely unfixable here — swap models). 6. `Ġ`/`Ċ` in output? → §7 (patch tokenizer class). 7. Cross-node hang or timeout? → §6 table. 8. Third-party image/quant? → §10 first.