# Running GLM-5.3-Flash on two DGX Sparks > **How to use this file.** A working, self-contained recipe for serving > **GLM-5.3-Flash** (a 320B-total / 18B-active MoE with 1M native context) > across two NVIDIA DGX Spark (GB10) boxes under vLLM. The primary path is an > NVFP4 build on a pre-patched sm121 image; an EXL3 alternative follows. It is > hardware-specific: **NVIDIA DGX Spark, GB10 Grace Blackwell, 128GB unified > memory, compute capability SM121 (12.1).** All node names, IPs, interfaces, > and users below are placeholders; substitute your own. Some of this is > time-sensitive; re-verify image tags and upstream bugs before trusting a > workaround. > > Measured on a 2-node DGX Spark cluster. --- ## Prerequisites - 2x DGX Spark, each with local NVMe (no shared filesystem assumed for the NVFP4 path). - A direct RDMA-capable interconnect (ConnectX or equivalent), confirmed up and addressed on both sides. - Docker with GPU support, IPC lock, and InfiniBand device access. - The full BF16/FP8 checkpoint (~320 GiB) does not fit two boxes, so GLM-5.3-Flash is served quantized. It runs **TP=2** on this cluster (the architecture does not pipeline-parallel on this stack). Both nodes are fully committed to one serve; nothing else runs during it. --- ## Primary recipe: NVFP4 ### 1. Pick the checkpoint Use **`RedHatAI/GLM-5.3-Flash-NVFP4`**, not `LibertAIDAI/GLM-5.3-Flash-NVFP4`. The LibertAIDAI ModelOpt-quantized build has a confirmed, open vLLM bug ([#54150](https://github.com/vllm-project/vllm/issues/54150)) that silently corrupts token IDs, invisible in English prose and only visible in non-English text or tool-call output. RedHatAI's compressed-tensors NVFP4 build is the bug report's own confirmed-clean control. `generation_config.json` for this checkpoint: `temperature: 1.0`, no `top_p`/`top_k`. Do not carry sampling values over from other GLM checkpoints; fetch this file fresh for whichever build you deploy. ### 2. Pull the image A pre-built image with the GB10/SM121 patches already applied exists, so no manual kernel patching is needed: ```bash docker pull ghcr.io/tonyd2wild/vllm-glm53-flash:sm121-v8 ``` Run this on **both** nodes. ### 3. Download the checkpoint to both nodes The launch script expects the checkpoint already on local disk (it does not let vLLM pull from HF at container startup): ```bash # on each node independently, no shared storage assumed huggingface-cli download RedHatAI/GLM-5.3-Flash-NVFP4 \ --local-dir /var/tmp/models/GLM-5.3-Flash-NVFP4-redhat ``` ~185 GB per node. A resumable `curl -C -` loop over the repo files works if `huggingface-cli` is not available. ### 4. Confirm your network Identify the interface and subnet for your node-to-node RDMA link (for example `ip -o addr show `), and which node will be rank 0 (head). ### 5. Launch, worker first then head **Order matters:** start the worker, wait ~20s, then the head. The reverse is more likely to leave the head hung on an incomplete handshake if anything goes wrong. The worker's command needs one extra flag, **`--headless`**, that the head's command must **not** have. Without it the worker starts a full server instead of a pure follower, crashes with `AssertionError: collective_rpc should not be called on follower node`, and the head hangs forever waiting on a peer that already died. - **Worker command:** everything below, PLUS `--headless` as the very last argument. - **Head command:** everything below, exactly as shown, nothing added. ```bash IMAGE="ghcr.io/tonyd2wild/vllm-glm53-flash:sm121-v8" MODEL_PATH=/var/tmp/models/GLM-5.3-Flash-NVFP4-redhat CACHE_PATH=/var/tmp/cache # any local writable dir; used for HF_HOME only docker run --gpus all -d \ --name vllm_glm53 --restart no \ --network host --ipc host --shm-size 32g \ --ulimit memlock=-1:-1 --cap-add IPC_LOCK \ --device /dev/infiniband:/dev/infiniband \ -v "$MODEL_PATH:$MODEL_PATH:ro" \ -v "$CACHE_PATH:/cache" \ -e VLLM_HOST_IP= \ -e HF_HOME=/cache/huggingface \ -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \ -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \ -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ -e TORCH_CUDA_ARCH_LIST=12.1a -e FLASHINFER_CUDA_ARCH_LIST=12.1a \ -e FLASHINFER_DISABLE_VERSION_CHECK=1 \ -e NCCL_NET=IB -e NCCL_IB_DISABLE=0 \ -e NCCL_IB_HCA= -e NCCL_IB_GID_INDEX=3 \ -e NCCL_IB_ROCE_VERSION_NUM=2 -e NCCL_IB_ADDR_FAMILY=AF_INET \ -e NCCL_IB_ADDR_RANGE= \ -e NCCL_SOCKET_IFNAME= -e GLOO_SOCKET_IFNAME= \ -e TP_SOCKET_IFNAME= -e MN_IF_NAME= \ -e NCCL_NVLS_ENABLE=0 -e NCCL_CROSS_NIC=0 -e NCCL_IB_MERGE_NICS=0 \ -e NCCL_CUMEM_ENABLE=0 -e NCCL_IGNORE_CPU_AFFINITY=1 -e NCCL_DEBUG=WARN \ -e TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \ "$IMAGE" \ "$MODEL_PATH" \ --served-model-name glm-5.3-flash \ --host 0.0.0.0 --port 8000 \ --trust-remote-code \ --tensor-parallel-size 2 \ --gpu-memory-utilization 0.85 \ --max-model-len 262144 \ --max-num-seqs 6 --block-size 2304 --moe-backend marlin \ --speculative-config '{"method":"mtp","num_speculative_tokens":4}' \ --kv-cache-dtype fp8_e4m3 --kv-cache-memory \ --enforce-eager --max-num-batched-tokens 8192 \ --tool-call-parser glm47 --enable-auto-tool-choice \ --reasoning-parser glm45 --default-chat-template-kwargs '{"enable_thinking": false}' \ --distributed-executor-backend mp \ --nnodes 2 --node-rank <0 on head, 1 on worker> \ --master-addr --master-port 29521 ``` Per node, two things differ: `VLLM_HOST_IP` (each node's own IP) and `--node-rank` (0 on head, 1 on worker), plus `--headless` appended to the worker's command only. `--kv-cache-memory`: do not copy a byte value from someone else's hardware. Launch once, read vLLM's own reported "estimated maximum" for your box, and set it from that. 262K is the fp8-KV context ceiling for this path; larger context needs an NVFP4-KV-cache variant (community-fragile, not used here). ### 6. Verify ```bash curl http://:8000/v1/models ``` Then send one real chat completion before trusting it further. `nvidia-smi` showing 0% GPU is a stronger signal of a dead process than `docker ps` showing "Up", since a hung distributed handshake still shows as running. ### Two things worth knowing - **`--reasoning-parser glm45` with `enable_thinking: false` is correct, not a mistake.** With thinking disabled, this checkpoint narrates its reasoning openly in `content` instead of using a `` block. That is real model behavior, not a parser leak; the parser only separates reasoning from content when a `` block actually exists. (Both quant paths leak planning prose into `content`; `tool_calls` stay clean.) - **Cache pressure during model load is real on this hardware.** Run a background loop on both nodes during load that flushes page cache if it exceeds ~40GB (`sync && echo 3 > /proc/sys/vm/drop_caches`, needs root). GB10's page cache can otherwise starve the GPU allocator mid-load. --- ## Alternative: EXL3 build A second mature path uses the MiaAI-Lab 2x-DGX-Spark EXL3 kit. The trade-off against NVFP4: a much larger served context and roughly 2x decode, at a much higher first-token latency on large prompts (EXL3 prefill plus per-shape CUDA graph capture). - Kit: [MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks](https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks) - Image: `ghcr.io/miaai-lab/glm-5.3-flash-2x-dgx-sparks:exl3-instanttensor` - Checkpoint: `brandonmusic/GLM-5.3-Flash-tr3-4bpw` (EXL3 TR3, routed-experts-only 4bpw, ~164 GiB). Drafter: `incoai/GLM-5.3-Flash-DFlash2`. Let the kit download to the HF cache; it wants HF-cache layout, not a flat `--local-dir` download. - Launch via the kit's `./start.sh` after editing its `.env`. Values that differ from the kit defaults: ``` HEAD_IP= WORKER_IP= HEAD_CX7_IF= WORKER_CX7_IF= HEAD_CX7_IB= WORKER_CX7_IB= NCCL_IB_GID_INDEX=3 HF_BIN= SKIP_BUILD=1 ``` - Kit defaults kept: TP=2 (mp executor), PORT=8888, MAX_MODEL_LEN=850000, GPU_MEM_UTIL=0.85, KV_CACHE_DTYPE=fp8 (fp8_ds_mla), ENFORCE_EAGER=0 (CUDA graphs on), LOAD_FORMAT=instanttensor, SPEC_METHOD=dflash DFLASH_TOKENS=7, reasoning-parser glm45, tool-call-parser glm47. `start.sh` runs baked overlay patches before `vllm serve`, which is why the kit is used rather than a hand-rolled `docker run`. - 850K is the kit default served context; 1M configures and boots but is not stress-tested to the edge on two nodes. - **Licensing:** the DFlash2 drafter is CC BY-NC-ND (non-commercial). Fine for personal or research use; flag it for commercial deployments. --- ## Measured performance (2-node cluster) Single-stream, short code prompt, temp 0: | | NVFP4 | EXL3 4bpw | |---|---|---| | served context | 262,144 | 850,000 | | decode (short prompt) | 23.3 tok/s | 42.3 tok/s | | big-prompt TTFT | 3-12 s | 325-432 s | | tool-calling | works | works | EXL3 has roughly 2x the raw decode and ~3.2x the context, but much higher first-token latency on large prompts. The big-prompt TTFT figures were measured on large prompts under concurrency, so expect them to vary with prompt size and load. For workloads that send large contexts, that prefill cost is the real trade-off against the context and decode wins. --- ## Risk and operational notes - **EXL3 prefill and graph-capture latency on large prompts** (several minutes TTFT seen on big prompts under concurrency). Validate with a representative prompt before relying on it, and warm the graph caches first. - **Keep `GPU_MEM_UTIL` at 0.85.** 0.87 has crashed heads. Keep a MemAvailable watchdog armed. - **Do not run a heavy download during a model load or serve.** A concurrent large download can starve GB10's unified memory and leave vLLM almost no KV budget, killing the head mid-load. Serialize staging and serving. - **A hung distributed handshake shows as a running container.** Confirm the endpoint answers and the GPUs are busy, not just that `docker ps` says "Up".