Local AI Can Red Team. Part 2: Now, Can You Trust the Scoreboard?

TL;DR#

Part 1 ended on a confession. I’d ranked 33 local models for red-team work with a single local judge that, on some runs, was grading its own output, and when I tried to hand the harder transcripts to a frontier model for a second opinion, it balked at a big chunk of them. A ranking is only as good as the thing that produced it, and mine had a soft spot right in the middle.

So this round I rebuilt the scoring before I touched the ranking. I stood up a nine-judge panel - seven open models on the two DGX Sparks plus two hosted frontier judges - and ran it over the whole benchmark instead of a sample: 2,979 graded transcripts, every model, every suite. The red-team suite alone went from 4 graded models in Part 1 to 52.

Here’s what came out the other side. On red-team work a small cluster of DeepSeek-V4-Flash models shares the top - a tie that also includes a non-DeepSeek model, Kimi-Dev-72B, with two or three others two hundredths of a point back. Across all three suites the nominal best all-rounder is DeepSeek-V4.1-Flash-REAP256, a pruned community build, and I mean “nominal” literally: its lead is a rounding error, it rides three security tasks, and it was graded while running slow. The reassurance I can offer is narrow but real - take the blind judges, pull its own family out of the vote, and it still lands on top.

The models were the easy half, same as last time. The hard half was earning the right to trust the scores, and the obstacle that kept resurfacing is an awkward one: the frontier lab whose judgment I’d most want grading offensive security is the one that balks at the hardest parts of it. Most of this post is the story of building a judge I could trust, and what it cost to find out which models I couldn’t.

Previously, on two DGX Sparks#

Short version, if you skipped Part 1: I put two NVIDIA DGX Sparks on a desk, loaded 33 open-weight models, and asked a blunt question. Can a model running on hardware you own actually do red-team work - BloodHound attack-path analysis, phishing design, offensive code? The answer was a qualified yes, and the loudest caveat I saved for the method. I scored those models with one local judge, ran each task exactly once, and leaned on hand-reviewed transcripts because the automated numbers came from a model that was, in places, marking its own homework.

This post is me going back and fixing the part I flagged. Same hardware, same three suites, a much bigger roster, and a scoring layer built to be trusted instead of apologized for. Part 1 asked which local model to run. Part 2 answers it, and then spends the rest of the post being honest about how far that answer goes.

A word on what grew, because “expanded” is doing real work here. Part 1’s red-team suite - the 50-prompt willingness-and-accuracy gauntlet - only ran against four models. Four. This round it ran against 52. Security capability went to 50 models, coding to 34, and every transcript was scored by the panel rather than skimmed by one judge. Most of the Part 1 roster was re-run on a new serving stack so the comparison wasn’t muddied by different plumbing, with a few exceptions I’ll name. If Part 1 was wide and shallow with a judge I didn’t trust, Part 2 is the narrower, deeper slice I can stand behind.

One thing to clear up before the table, because a Part 1 reader will go looking for it. My Part 1 day-to-day pick, Qwen3.6-35B, isn’t on the all-rounder board below. It didn’t lose. My results sheet keyed its three suite runs as three separate rows and I didn’t catch it before the roll-up, so it never got a single combined score. That’s a bug in my bookkeeping, not a verdict on the model, and it stings extra because silent keying bugs are a running theme of this whole post. I chose not to hand-patch one model’s rows after the fact rather than apply a fix I couldn’t apply uniformly to everyone, so it sits out the combined ranking and I point at its per-suite numbers where they matter. The headline list is the 17 models that carried a combined score across the three suites (15 of them with complete coverage; two were short a coding task or two).

The short version of the ranking, before the methodology: the top all-rounder was DeepSeek-V4.1-Flash-REAP256 (0.678 on a 0-to-1 scale), a nose ahead of Qwen3.6-abliterated (0.621) and Qwen3-Coder-Next-abliterated (0.615). Hold those numbers loosely. By the end of this post you’ll know exactly how much weight they can bear, which is less than the ordering suggests.

Where these models came from, and why#

The roster is bigger, but it isn’t random. Three rules shaped it.

First, architecture over size, which was Part 1’s hardest-won lesson. On this hardware the bottleneck is memory bandwidth, so a sparse mixture-of-experts model runs circles around a dense one of the same weight. Every candidate got weighed on that basis before it got downloaded, which is why the list leans toward MoE checkpoints.

Second, deliberate pairings. A recurring argument in this space is whether “abliterated” community fine-tunes, the ones with refusal behavior surgically removed, quietly damage the model in the process. You can’t answer that by looking at abliterated models alone. So I ran base-and-abliterated pairs on the same footing wherever I could: Qwen3.5-35B against its Huihui abliteration, GLM-4.7-Flash against its abliteration, and a three-way on the Qwen3.8-27B family - the official FP8 base, an uncensored community build, and an abliterated fine-tune - so the comparison is apples to apples. That choice pays off later; the abliteration answer this round is more interesting than Part 1’s.

Third, offensive-security relevance. The security-specialist models from Part 1 came back - RedSage, Foundation-Sec, WhiteRabbitNeo, DeepHat - alongside new blood like CyberStrike-OffSec-35B.

On where the weights come from, my rule hasn’t changed since Part 1, and it’s worth saying plainly because half the useful builds for this hardware are community patches: the bar is verification, not origin. Before anything runs I check who published it, whether the build is inspectable, whether the weights trace back to the model author’s actual org, and the blast radius if it turns out compromised, and I write the outcome of that check into the ledger. A community build from an established team with an inspectable recipe clears the bar; an anonymous binary doesn’t.

The top all-rounder is a build most people won’t recognize, so it’s worth a line. DeepSeek-V4.1-Flash-REAP256 is a community “expert-pruned” version of DeepSeek-V4.1-Flash - REAP drops a slice of the mixture-of-experts weights to shrink the model while trying to preserve its behavior. It earned its slot the hard way, which is a story in itself (see the P.S.).

Some models are in the ledger as a flat no, which is still a result. A few community repos I’d scored in Part 1 had simply vanished by this round - a Dolphin abliteration and the Qwen2.5-Coder abliterated trio all 404’d - so their only surviving numbers are Part 1 transcripts, re-judged by the new panel rather than re-run. A couple more hit kernel limits on the GB10 silicon that no config flag routes around. I note them where they’d otherwise look like silent gaps.

A benchmark is only as good as its judge#

Here’s the fact I couldn’t get past after Part 1. The ranking everyone quotes from a benchmark is downstream of one decision nobody looks at: who graded it. In 2026 that grader is almost always another language model. LLM-as-a-judge sits behind a large share of the leaderboards and reward models the field runs on, and the research on whether those judges are any good is not reassuring. A large-scale audit of 21 judges found that a judge’s ranking of the same models can shift by up to 14 positions when you swap the benchmark under it, and that production judges carry measurable, systematic position bias. The judges are reliable in the narrow sense that they repeat themselves; whether they’re measuring the thing you think they are is the open question.

My Part 1 judge had a cruder problem. It was a single uncensored local model, and because it was also one of the models under test, on some runs it was scoring its own transcripts. I was upfront about it and leaned on hand review to compensate, but “trust the human spot-checks, not the automated totals” is not a thing you can say twice and keep a straight face. If Part 2 was going to re-crown anything, the judge had to be rebuilt first.

The fix isn’t mine to claim - swapping a single judge for a panel, a jury of models whose quirks cancel out, is a known line of work, and so is the pile of 2026 papers warning how shaky single judges are. That’s the direction I went. The twist, the part that makes offensive security its own problem, is in the next section: the judges you’d most want are the ones most likely to refuse.

The panel: nine judges, and the one that walked#

The panel settled at nine. Seven are open models running on the two Sparks - DeepSeek-V4-Flash, Qwen3.6-35B-A3B, Nemotron-3-Super-120B, Qwen3.5-122B, GLM-4.7-Flash, Qwen3-235B-A22B, and the pruned DeepSeek-V4.1-Flash-REAP256. Two are hosted frontier models, astra and daybreak-blue. The open judges are free and fast once they’re up but need real reliability engineering to trust (the next section is a catalogue of what that took). The frontier judges are steadier, cost money, and, in one case, refused a chunk of the work.

That one case is a finding on its own. I wanted three frontier judges. The third was Anthropic’s Claude Opus, exactly the grader you’d reach for first. It scored most of the red-team suite, but it refused 448 of 2,550 transcripts - about 18% - flagging them on content policy rather than failing to score them, and the refusals clustered on the most explicit, abliterated-model outputs. The same safety training that makes Opus good at reasoning about attacks makes it decline to read the roughest ones closely enough to grade. astra and daybreak-blue scored all 2,550 with no refusals. An 18% hole in one judge’s column is too big to reconcile, so I pulled Opus rather than publish a ranking with a fifth of its votes missing from the hardest cases. (A correction to Part 1 while I’m here: I wrote back then that every frontier model I tried declined the job. With this cleaner setup that was too broad - two of three frontier judges graded everything. Opus is the holdout.)

One of the nine, daybreak-blue, has a second job. After all nine score a transcript, a master judge reconciles the nine opinions into the final number, and daybreak-blue - a GPT-5.6-class model - is that master. That matters for how you read everything downstream: the published ranking is not a panel average. It’s one frontier model’s reconciliation, with the panel’s nine votes and written justifications as its evidence. I’ll come back to what that buys and what it costs, because it costs more than it first looks.

Running this wasn’t free, though it was cheap for what it is. The whole hosted-judge bill - the frontier gap-fill plus the final reconciliation pass - came in around $85, held down by capping judge output and seeding every pass to skip cells already scored.

The shape of the job, by the numbers:

That’s 2,979 graded transcripts, the security suite scored by all nine judges across the board, the other two by up to nine.

When the judges broke#

The numbers above were the easy half, same as last time. Getting nine judges to produce a score I could actually believe was the hard part, and every bug below would have moved the ranking if it had slipped through.

The scores were a lie until I fixed how the model reasoned. The eventual top all-rounder, DeepSeek-V4.1-Flash-REAP256, first came through the coding bench at 6 of 34 Python and 1 of 39 Go - a small fraction of what its own predecessor had scored on the same tasks. A model that bad doesn’t become a useful judge, and it shouldn’t top a leaderboard. The model wasn’t the problem. Three stacked config bugs were: a flag that silently disabled its reasoning, a reasoning parser that then swallowed the answer whole, and a default sampling temperature low enough to send a reasoning model into a loop. Fixed - reasoning on, parser off, temperature back up - it scored 22 of 34 Python and 9 of 39 Go, and went on to become the seventh open judge. The lesson that stuck: the coding bench, which demands exact correctness, caught a mis-served model the prose suites happily waved through on fluency. (The config fixes rescued the scores. A separate problem, a broken GPU kernel, was wrecking its speed and context length, and that’s the P.S.)

Judges reason forever, exactly like the models they grade. Two of the heavier open judges, Qwen3.5-122B and Nemotron-3-Super-120B, would occasionally never stop thinking on a hard transcript. Not a timeout - a genuine non-terminating reason loop, confirmed by re-running the same input at larger and larger token budgets with no change. The fix was a reasoning-budget cap that forces the model to stop thinking and commit rather than just raising the ceiling. I watched one specific transcript fail three normal attempts and then score cleanly the instant the cap kicked in. A judge that hangs on the hardest cases is a judge that drops your hardest cases.

A data-shape mismatch quietly ate 156 transcripts. The offensive-tooling coding tasks use a split rubric - one set of criteria for compliance, another for quality - and the scorer was only ever reading a flat rubric key those tasks don’t have. Every one of those 156 transcripts failed an “is this scoreable” check and got logged as an ordinary skip instead of flagged as a gap. Nothing crashed. The totals just undercounted a whole category until I read the task’s own design doc and saw the shape didn’t match. This is the one that scares me, because it’s invisible: no error, no stack trace, just a number that’s wrong and confident.

And a couple of shorter ones. Deploy recipes drifted between the two nodes more than once - the same model file with three different checksums across my laptop and the two Sparks, one of them carrying a stale, buggy setting - a reminder that “it worked on the other node” is not proof of anything. And the hosted-judge credits ran dry partway through one coding pass; I stopped rather than burn a pile of guaranteed-failing calls, and came back to finish the pass once I topped the budget up.

And here’s the trap in all of that, so I’ll say it before you do: finding four bugs proves I looked, not that the numbers are clean. I caught the 156-transcript miscount by chance-reading a design doc and the kernel bug after being wrong four times; neither came from a systematic audit that could tell me how many silent errors are left. By the post’s own logic - the worst failures are the invisible ones - finding one raises the odds there’s another. Treat the board as my best effort, not a certified count.

(One meta-note, since I built a lot of this with an AI agent at my shoulder: the agent’s own guardrails have sharp edges too. It balked at live secrets in backgrounded shell commands, at multi-line commit messages, and at destructive git operations - which, given the subject of this post, I found hard to be annoyed about.)

A field guide to a judge panel you can trust#

The rules worth stealing even if you never touch a DGX Spark:

• Cap the thinking, not just the tokens. A reasoning model used as a judge needs a hard reasoning-budget cap or it will hang on your hardest transcripts and drop them. Raising the output ceiling doesn’t fix it; forcing a conclusion does.
• A judge is good or bad per suite, not in general. The same model can grade code cleanly and come apart on red-team prose, or the reverse. Validate each judge on each suite before you trust its column.
• Read the task’s design doc before you trust the data shape. The 156 eaten transcripts came from a scorer assuming a rubric shape the task didn’t use. If a category’s counts look light, suspect the parser before the models.
• Strip the reasoning out of the answer. A thinking model served without the right parser hands you its scratchpad and its verdict in one field; the scoring code has to split on the end-of-reasoning marker or it grades the wrong text.
• Exclude retry-archive files from whole-tree scoring passes. The per-attempt JSON files don’t carry a model key and will crash or pollute a naive aggregation.
• Use inter-judge agreement as a trust signal, carefully. When a new judge tracks the established panel you can lean on it. But agreement among models from overlapping families can mean a shared blind spot as easily as a shared truth, so it bounds your confidence rather than confirming you’re right.
• Keep frontier judges for calibration, not as the whole panel. They’re the steady reference the open judges get checked against - but as the refusal showed, you can’t always get the one you want.
• Write the provenance check into the ledger. Every model and image: who published it, whether you verified the weights, recorded next to its results.

The ranking, and how much to trust it#

Here’s the all-rounder board - the 17 models that carried a combined score across all three suites, on a 0-to-1 scale. I’m showing the top as a band rather than a podium, because the margins up there are smaller than the noise.

The per-suite winners, since the combined number hides them: security went to DeepSeek-V4.1-Flash-REAP256 (8.0 of 10, its one clean win), and coding to DeepSeek-V4-Flash-abliterated (6.14 of 10, tied with a coding-only Qwen build).

DeepSeek-V4.1-Flash-REAP256 is on top at 0.678. Now let me take the shine off it, because on its own that number is doing way more work than it earned.

The lead is thin, and it rides three tasks. In the published scores, V4.1’s margin over second place is 0.057. On the raw panel averages it’s between 0.004 and 0.010 - a rounding error. And almost all of it comes from security: V4.1 scored 8 of 10 where the all-rounder runner-up’s security sub-score was 6, and that single gap, across just three security tasks, is worth more combined score than its entire lead. Resample the tasks and V4.1 comes out first about 85% of the time on the published scores, but only around 56% of the time on the raw panel means. It’s the nominal best all-rounder. It is not a runaway, and anyone who tells you a 0.057 gap on a 0-to-1 scale is decisive is selling something.

It was graded slow. This cuts against Part 1’s whole thesis, so I’ll say it loudly. Part 1’s lesson was that fast sparse models win on this hardware. The all-rounder leader was judged on transcripts generated at 32,768 tokens of context, with CUDA graphs disabled, at 7 to 11 tokens per second - a crawl. A transcript is a transcript, so its quality score is still fair; a slow lane doesn’t make a good answer worse. But that score says nothing about whether you’d enjoy running it, and one caution: the build that later ran much faster is a different quantization, not the same weights served better, so even the quality number doesn’t automatically carry across to it. Keep quality and throughput in separate mental columns.

On red-team, nobody actually runs away - and this is where I have to not swallow my own hype. The top of the suite is a four-way tie at 3.04 out of 4 - except it’s really three models, because one of them, the abliterated DeepSeek-V4-Flash fork, shows up twice under two repo names with byte-for-byte identical weights. The three are base DeepSeek-V4-Flash, that abliterated fork, and the non-DeepSeek Kimi-Dev-72B, with GLM-4.6 and Mistral-Small-4 two hundredths of a point back at 3.02. So the honest line is that two DeepSeek-V4-Flash checkpoints share the top with Kimi-Dev-72B, not that the family ran the table. “The family leads red-team” only holds if you let a duplicate repo count as its own slot, and you shouldn’t. Which is exactly where someone should cry foul, so let’s get to it.

Two of my nine judges are DeepSeek-V4 models. Did they rig it for their own family? It’s the first thing I’d throw at someone else’s benchmark, so I threw it at mine. Short answer: yeah, there’s a small thumb on the scale; no, it doesn’t flip the result I can actually test; and I’ll be straight about the one test I can’t.

The two DeepSeek judges rate DeepSeek models about 0.19 points higher - on the 0-to-4 red-team scale, a fifth of a point - than the other seven judges do. Measurable, not zero. Two things keep it from being damning. First, every judge rates the DeepSeek models highly, not only the two DeepSeek judges; they’re genuinely strong, and the nine panel judges are blind to model names, so a judge can’t play favorites by identity even if it wanted to. At most it recognizes a house style. Second, when I test whether the DeepSeek pair’s tilt is unusual against all 36 possible judge pairings, it lands in the upper middle on seven of the eight measures I ran - real, but inside the spread of ordinary judge quirks - and comes out first on the eighth, a borderline result (p about 0.03). The one number I won’t bury: the V4.1 judge gives its own transcripts the biggest self-bonus of any judge, about +0.41, high enough to notice.

So pull the DeepSeek judges out and see what moves. Re-rank red-team with only the seven non-DeepSeek judges and the order still correlates with the full panel at 0.995; do it on the all-rounder board and V4.1 is still first, same top-six set under it. Here’s the catch I’m not going to bury, because it’s the sharpest knife anyone paying attention could put in this post. That test runs on the blind panel’s raw averages, not on the published board. The published board is the master judge’s reconciliation, and I deleted its key after the run, so the headline number itself cannot be recomputed with the DeepSeek judges removed. And the master, unlike the nine panel judges, is not blind - its prompt names the model under test. So the single number that crowns V4.1 is the one most exposed to both the family tilt and name-awareness, and it’s the one the de-conflicting test can’t reach. On the blind panel means it can reach, V4.1 is the top all-rounder only about 56% of the time. The honest version: the top band is robust, the #1 slot is a coin-flip, and the check I’d most want to run on the published number is the one I can’t. I’m telling you that rather than letting the 0.678 carry an authority it hasn’t earned.

One more finding, because it surprised me. The model that tops the all-rounder board was, as a generator, mediocre at code: 9 of 39 on Go. As a judge it correlated solidly with the rest of the panel (Pearson 0.78 to 0.85, closest to Qwen3.5-122B), grading a shade harsher than average. Topping the board and grading well are different skills, and this benchmark is partly a demonstration of how far apart they can sit.

Since this is a sequel, what held up from Part 1? The one I can actually re-examine with this data is abliteration, and the answer is more mixed than Part 1 let on. Take the cleanest pair, Qwen3.5-35B against its Huihui abliteration: the abliterated build was better on red-team (2.86 vs 2.04) and worse on security (3.33 vs 6.67, a halving). The DeepSeek-V4-Flash family ties base-to-abliterated on red-team and the base edges the forks on security (7.33 vs 5.67). So removing refusals doesn’t lobotomize a model on offensive tasks - Part 1’s finding holds there, and sometimes abliteration even tracks with more red-team willingness - but it is not free, and at least one abliteration clearly paid for it on security. It depends on the model and the task, not a blanket yes or no. The other two Part 1 headlines - willingness is cheap and accuracy scarce, architecture beats size - I didn’t re-measure at this scale, so take them as carried forward, not re-proven; Part 1 has the per-model autopsies.

A closing caution on reading any of this as a buy recommendation. “Best on the benchmark” and “the model I’d actually run behind an agent” are different questions, the same way they were in Part 1, and which model I put into production is its own write-up, not this one. This section ranks quality, measured carefully and hedged honestly. What you run depends on your latency budget and your threat model as much as on a score.

The honest caveats#

Part 1’s caveat list had one entry that embarrassed me: a judge grading itself. Here’s Part 2’s list, with that one replaced by problems I can at least measure.

1. The final judge is itself a frontier model, and it’s not blind. The published ranking isn’t the panel’s average. The number you see is one frontier model’s call - daybreak-blue, reconciling the nine votes, with the model’s name in front of it. I traded a self-grading local model for a single non-blind frontier reconciler. That’s a real gain in capability and consistency, and it’s still one model’s judgment at the end of the chain, and the one place in the pipeline where knowing which model it’s scoring could matter. I documented it. That’s the most I can say for it.
2. There’s no human gold standard. “Trust” here means the judges agree with each other and with my spot-checks, not that they agree with a panel of human experts. I don’t have that panel, and agreement among models from overlapping families can reflect a shared blind spot as easily as a shared truth. Worse, I had partial ground truth for the hardest task - the BloodHound exam from Part 1 has a known answer key, the ESC1 path and the disabled-account trap - and I still scored it by judge opinion rather than against that key. That’s a point I left on the table, and I’d change it next round.
3. The self-preference thumb is real, just small. Covered above: the DeepSeek judges tilt toward DeepSeek models by about a fifth of a point on the accuracy axis, the result survives removing them from the blind panel, and I can’t run that removal on the published board. You have the numbers; take them over my reassurance.
4. Still n=1. Every model generated each task exactly once. The panel fixed how transcripts get scored, not how many times they were sampled. A bad or a lucky day is still baked into a row.
5. Coverage isn’t perfectly even. About 30 of the roughly 3,000 cells came in below nine judges - a handful at eight, three coding cells at seven - from a judge emitting non-JSON on a transcript, a reasoning loop, or a context limit, and the master reconciled those from the votes it had.

Who gets to grade offensive AI#

Pull back from my two desktop boxes for a second, because the thing this project keeps tripping over is bigger than a leaderboard. Almost every ranking you trust right now - the leaderboards, the reward models, the eval dashboards - has a language model grading it somewhere, and the audits say those graders shift rankings by double-digit positions between benchmarks and carry systematic bias. That’s for benign work. For offensive security it gets worse in a specific way: the frontier judge you’d most trust balks at the hardest parts of the work, so you lean on the ones that will take it and you verify them yourself. The oversight for the single most dangerous capability these models have is something you assemble out of the models still willing to look, and then check by hand.

Part 1 ended on a related worry: the hard problem had stopped being whether a model can act as a red-team operator and become whether anyone can keep one inside the box they built for it. Part 2 is that worry one step down the stack. Part 1 asked whether you can trust the operator. This post asks whether you can trust the thing grading it. I built a panel I trust more than Part 1’s single judge, and I can still only tell you how far that trust extends, not that it’s absolute.

Which is as good a way as any into what’s still broken. Neither Part 1 nor Part 2 has closed the big one: I’m still grading analysis - the essay, not the break-in. Every score in this post rewards a model for reasoning well about an attack on paper - naming the ADCS path, writing the plausible pretext, explaining the privilege escalation. None of it checks whether the model can actually do the thing against a live target. The ground truth is still an answer key I wrote, and an answer key can be wrong.

So Part 3 changes the question from “can it explain the attack” to “can it run the attack.” I’m building a deliberately-vulnerable lab - real machines, real misconfigurations, real exploitable paths - and pointing an agent at it with the models from this benchmark behind it. The score is simple: did the box fall. That’s a harder benchmark to build and a much harder one to fake, and it’s the only way I know to close the gap between “it produced a convincing attack path” and “the attack path was real.”

What I take away#

The scoring layer is the real project. I spent more effort making nine judges trustworthy than I did running the 52 models they graded, and that ratio is the lesson. If you’re standing up a local eval for anything that matters, budget for the judge, not just the models.

A good generator and a good judge are different models. The model that topped the all-rounder board couldn’t write Go, and graded security analysis as well as any open judge on the panel. I’d assumed the model that produces the best output is the one that should grade it. I was wrong.

You can build local offensive-security eval you mostly trust, and you own the part you can’t outsource. The capable judges will grade this work; the one you’d most want balks at the worst of it. That’s not an excuse to skip the oversight. The oversight is yours to build and yours to check. Nobody else will.

So, the title question: can you trust the scoreboard behind local red-team AI? Further than Part 1’s. Not all the way. But I can now put my finger on exactly where the trust runs out - and that’s the only kind of yes that’s worth anything.

If any of this is useful - running local models for offensive work, building an eval you’d stake a finding on, keeping an agent honest - my email and socials are on the site. Part 1 closed on Feynman: you are the easiest person to fool. Part 2 is that principle one level down. Before you trust what the model told you, check the thing that told you the model was right.

“Journey before destination.” - Brandon Sanderson, The Stormlight Archive

P.S. - It was never the config#

For the people who, like me, can’t leave a performance mystery alone. The DeepSeek-V4.1-Flash build that became a panel judge spent two days convincing me it was four different bugs before I found the real one.

It started slow and short: 11 to 13 tokens per second, and a hard crash the moment a prompt crossed about 35,000 tokens. I diagnosed it confidently four times, and each diagnosis was wrong. First I blamed the launch mechanism - the head process was dying as a child of a backgrounded SSH session, torn down by a hangup signal - and fixed that, and it was still slow. Then CUDA out-of-memory, falsified by the timestamps and the clean shutdowns. Then speculative decoding, disproven when the same crash happened with it off. Then the prefill chunk size, which “fixed” it by papering over it and left me stuck at 11 tok/s.

What broke it open wasn’t more instrumentation. It was a question: why were these numbers so much worse than the DeepSeek-V4-Flash run I’d already done on the same two boxes? I went back to my own ledger, and there it was - V4-Flash, same hardware, serving at over a million tokens of context and 67 to 117 tok/s. The config wasn’t the difference. The kernel was. V4.1’s attention path was hitting a FlashInfer sparse-MLA kernel that’s broken on the GB10 silicon, which forced the engine into an eager execution mode, and eager mode is where all the throughput went. The fix was to serve it through a build with a working kernel - the ExLlamaV3 path, which ships its own. On that lane a 2.9-bit EXL3 build of the model ran at 22.9 tok/s and served a 562,940-token prompt without blinking, memory flat the whole way. That faster lane is the one thing in this saga the benchmark never saw: the scored transcripts were all the REAP-256E build on the slow, broken-kernel lane, before the fix.

The twist I enjoyed most: I went looking for a lighter checkpoint to dodge the whole problem and enumerated all 103 V4.1-Flash builds on Hugging Face. Nothing meaningful was lighter - the experts are already 4-bit upstream, so the “smaller” quants aren’t. The build I shipped is heavier per node than the one it replaced. Sometimes the fix is bigger, not smaller.

The lesson, free of charge: before you diagnose a regression, check your own prior results on the same hardware. I had the answer in my ledger the whole time. I just didn’t look until I’d been wrong four times.

Bonus: the two-Spark recipes#

Two of the heavier models here are genuinely awkward to serve across two DGX Sparks, and the fixes aren’t obvious. I wrote both up as standalone runbooks you can hand straight to your own cluster, or to an ops agent - generic, with placeholders for hostnames and IPs and none of the benchmark plumbing:

• DeepSeek-V4.1-Flash on two DGX Sparks - the fast lane from the P.S.: the ExLlamaV3 EXL3 build that hits ~22.9 tok/s and a ~600K context instead of crawling on the broken kernel.
• GLM-5.3-Flash on two DGX Sparks - a second large MoE on the same two-node setup, start to finish.