# Protocol and known limitations The full write-up of the round whose outputs this repository publishes: what was run, every deviation from the published BenchMAX protocol, and every artefact of the setup that a number here might otherwise be mistaken for a property of a model. [`../README.md`](../README.md) is the short version; this is the one to have open while reading a score. --- A re-run of the Czech portion of [BenchMAX](https://github.com/CONE-MT/BenchMAX) ([paper](https://arxiv.org/abs/2502.07346)) against a set of current open-weight models, with English kept alongside as the control. The point of the benchmark is the **gap between a model's English and its Czech performance**, so every task is run in both languages and nothing is reported for Czech that is not also available for English. Every model is additionally run twice — thinking on and thinking off — so the cost of reasoning is measured rather than assumed. Everything here is deliberate. Where this setup departs from the published protocol, the departure and its reason are written down below rather than being buried in a script. **Status (2026-08-12): the round is finished.** All five models have been run in both conditions — ten `results/` directories — over every task in §1 except the domain translation sets, which were not run for anyone (§7). Nothing is queued and the model servers have been released. Two findings that landed with the last runs are worth reading before the numbers: the e-infra gateway silently caps generation at 8,192 tokens for the harnesses that send `max_tokens`, which cost DeepSeek four of its seven tasks' budgets (§1.1); and BenchMAX scores only the *first line* of a translation, which is what the three Gemma translation scores are actually measuring (§3.4). --- ## 1. What is being evaluated ### Models under test Four models are served locally with vLLM 0.26.0 over an OpenAI-compatible API; the fifth is reached through a shared gateway. Each is selected with `BM_PRESET`, which sources a file from `scripts/models/`. | preset (`BM_TAG`) | model id | endpoint | notes | |---|---|---|---| | `qwen3.6-27b` | `Qwen/Qwen3.6-27B` | `tdll-8gpu2:8000` | dense, TP4 | | `qwen3.6-35b-a3b` | `Qwen/Qwen3.6-35B-A3B-FP8` | `tdll-4gpu1:8000` | MoE, FP8 | | `gemma-4-31b` | `google/gemma-4-31B-it` | `dll-4gpu3:8001` | TP4 | | `gemma-4-e4b` | `google/gemma-4-E4B-it` | `dll-3gpu1:8000` | | | `deepseek-v4-flash` | `deepseek-v4-flash` | `llm.ai.e-infra.cz` | shared gateway: 2 concurrent requests, and an 8,192-token output cap on half the tasks (§1.1) | Each has a `-nothink` counterpart preset (`qwen3.6-27b-nothink`, …) — same model, same endpoint, same token budget, same concurrency, reasoning disabled. That is the only difference, so any delta between the pair is attributable to reasoning alone. A preset only sets defaults, so any variable can still be overridden on the command line. Nothing in the harness is model-specific beyond these files — anything behind an OpenAI-compatible URL can be evaluated. ### 1.1 The e-infra endpoint is not like the others DeepSeek V4 Flash is served by someone else, and five things follow. The first four are handled in `scripts/models/_einfra.inc.sh`, shared by both of its presets; the fifth was only identified after the runs finished and is a live caveat on the results, not a fixed problem. - **The key never enters the repo.** It is read at run time from the opencode credential store (`~/.local/share/opencode/auth.json`, overridable with `BM_EINFRA_AUTH`), and `BM_API_KEY` still wins if exported by hand. - **Concurrency is capped server-side at 4**, not by our throughput guess. At 8 requests in flight, 4 of 8 came back `429 … Limit type: max_parallel_requests. Current limit: 4, Remaining: 0` with `retry-after: 1800`. We use **2**. A short probe made 3 look safe (24/24 successes, where 4 measured 23/24), but over a full task it is not: the first attempt at the suite hit exactly one 429 in nexus and one in translation, both past request 300, so the gateway's slot accounting evidently lags a little behind ours. The endpoint is latency-bound rather than slot-bound, so the third slot bought little anyway. The local vLLM servers, by contrast, run 64. - **Retries have to be written by hand** — twice over, for rate limits and for outages. See the traps below. - **Wall-clock limits had to be raised.** Two concurrent requests is roughly 115 tok/s against ~4.5k on a local server, so `submit.sh` takes `BM_TIME` (default `2-0`) and the e-infra chain is submitted with `BM_TIME=7-0`. - **The output budget is not what we asked for** — and this one is *not* fixed. The gateway clamps `max_tokens` to 8,192 while honouring `max_completion_tokens`, so four of the seven tasks DeepSeek ran (IFEval, MGSM, GPQA, LCB) used an eighth of their configured budget. Third trap below; consequences in §3.1. #### The trap: `max_retries` does nothing here Every harness constructs its client with `openai.OpenAI(..., max_retries=5)`, and against this endpoint **not one of those retries ever fires**. `_should_retry` in the SDK bails out before consulting `max_retries` at all: ```python if retry_after is not None and retry_after > MAX_RETRY_AFTER_DELAY: # 120s return False ``` e-infra answers a 429 with `retry-after: 1800`, so the SDK classifies it as not worth retrying and raises on the first one. This is easy to miss precisely because the same constructor argument behaves exactly as documented against the local vLLM servers. The consequence is out of all proportion to a ~0.3% failure rate. Nexus and translation both submit every item to a `ThreadPoolExecutor` and then read the futures back in order, so a single 429 anywhere in the batch raises out of `.result()` and discards **everything the batch already generated**. Observed: nexus `en` died on request 60 of 318 with 317 telemetry rows already on disk and no output file written at all; translation lost 1011 of 1012 completed segments the same way. `analysis/retry.py` (installed as `bm_retry`, §4.2) wraps those two call sites in a deliberately slow schedule — 15s, 30s, 60s, then 120s, ~10 minutes before giving up. Against a run that takes days, ten minutes of patience is free. evalplus and LiveCodeBench already had retry loops of their own and were never affected. lm-eval does not use the SDK at all; it retries with tenacity, and needs both of its knobs turned — see the next section. #### The second trap: five minutes is not patience The gateway is not merely rate-limited, it goes down. At 00:08 on 2026-08-08 it began answering 502/503, and within four and a half minutes the consequences were: - a 3h38m `xgpqa_cs` run discarded at item 248 of 448, and - **all eight jobs still queued behind it burned in under two seconds each**, because `bm_check_endpoint` ran, could not reach the gateway, and exited 1. The chain uses `afterany` precisely so one failure does not strand the rest; an instant-exit precheck defeated that completely. Three things were wrong, and all three are fixed: 1. **lm-eval's backoff is capped at 10s**, so `BM_MAX_RETRIES=30` bought only ~5 minutes — and cost 5798 error lines to do it. `patch_venv.py` now makes the cap settable, and e-infra sets `BM_LMEVAL_WAIT_MAX=60` with `BM_MAX_RETRIES=60`: **~54 minutes of patience at a tenth of the request volume**. 2. **The real exception was invisible.** lm-eval's `except BaseException` handler logs `f"Exception:{repr(e)}, {outputs}, retrying."`, but `outputs` is only bound after a successful `response.json()`. Every failure before that point makes the *handler* raise `UnboundLocalError`, replacing the real error — the traceback named tqdm and tenacity and never once mentioned the gateway. `patch_venv.py` binds `outputs = None` first. 3. **`bm_check_endpoint` now waits** rather than exiting on the first failed probe: it retries every 30s for `BM_ENDPOINT_WAIT` (default 1800s). Waiting costs nothing when the endpoint is up and saves the remaining suite when it is briefly down. #### The third trap: `max_tokens` is capped at 8,192, `max_completion_tokens` is not Every DeepSeek generation that ran out of budget on IFEval, MGSM, GPQA or LCB stopped at **exactly 8,192 completion tokens** — 116 of 448 on Czech GPQA, 197 of 713 on Czech LCB — even though those tasks ask for 65,536 and 32,768. Meanwhile HumanEval+ produced generations of 32,768 tokens in the same round, its full budget. The difference is not the task, it is the request field: the gateway honours `max_completion_tokens` and clamps `max_tokens` to 8,192. Probed directly, one prompt, `reasoning_effort: "none"`: | field sent | value asked | completion tokens | finish_reason | |---|---|---|---| | `max_tokens` | 20,000 | **8,192** | `length` | | `max_completion_tokens` | 20,000 | 11,001 | `stop` | Small values are honoured either way (`max_tokens: 20` returns 20 tokens), so the field is read, not ignored — it is silently clamped, and the response looks exactly like a model that chose to keep thinking. Which harness sends which field decides who is affected: | harness | field | DeepSeek's effective budget | |---|---|---| | lm-eval (IFEval, MGSM, GPQA) | `max_tokens` | **8,192** (asked for 16,384 / 65,536) | | LiveCodeBench | `max_tokens` | **8,192** (asked for 32,768) | | HumanEval+, Nexus, translation | `max_completion_tokens` | as configured | So DeepSeek's IFEval, MGSM, GPQA and LCB were run under a budget **8× smaller** than every other model's, in both conditions, and its truncation rates in §3.1 must be read against that, not against Qwen's and Gemma's. The other four models are unaffected — vLLM honours `max_tokens` — so this is a DeepSeek-only caveat. Nothing was re-run: the fix is a one-line change per harness and belongs to a round that redoes DeepSeek deliberately (§7). The gateway also **ignores `chat_template_kwargs` entirely** — it does not expose the chat template. Sending `{"enable_thinking": true}` and `{"enable_thinking": false}` produced byte-identical responses. It toggles reasoning with the standard top-level `reasoning_effort` field instead, which is why `BM_EXTRA_BODY` exists alongside `BM_CHAT_TEMPLATE_KWARGS` (§4.2). Two further things were established by probe before committing to a multi-day run, because guessing either wrong would have silently produced the wrong experiment: - **Bare `deepseek-v4-flash` is the *non*-thinking condition.** On fresh prompts it returns 0 reasoning chars. (A first probe suggested otherwise; that was the gateway serving a cached response from a previous thinking request. Every measurement below uses a unique prompt id.) Both directions are therefore set explicitly: `reasoning_effort: "high"` and `reasoning_effort: "none"`. - **The `deepseek-v4-flash-thinking` id is the same model with the default flipped**, not a second checkpoint: `deepseek-v4-flash` + `reasoning_effort=high` and `deepseek-v4-flash-thinking` at its default returned the same reasoning length and the same completion-token count. Keeping one `BM_MODEL` across the pair means the think/nothink contrast is carried by the request, exactly as it is for the other four models, rather than by a different endpoint alias. Measured on one prompt, `deepseek-v4-flash`, fresh each time: | `reasoning_effort` | reasoning chars | completion tokens | |---|---|---| | *(none sent)* | 0 | 3 | | `none` | 0 | 3 | | `low` | 117 | 39 | | **`high`** | **144** | **49** | Tool calls come back structured in both conditions, so Nexus needs no special handling here. ### Tasks | task | dataset | n (per lang) | metric | script | |---|---|---|---|---| | Instruction following (rule-based) | IFEval | 429 | accuracy | `run_lmeval.sh ifeval` | | Math | MGSM | 250 | exact match | `run_lmeval.sh mgsm` | | Science | GPQA | 448 | accuracy | `run_lmeval.sh gpqa` | | Tool use | Nexus (`standardized_queries`) | 318 | accuracy | `run_nexus.sh` | | Function completion | HumanEval+ | 164 | pass@1 | `run_evalplus.sh` | | Problem solving | LiveCodeBench v4 | 713 | pass@1 | `run_lcb.sh` | | Translation (general) | Flores + TED + WMT24 | 1012 / 1000 / 997 | spBLEU, chrF | `run_translation.sh general` | | Translation (domain) | 7 in-benchmark sets | — | spBLEU, chrF | `run_translation.sh domain` | `submit_all.sh` runs everything except the domain translation sets, which have not been run for any model this round. **Out of scope, by decision:** - **RULER (long-context)** — dropped entirely. Not run, not reported. - **Arena-Hard (model-based instruction following)** — postponed. See §7. ### Languages `BM_LANGS="cs en"`. Both are always run. IFEval's Czech set is the upstream one, from which language-specific instructions have already been filtered out by BenchMAX (hence 429 rather than ~500 samples). --- ## 2. The one decision that breaks comparability with the paper **Thinking mode is the primary condition, with substantially raised token budgets.** The published BenchMAX tables were produced with non-reasoning models. Every model here can reason, and running them with thinking disabled would measure something other than what these models are. That choice was made deliberately and it means: > **Our thinking-mode numbers are not directly comparable to the published > BenchMAX table.** Where they are shown side by side (see §6), treat the > comparison as indicative, never like-for-like. The `-nothink` runs exist partly to make that gap measurable rather than rhetorical: they are the closer analogue to the published protocol, and the viewer shows both conditions per model. Two consequences follow, and both are handled explicitly. ### 2.1 Token budgets had to be raised BenchMAX's task YAMLs set `max_gen_toks` to 1024–2048. Under thinking mode the hidden chain of thought is charged against that same budget, so those values truncate the model *before it emits any answer at all* — the response comes back with `content: null` and scores as a hard miss. Budgets are overridden **on the command line, not by editing upstream YAMLs**, so the deviation stays visible in the run scripts: | task | budget | why | |---|---|---| | IFEval, MGSM | 16,384 | `run_lmeval.sh` | | GPQA | 65,536 | measured: 16 Czech items averaged 15.1k completion tokens | | HumanEval+, Nexus, LiveCodeBench | 32,768 | `BM_MAX_TOKENS` default | | Translation | 16,384 | measured: 16 Flores en→cs segments, median 3,404 tokens, max 7,432 | The `-nothink` runs keep the same budgets, so the pair differs in exactly one variable. They never come close to using them (§3.1). These are the budgets *asked for*. Four of the five models got them; DeepSeek did not, because the e-infra gateway clamps `max_tokens` to 8,192 and lm-eval and LiveCodeBench send that field (§1.1). Its effective budget on IFEval, MGSM, GPQA and LCB was 8,192 in both conditions. ### 2.2 Each family needs a different switch There is no portable way to ask for thinking. Three mechanisms are in play, one per family, and all three are set **per request from `scripts/models/*.env`** — never with a server flag. A server flag makes "thinking was on" a property of how a job happened to be launched: invisible in the repo and unrecoverable from the results afterwards. | family | on | off | |---|---|---| | Qwen3.6 | *(default)* | `chat_template_kwargs {"enable_thinking": false}` | | Gemma 4 | `chat_template_kwargs {"enable_thinking": true}` | `{"enable_thinking": false}` | | DeepSeek V4 Flash (e-infra) | `reasoning_effort: "high"` | `reasoning_effort: "none"` | Gemma 4 defaults to thinking **off**, and launching the server with `--reasoning-parser gemma4` is not enough — the parser has nothing to separate until the *request* asks for it, so the model returns a plain answer and an empty `reasoning` field. Measured on `gemma-4-31B-it`, same prompt: | request kwargs | reasoning chars | completion tokens | |---|---|---| | *(none)* | 0 | 1,015 | | `thinking=True` | 0 | — | | `reasoning=True` | 0 | — | | `include_thoughts=True` | 0 | — | | **`enable_thinking=True`** | **2,807** | **2,020** | Only `enable_thinking` works, and only for the two self-hosted families — the e-infra gateway drops it (§1.1). Gemma runs made before this was discovered are non-thinking and have been moved to `archive/gemma_nothinking_*`. They predate the systematic `-nothink` presets and are superseded by them. --- ## 3. Known limitations in the results These are real and are **deliberately left in place**, not silently corrected. ### 3.1 Truncated generations score as wrong A generation that spends its entire budget inside the CoT returns no answer and is scored as a miss. Counts of `finish_reason == "length"`, from telemetry, for the thinking runs: | task · lang | qwen3.6-27b | qwen3.6-35b-a3b | gemma-4-31b | gemma-4-e4b | deepseek-v4-flash † | |---|---|---|---|---|---| | lcb · cs | **194/713 (27.2%)** | **204/713 (28.6%)** | 45/713 (6.3%) | 0/713 | **197/713 (27.6%)** | | lcb · en | **202/713 (28.3%)** | **186/713 (26.1%)** | 54/713 (7.6%) | 0/713 | **183/713 (25.7%)** | | gpqa · cs | 44/448 (9.8%) | **88/448 (19.6%)** | 2/448 (0.4%) | 0/448 | **116/448 (25.9%)** | | gpqa · en | 1/448 | 11/448 (2.5%) | 3/448 (0.7%) | 1/448 | **114/448 (25.4%)** | | evalplus · cs | 22/164 (13.4%) | 23/164 (14.0%) | 0/164 | 0/164 | 3/164 (1.8%) | | evalplus · en | 26/164 (15.9%) | 6/164 (3.7%) | 0/164 | 0/164 | 4/164 (2.4%) | | ifeval · cs | 17/429 (4.0%) | 30/429 (7.0%) | 25/429 (5.8%) | 2/429 | 52/429 (12.1%) | | ifeval · en | 11/429 (2.6%) | 16/429 (3.7%) | 4/429 (0.9%) | 1/429 | 22/429 (5.1%) | | mgsm · cs | 1/250 | 16/250 (6.4%) | 0/250 | 0/250 | 9/250 (3.6%) | | mgsm · en | 0/250 | 0/250 | 0/250 | 0/250 | 0/250 | | nexus · cs | 0/318 | 4/318 (1.3%) | 0/318 | 0/318 | 0/318 | | nexus · en | 0/318 | 1/318 | 0/318 | 0/318 | 0/318 | | translation-ted · en→cs | 6/1000 | 5/1000 | 7/1000 | 1/1000 | 0/1000 | | translation-wmt24 · en→cs | 13/997 (1.3%) | 18/997 (1.8%) | 9/997 | 0/997 | 0/997 | † **DeepSeek's column is not on the same axis as the others.** Its IFEval, MGSM, GPQA and LCB budget was 8,192 tokens, not the 16k/64k/32k asked for, because the gateway clamps `max_tokens` (§1.1) — every one of those truncations is at exactly 8,192. Its HumanEval+, Nexus and translation cells *are* comparable: those harnesses send `max_completion_tokens`, which the gateway honours. Read the four clamped rows as "what a small budget costs", not as a property of the model; the near-identical cs/en rates say the same thing from the other side. **LiveCodeBench is the worst-hit task and was not visible in this table until the full runs landed**: better than a quarter of both Qwen models' LCB attempts never produce a program. Their LCB pass@1 is a lower bound by a wide margin. **This is not a Czech-language problem.** Two independent readings of the table say so: LCB and HumanEval+ truncate at nearly the same rate in English as in Czech, and the two model families behave in opposite directions. Median reasoning length on GPQA, Czech vs. English: | model | en | cs | cs/en | |---|---|---|---| | qwen3.6-35b-a3b | 12,936 | 43,280 | **3.3×** | | qwen3.6-27b | 11,970 | 34,802 | **2.9×** | | gemma-4-e4b | 5,070 | 5,317 | 1.05× | | gemma-4-31b | 9,363 | 4,866 | **0.52×** | | deepseek-v4-flash | 11,451 | 6,573 | 0.57× (censored at 8,192 tokens) | (chars of reasoning content, from telemetry) DeepSeek's row is the one to distrust: a quarter of its GPQA generations in each language were cut off at the gateway's cap, so both medians are censored and the ratio is a ratio of censored distributions. It is included because the *shape* still separates it from Qwen — a model that ran away in Czech would show cs > en before the cap bit, and this one does not. So there are three distinct effects. On **GPQA** the Qwen models inflate their CoT roughly threefold when the question is in Czech and then hit the ceiling, while Gemma 4 31B actually reasons *less* in Czech than in English and neither Gemma truncates materially. Inspection of the Qwen runaways shows degenerate repetition (`Let's try **záštípání**? No. Let's try **záštípání**? No. …`), so these are loops rather than genuinely harder deliberation — the trigger looks like Czech morphology in the reasoning channel, not Czech task difficulty. On **LCB and HumanEval+** the Qwen models simply overrun the budget in either language. And **DeepSeek truncates symmetrically on the four clamped tasks** because 8,192 tokens is not enough for a reasoning model on any of them, in any language. **Therefore: part of the measured cs↔en gap on the Qwen models is a token-budget artifact, not a language-competence finding.** Treat Qwen's Czech GPQA figures as lower bounds, and both Qwen models' LiveCodeBench figures as lower bounds in *both* languages. The Gemma figures need no such caveat on GPQA, so the two families' cs↔en gaps are not directly comparable to each other either. DeepSeek needs the same caveat for a different reason: its IFEval, MGSM, GPQA and LCB figures are lower bounds in both languages, under a budget nobody else was held to. Raising budgets does not fix this — 65,536 already fails at scale. vLLM's `thinking_token_budget` (which forces `` at a cap so the model must answer) was evaluated and **deliberately not adopted for this round**: the experiments were finished under one consistent policy first. Notes for a future round are in §7. The `-nothink` runs truncate essentially nowhere: the worst cell across all five models is 22/448 (4.9%) — DeepSeek on Czech GPQA, against the same 8,192 cap — and 16/713 on LCB for Qwen 27B. Every single `empty_answer` count is 0. Whatever those runs measure, it is not budget exhaustion. ### 3.2 Czech number formatting breaks MGSM's answer extraction BenchMAX extracts MGSM's final number with: ``` strict-match "Odpověď je (\-?[0-9\.\,]+)" flexible-extract "(-?[$0-9.,]{2,})|(-?[0-9]+)" group_select: -1 ``` Both misfire here, for two independent reasons: 1. Under thinking mode the model does not reproduce the `Odpověď je …` template, because the CoT is returned in a separate `reasoning` field and the visible content is a clean summary. **strict-match therefore scores ~0** and everything falls through to flexible-extract. (This is why the viewer keeps the filter name attached to every metric — collapsing `strict-match` and `flexible-extract` into one "exact_match" would hide a 0.00 vs 0.84 disagreement.) 2. flexible-extract cannot read Czech numbers. Czech writes thousands with a space or NBSP: `70 000`. The pattern matches `70` and `000` as two separate numbers and `group_select: -1` keeps the last, so a correct 70000 is recorded as `000`. 9.2% of Czech MGSM items have |answer| ≥ 1000. The viewer (in `viewer/`, its own nested git repo) reports a locale-aware third metric **alongside** the two native ones. It does not modify the harness: the published-protocol number and the corrected number are both visible. Over the finished round (`python viewer/rescore.py results//lmeval/mgsm`), native `flexible-extract` → corrected, with `strict-match` at 0.000 for every model in both languages: | model | cs | en | |---|---|---| | qwen3.6-27b | 83.2 → **92.4** | 89.6 → 95.6 | | qwen3.6-35b-a3b | 72.8 → **82.8** | 86.4 → 92.0 | | gemma-4-31b | 82.4 → **94.0** | 87.6 → 93.2 | | gemma-4-e4b | 74.4 → **84.4** | 79.6 → 86.0 | | deepseek-v4-flash | 82.4 → **89.2** | 87.6 → 92.4 | English gains too (+4.8 to +6.4), because thinking mode breaks the `Odpověď je …` / `The answer is …` template in both languages and flexible-extract then mis-reads grouped digits and trailing punctuation. Czech gains roughly twice as much (+6.8 to +11.6), which is the thousands-separator effect on top. **A good part of the measured cs↔en MGSM gap is therefore an extraction artifact**: it falls from 6.4 to 3.2 points for the 27B, from 5.2 to 1.6 for gemma-4-e4b, and for gemma-4-31b it inverts (82.4/87.6 becomes 94.0/93.2, Czech ahead). Only the 35B-A3B keeps a large gap, 13.6 → 9.2, and that model is also the one truncating 6.4% of its Czech MGSM CoTs (§3.1). ### 3.3 Empty answers are truncations — there is no second effect An earlier version of this section claimed that Czech yields more empty answers *independently* of truncation. The finished round says otherwise. Over the six prompt-language tasks, 2,322 requests per model per language: | model | cs empty | en empty | empty but **not** truncated | |---|---|---|---| | deepseek-v4-flash | 370 (15.9%) | 314 (13.5%) | **0** | | qwen3.6-35b-a3b | 360 (15.5%) | 213 (9.2%) | **0** | | qwen3.6-27b | 270 (11.6%) | 229 (9.9%) | **0** | | gemma-4-31b | 72 (3.1%) | 60 (2.6%) | **0** | | gemma-4-e4b | 0 | 1 | **0** | Not one empty answer in the round came from anything other than a generation that hit its budget. (The converse happens a little: up to 11 truncations per model and language had emitted a usable answer before the cut.) Czech is emptier than English because Czech truncates more — §3.1, not a separate finding. The telemetry still records both fields per task and language, which is how this could be checked at all. ### 3.4 Two Gemma translation scores are a format failure, and only the first line is scored Three of the ten runs have translation scores that do not mean what they look like — all three Gemma, one of them fatally. They come from a collision between how Gemma answers a translation request and how BenchMAX reads the answer. **BenchMAX scores the first line of the response and discards the rest.** From `tasks/translation/utils.py`: ```python case "flores" | "ted" | "wmt24": outputs[i] = outputs[i].strip().split("\n")[0] ``` That is a sound rule for a model that answers with the translation and nothing else, which is what the other three models do. It is a disaster for a model that answers with a lead-in and then a list: > Here are a few options for the Czech translation, depending on the desired nuance: The lead-in is line one, so the lead-in is the hypothesis, and every candidate translation under it is thrown away before scoring. **Gemma 4 E4B `-nothink` is the extreme case.** Its pooled scores are **0.82 spBLEU cs→en and 0.09 en→cs** — not a weak model, a scoring wipeout. Between 0.6% and 10.5% of its segments per dataset survive the rule with an actual translation; 94.7–99.5% of its stored hypotheses are a single line ending in a colon, median 82 characters, while telemetry recorded a median of 967 content characters per response (minimum 89 over 1012 Flores segments). **89–93% of the characters the model generated never reached the scorer.** Nothing else in the telemetry flags it: `finish_reason` is `stop` on 1011 of 1012 Flores en→cs segments, truncation is the remaining one, and `empty_answer` is zero. The only symptom is the score. Its thinking counterpart is unaffected — 31.8 cs→en, 27.7 en→cs pooled — so this is specific to the non-thinking condition of this model. **Gemma 4 31B thinking is the milder case**, and it is why the model's pooled 24.5 cs→en sits below the 8B-class E4B's 31.8. The direction matters: on en→cs it scores 28.2, marginally *above* E4B's 27.7 — and en→cs is also where it keeps its answer to one line most often. Compliance tracks the score direction by direction. `scripts/diagnose_translation_format.py` rescores the compliant subset: | dataset · direction | compliant | spBLEU (all) | spBLEU (compliant) | |---|---|---|---| | flores en→cs | 93.6% | 39.23 | 42.13 | | flores cs→en | 67.9% | 31.80 | 49.61 | | ted en→cs | 88.2% | 21.88 | 24.45 | | ted cs→en | 63.9% | 14.72 | 27.46 | | wmt24 en→cs | 75.1% | 22.69 | 29.09 | On the segments where it answers in one line, flores cs→en is 49.6 spBLEU — the highest of any model in this round. Measured the same way as above, the first-line rule discards 68–76% of the characters this model generated. Its own `-nothink` run, which does not do this, scores 37.1 cs→en against the thinking run's 24.5. **It is a Gemma problem, and only a Gemma problem.** Compliance per run, worst dataset first: | run | compliance range | cost | |---|---|---| | gemma-4-e4b-nothink | **0.6 – 10.5%** | total; the score is meaningless | | gemma-4-31b (thinking) | 63.9 – 93.6% | 2.6 – 17.8 spBLEU per set | | gemma-4-31b-nothink | 88.7 – 99.9% | 0.03 – 1.9 spBLEU per set | | gemma-4-e4b (thinking) | 98.3 – 100% | ≤ 0.4 | | qwen3.6-27b, qwen3.6-35b-a3b, ±nothink | 98.2 – 100% | ≤ 0.6 | | deepseek-v4-flash, ±nothink | 99.8 – 100% | 0.00 | Both Qwen models and DeepSeek answer with a bare translation essentially always, so the first-line rule costs them nothing and the comparison between *them* is sound. Three of the ten runs are affected, all Gemma, and one of them fatally. What cannot be checked from what is on disk: whether the discarded lines were good translations. Only the first line is persisted, and the model servers have been released. The character counts and the preamble wording make "preamble + list of candidate translations" the only plausible reading, but the compliant-subset scores above remain the defensible estimate of what these models can translate, and they are computed over an easier, self-selected subset. Nothing was worked around. The comparison table keeps the unforgiving number, because changing `post_process` would make our translation protocol differ from the published one — the same reason the reference rows in §6 stay as upstream computed them. The honest summary is that BenchMAX's translation metric measures translation *and* one-line answer formatting, and for the affected Gemma runs the second term dominates. §7 lists what a next round should do about it. --- ## 4. Patches to upstream code All patches live in `benchmax_repo/` (the BenchMAX checkout) or are applied to the venv by `scripts/patch_venv.py`. Each carries an in-place comment explaining itself. **Two were scoring bugs that invalidated results outright.** ### 4.1 Correctness fixes **GPQA gold/prompt misalignment — `tasks/gpqa/native_cot_zeroshot/utils.py`.** `process_docs` shuffled the answer options with an unseeded `random.shuffle`. lm-eval calls `process_docs` more than once per run — once when prompts are built, again on the path that supplies the gold label — so each invocation produced a different permutation, and the letter stored in `answer` referred to an ordering the model was never shown. Gold agreed with the prompt in only 192/896 cases, pinning the score to chance (~25%) regardless of actual ability. Fixed with a per-document seeded RNG keyed on the stable `Record ID`, which keeps the permutation fixed within a run and reproducible across runs while still varying the correct letter between questions. Recovered true scores: 27B cs 65.62 / en 74.78; 35B-A3B cs 55.58 / en 72.77 (measured 25.7% before the fix). **evalplus dropped `max_new_tokens` — `tasks/evalplus/evalplus/provider/__init__.py`.** Every backend forwarded the parameter except the `openai` branch, so `--max_new_tokens` was accepted on the command line and then silently replaced by `DecoderBase`'s default of 768. A reasoning model spends that entirely inside the CoT and returns `content=None`, scoring a flat **0.000 pass@1 on all 164 tasks**. ### 4.2 Serving against a self-hosted endpoint - **Per-request extras** — `analysis/chatkwargs.py`, installed into site-packages as `bm_chatkwargs` and wired into lm-eval's `_create_payload` by `scripts/patch_venv.py`, plus the four SDK-based harnesses. It carries two shapes, because the reasoning switch is not the same field everywhere (§2.2): `BM_CHAT_TEMPLATE_KWARGS` becomes a nested `chat_template_kwargs` object, and `BM_EXTRA_BODY` merges into the top level of the request body. Both go through the same two entry points (`extra_body()`, `payload_extra()`), so adding the second shape needed no new call sites and no new patches. lm-eval's `--gen_kwargs` CLI only parses flat `key=value` pairs and cannot express a nested JSON object, which is why this needs a source patch rather than a flag. - **`bm_check_endpoint` sends `Authorization`** — the local vLLM servers ignore it, but the e-infra gateway 401s `/models` without it, which read as "server down". Same fix in `scripts/probe_endpoint.py`. It also **waits** for the endpoint (every 30s up to `BM_ENDPOINT_WAIT`, default 1800s) instead of exiting on the first failed probe, which had turned one gateway outage into eight instantly-dead queued jobs (§1.1). - **LiveCodeBench** — `lcb_runner/runner/oai_runner.py` hardcoded `api.openai.com` and read only `OPENAI_KEY`; now honours `OPENAI_BASE_URL` / `OPENAI_API_KEY`. `lcb_runner/lm_styles.py` registers models from `LCB_EXTRA_MODELS` so no source edit is needed per model. It also received the telemetry hook and the request extras — without the latter, LCB would have been the one task where Gemma silently ran non-thinking. - **Client timeouts** — the OpenAI client defaults to 600 s with 2 retries. A single thinking-mode request can sit behind hundreds of others when several jobs share an endpoint; `gemma-4-31B` translation died on `APITimeoutError` after an hour of otherwise-fine generation. Now 3600 s / 5 retries in nexus, evalplus, translation and LCB, and `BM_REQUEST_TIMEOUT=3600` for lm-eval (whose tenacity wrapper has `reraise=True`, so a timeout kills the whole run rather than dropping one sample). One Czech GPQA item was measured at 543 s on an *idle* server. - **Patient rate-limit retry** — `analysis/retry.py`, installed as `bm_retry`, wrapped around the `create()` calls in nexus and translation. The client's own `max_retries` is inert against a gateway that sends `retry-after: 1800`, and those two harnesses lose a whole batch to one exception (§1.1). Tunable with `BM_RETRY_ATTEMPTS` / `BM_RETRY_WAIT`; `BM_RETRY_ATTEMPTS=1` restores the old fail-fast behaviour. - **lm-eval's own retry loop** — `patch_venv.py` binds `outputs = None` before the `try` in `api_models.amodel_call` (its error handler otherwise masks every pre-response exception with an `UnboundLocalError`) and makes the tenacity backoff cap settable via `BM_LMEVAL_WAIT_MAX`, which is the only way to buy more than ~5 minutes of patience without tens of thousands of log lines (§1.1). - **`None`-content guards** — nexus, translation and LCB all called `.strip()` or `.split()` on `message.content`, which is `None` when the budget is exhausted mid-CoT. One such response aborted an entire run and discarded every generation already produced. All now coerce to `""`, so the failure costs one sample instead of the job. - **`sentencepiece`** — required by sacrebleu's `flores200` tokenizer for spBLEU. Missing it lets generation succeed and then kills scoring. --- ## 5. Telemetry The harnesses keep only the final answer, so without this there is no way to report how long a model thinks, nor to tell a wrong answer apart from one that was truncated mid-CoT. `analysis/telemetry.py` (installed as `bm_telemetry`) records one JSONL row per generated choice into `results//telemetry/_.jsonl`: ``` finish_reason, prompt_tokens, completion_tokens, reasoning_tokens, reasoning_chars, content_chars, empty_answer, n_tool_calls, task, lang, t ``` `bm_telemetry_for ` in `scripts/common.sh` points it at a fresh file and truncates, so re-running a task replaces its telemetry rather than appending. `content_chars` earns its place twice over: it is the only record of how much text a response contained *before* a harness reduced it, which is what exposed the first-line translation rule in §3.4 — the generations on disk cannot show you what was thrown away, and the telemetry can. Two consequences worth knowing: - **`run_lmeval.sh` invokes `lm_eval` once per language** rather than passing a comma-separated task list. The telemetry hook sits in `parse_generations`, which sees the raw response but not which task it belongs to, so one language per process is the only way to attribute reasoning length correctly. Inference dominates, so the extra invocation is free. - **Re-running translation can empty its telemetry.** `generate_translation.py` skips a direction whose output file already exists ("Skipping cs-en as the output file already exists"), but `bm_telemetry_for` has already truncated the telemetry file by then — so the results survive and the telemetry does not. This is why `qwen3.6-35b-a3b`'s two `translation-flores_*.jsonl` files are 0 bytes while its Flores scores are present — still the case at the end of the round; that model contributes 1997/1000 rows to the reasoning-length figure where the others contribute 3009/2012, and the figure footnotes it. Delete the generations too if you want the telemetry back. --- ## 6. Published baselines `scripts/fetch_reference.py` vendors selected rows from [CONE-MT/BenchMAX/results](https://github.com/CONE-MT/BenchMAX/tree/main/results) into `analysis/reference_benchmax.json`, which the viewer renders alongside our own columns. Vendored rather than fetched live because the viewer runs on a submit node with no outbound network, and a baseline that changed between page loads would be worse than useless. Models carried over: **gpt-4o-mini-2024-07-18**, **DeepSeek-V3** (671B), **Aya-Expanse** (32B), **Llama3.3** (70B). Size disambiguates — Aya-Expanse and Llama3.x each appear at several scales upstream. Superimposing numbers only means something if both sides are the same measurement. Three metric definitions had to be harmonised: - **IFEval** is published as the **mean of four accuracies** (prompt-strict, prompt-loose, inst-strict, inst-loose), per the paper's appendix. None of our four separate rows is comparable on its own. - **HumanEval** is published as **HumanEval+** (paper Table 3), i.e. the extended test suite. evalplus counts a problem as passing only when it passes *both* base and plus, and the comparison uses that. - **Translation** is published as spBLEU over the **concatenation** of Flores + TED (+ WMT24 for en→cs), scored in one pass — not any single dataset, and not the average of their scores, since the sets differ in size and segment length. `run_translation.sh pool` reproduces it by re-reading generations already on disk, at no request cost. MGSM and GPQA use lm-eval's `flexible-extract` filter **without** the Czech correction from §3.2 in that table, as the closest match to upstream. The corrected figures are higher and appear in the per-model tables. Three of our rows in that table carry caveats the published rows do not, and none of them is visible in the number itself: DeepSeek's IFEval, MGSM and GPQA were generated under an 8,192-token cap (§1.1), and the Gemma translation cells are dominated by answer format rather than translation quality (§3.4). The comparison is honest about the protocol and misleading about the models — read those cells with §3 open. --- ## 7. Not done this round — the queue for the next one The round is closed; nothing below was attempted after the fact, because re-running one model under changed conditions would have broken the one thing this setup is for (§2). In rough order of how much they would change the numbers: - **Send `max_completion_tokens` to e-infra.** DeepSeek's IFEval, MGSM, GPQA and LCB ran on 8,192 tokens instead of the configured budgets, and truncated a quarter of GPQA and LCB for it (§1.1, §3.1). The fix is one field name in lm-eval's `_create_payload` and in `lcb_runner/runner/oai_runner.py`, both of which are already patched files — but it invalidates four DeepSeek tasks in both conditions, so it is a re-run, not an edit. - **Decide what to do about the first-line translation rule** (§3.4). Three of ten runs have translation scores distorted by it, one of them totally (Gemma 4 E4B `-nothink`, 0.09 spBLEU en→cs). Three options, none free: keep upstream's rule and report the compliance rate beside every translation score; keep the full response and score the best line, which stops being BenchMAX; or persist the full generations — cheap, purely additive, and it would at least make the question answerable next time. **Persisting the generations is worth doing regardless**, since this round cannot answer it. - **`thinking_token_budget`.** vLLM can cap the CoT and force ``, turning a truncated non-answer into a real answer. Verified working on the Qwen endpoints via `extra_body` with no server restart (`scripts/probe_thinking_budget.py`), and verified to recover genuinely runaway Czech IFEval prompts. **Not adopted** — the round was run under one consistent policy. Given how much of LiveCodeBench it would recover (§3.1), it is the first thing to revisit on the local models. If revisited, note: - `--reasoning-config` is *not* required when the reasoning parser already declares its boundary strings, which `qwen3` does. - The Gemma servers reject it with `thinking_token_budget is not yet supported by the V2 model runner` and would need relaunching with `VLLM_USE_V2_MODEL_RUNNER=0`. - Gemma 4 uses a channel scheme, not ``/``: from its chat template, `<|think|>` is a *gate* injected into the system turn, while the CoT is delimited `<|channel>` … ``. - It does not exist on the e-infra gateway at all; `reasoning_effort: "low"` is the nearest available knob. - Nexus must be exempted — [vLLM #44676](https://github.com/vllm-project/vllm/issues/44676) injects the forced end-token into the middle of `` arguments. - **Arena-Hard.** `run_arenahard.sh` exists and is unused. The blocker was a judge API key; the e-infra key now in use would technically serve, but judging ~500 pairwise comparisons per model at 2 concurrent requests is its own multi-day job, and a self-hosted judge of this class is not the judge the published numbers used. Still postponed, now for a different reason. - **RULER** — dropped, see §1. - **Domain translation** (`run_translation.sh domain`) — the script works; the 7 in-benchmark sets have not been run for any model. They are exempt from the §3.4 problem, though: `post_process` applies the first-line rule only to `flores | ted | wmt24`, and the domain task names fall through it. --- ## 8. Running it ### Environment Any virtual environment will do — the runs were made from one built with `lm_eval==0.4.9.1` and the harness dependencies of the BenchMAX checkout: ```bash python -m venv .venv && . .venv/bin/activate pip install lm_eval==0.4.9.1 evalplus sacrebleu sentencepiece openai python scripts/patch_venv.py # install bm_* modules, patch lm_eval ``` `patch_venv.py` is idempotent, and must be re-run after editing `analysis/telemetry.py`, `analysis/chatkwargs.py` or `analysis/retry.py` — the harnesses import the site-packages copy, not the one in the repo. If it reports that lm-eval's source no longer matches, force-reinstall rather than trying to patch around it: ```bash uv pip install --force-reinstall --no-deps lm_eval==0.4.9.1 ``` Both helper modules are installed into site-packages rather than imported from the repo because nltk's `inisec.py` sets `PYTHONSAFEPATH=1`, which drops the script directory from `sys.path`. ### Submitting jobs Inference happens on the model server, so the client jobs need no GPU — but they must outlive an ssh session, hence sbatch rather than the submit node. One task at a time: ```bash BM_PRESET=qwen3.6-27b ./scripts/slurm/submit.sh run_lmeval.sh ifeval BM_PRESET=qwen3.6-27b ./scripts/slurm/submit.sh run_lmeval.sh mgsm gpqa BM_PRESET=qwen3.6-27b ./scripts/slurm/submit.sh run_nexus.sh ``` A whole model, chained so only one job hits the endpoint at a time (cheapest task first, so a broken preset surfaces in minutes rather than after the multi-hour code benchmarks): ```bash BM_PRESET=qwen3.6-27b ./scripts/slurm/submit_all.sh ``` Both conditions of one model, strictly sequential — the second suite is chained behind the last job of the first. This is required for e-infra, where the two suites would otherwise compete for the same 2-request budget: ```bash export BM_TIME=7-0 last=$(BM_PRESET=deepseek-v4-flash ./scripts/slurm/submit_all.sh 2>&1 | sed -n 's/^last job: //p') BM_AFTER=$last BM_PRESET=deepseek-v4-flash-nothink ./scripts/slurm/submit_all.sh ``` ### Concurrency `BM_CONCURRENCY` is **per job**, and concurrent jobs add up. The local server was measured at ~4.5k tok/s aggregate at concurrency 256 with zero failures and still scaling at 128, but four jobs at 128 put 512 requests in flight and pushed lm-eval into visible retry storms. The default of 64 keeps a typical 3–4 job fan-out inside the measured envelope. The e-infra presets set 2 against a server-side cap of 4 (§1.1); do not raise it — 3 passed a short probe and still lost two tasks to a 429 over a full run. ### Checking an endpoint before a long run ```bash python scripts/probe_endpoint.py http://host:port/v1 --model # --sweep also times throughput python scripts/probe_endpoint.py https://llm.ai.e-infra.cz/v1 --model deepseek-v4-flash --api-key "$KEY" ``` Verifies the served model id (vLLM matches it exactly and 404s otherwise — note `gemma-4-31B-it` has a capital B), `max_model_len`, where the reasoning field lands, whether `` leaks into content, whether usage is populated, and whether tool calls come back structured. Against a gateway, probe with a **unique prompt each time**: e-infra caches responses, and a cached reply from an earlier thinking request will make a non-thinking configuration look like it is thinking (§1.1). --- ## 9. Layout ``` scripts/ common.sh shared config: endpoint, budgets, paths, venv, telemetry models/*.env per-model presets (X and X-nothink for each model) models/_einfra.inc.sh shared setup for the e-infra gateway: key, cap, timeouts run_*.sh one script per task slurm/submit.sh sbatch wrapper (cpu-ms,cpu-troja; 16 cores for lcb/evalplus) slurm/submit_all.sh one model's whole suite as a chain of dependent jobs patch_venv.py installs bm_telemetry/bm_chatkwargs/bm_retry, patches lm_eval fetch_reference.py vendors published BenchMAX numbers probe_*.py endpoint / thinking-budget / truncation diagnostics diagnose_translation_format.py compliance split for §3.4 analysis/ telemetry.py per-request generation telemetry chatkwargs.py per-model request extras (chat_template_kwargs, body fields) retry.py patient rate-limit retry the openai SDK will not do plot_reasoning_length.py reasoning-length figure for write-up (--table for numbers) figures/ its output: reasoning_length.{pdf,png} reference_benchmax.json viewer/ its own nested git repo (not a submodule): the viewer viewer.py live HTML results browser (needs Python running) rescore.py Czech-aware MGSM/GPQA re-extraction build.py regenerates the static index.html/data from results/ index.html, app.js, style.css, data/ the built static viewer results// lmeval/ nexus/ evalplus/ lcb/ translation/ telemetry/ 10 tags: 5 models x {thinking, -nothink}, all complete docs/PROTOCOL.md this file ``` Two directories from the working tree are **not** published: `benchmax_repo/`, the patched BenchMAX checkout (§4 describes every patch; the upstream repository is the place to apply them), and `archive/`, which holds superseded runs — pre-telemetry, pre-GPQA-fix, and the Gemma runs made before `enable_thinking` was understood (§2.2). Nothing in either is a result of this round. The per-task run logs are also left out; the telemetry under `results//telemetry/` is the part of them that carries information (§5). ### Viewer Two front ends over the same reduction, and the reduction lives in one place: `viewer/viewer.py`. `viewer/build.py` imports it rather than restating it, so the static build cannot drift from the live one. The whole thing — viewer, build, and the live server — is its own nested git repository under `viewer/`. **Static, no server** — what this repository ships, and what [`viewer/index.html`](../viewer/index.html) is: ```bash python viewer/build.py # rebuild after results/ changes; ~30 s ``` It writes the overall tab as flat HTML and the per-benchmark browsers as JavaScript data files, loaded by `