BenchMAX · Czech

vs. published BenchMAX results

Ours (left) against the numbers published in CONE-MT/BenchMAX/results (right, shaded). Metrics are harmonised per row. Each model spans two sub-columns: thinking is the primary run, non-thinking is the same model, endpoint and token budget with reasoning disabled.

Ours run with thinking enabled and much larger token budgets; the published numbers are non-thinking runs of older models. Treat the comparison as indicative, not like-for-like.
MGSM and GPQA use lm-eval's flexible-extract filter without the Czech number-format correction, which is the closest match to upstream. The locale-corrected figures are higher — see the per-benchmark tabs.
Translation is the pooled general-translation score over flores + TED (+ WMT24 for en→cs), matching BenchMAX's evaluate_general_translation.py. Per-dataset scores are in the Translation tab.
gemma-4-31b's translation row understates its translation quality: 6–36% of its answers are a meta-preamble with no translation under it (not truncation — finish_reason is stop). On compliant segments it leads the round. See README §3.4.

benchmark	lang	deepseek-v4-flash		gemma-4-31b		gemma-4-e4b		qwen3.6-27b		qwen3.6-35b-a3b		gpt-4o-mini	DeepSeek-V3	Aya-Expanse-32B	Llama3.3-70B
benchmark	lang	thinking	non-thinking	thinking	non-thinking	thinking	non-thinking	thinking	non-thinking	thinking	non-thinking	gpt-4o-mini	DeepSeek-V3	Aya-Expanse-32B	Llama3.3-70B
IFEval (avg of 4)	en	92.1	91.9	95.9	95.6	93.8	92.3	90.5	92.7	90.5	90.9	86.9	91.8	77.0	94.6
IFEval (avg of 4)	cs	82.5	87.2	88.2	90.8	87.2	83.7	87.3	88.3	84.8	84.1	78.7	85.1	67.3	86.3
MGSM	en	87.6	92.0	87.6	85.6	79.6	78.0	89.6	87.2	86.4	86.4	82.0	87.2	88.0	84.8
MGSM	cs	82.4	82.4	82.4	80.4	74.4	70.4	83.2	82.4	72.8	77.2	78.0	81.2	74.8	86.8
GPQA	en	67.9	62.9	78.6	73.7	53.6	52.5	78.6	79.7	76.6	77.0	38.2	54.9	28.3	54.0
GPQA	cs	61.8	65.8	71.7	66.3	48.9	44.9	67.6	75.0	54.7	73.7	30.4	49.8	25.9	49.5
Nexus (tool use)	en	70.1	61.3	80.8	78.0	78.3	69.2	74.8	79.2	59.1	40.6	76.7	71.7	68.6	44.3
Nexus (tool use)	cs	64.5	56.6	78.3	76.1	75.2	65.7	67.9	73.3	51.9	34.9	71.7	67.9	61.3	41.8
HumanEval+	en	93.9	89.6	95.1	95.1	81.7	84.8	79.9	92.7	91.5	90.9	82.9	87.2	64.6	78.0
HumanEval+	cs	90.9	88.4	93.3	91.5	75.6	82.3	80.5	92.7	83.5	91.5	79.9	86.0	58.5	74.4
LiveCodeBench	en	63.0	76.9	84.0	87.0	73.9	67.6	69.3	71.4	70.3	80.2	39.3	62.2	17.7	37.1
LiveCodeBench	cs	62.4	76.3	84.7	85.6	71.8	66.8	70.3	79.5	66.5	78.7	38.9	61.4	18.0	36.7
Translation en→cs (spBLEU)		38.4	38.6	28.2	25.8	27.7	0.1	34.6	31.4	34.1	31.4	33.8	38.3	34.7	34.8
Translation cs→en (spBLEU)		38.3	38.6	24.5	37.1	31.8	0.8	37.6	34.8	36.9	34.2	36.9	37.6	38.6	36.6

models side by side

task	metric	deepseek-v4-flash		gemma-4-31b		gemma-4-e4b		qwen3.6-27b		qwen3.6-35b-a3b
task	metric	thinking	non-thinking	thinking	non-thinking	thinking	non-thinking	thinking	non-thinking	thinking	non-thinking
xgpqa_main_native_cot_zeroshot_cs	exact_match [flexible-extract]	61.8	65.8	71.7	66.3	48.9	44.9	67.6	75.0	54.7	73.7
xgpqa_main_native_cot_zeroshot_cs	exact_match [strict-match]	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
xgpqa_main_native_cot_zeroshot_en	exact_match [flexible-extract]	67.9	62.9	78.6	73.7	53.6	52.5	78.6	79.7	76.6	77.0
xgpqa_main_native_cot_zeroshot_en	exact_match [strict-match]	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
xifeval_cs	inst_level_loose_acc	84.0	91.3	90.2	93.4	90.5	87.2	90.5	91.8	86.7	88.5
xifeval_cs	inst_level_strict_acc	82.0	87.2	88.2	91.8	88.3	85.8	88.5	88.8	85.2	85.3
xifeval_cs	prompt_level_loose_acc	83.4	87.9	88.8	90.2	86.5	81.8	86.7	88.1	84.8	83.4
xifeval_cs	prompt_level_strict_acc	80.4	82.3	85.8	87.9	83.4	80.0	83.7	84.4	82.5	79.3
xifeval_en	inst_level_loose_acc	93.2	94.5	96.8	97.2	96.1	94.6	92.7	95.6	92.3	94.0
xifeval_en	inst_level_strict_acc	91.6	92.4	95.9	95.6	93.8	92.6	90.8	92.4	90.4	91.0
xifeval_en	prompt_level_loose_acc	93.0	91.8	96.0	95.8	94.2	92.1	90.7	93.5	91.1	91.4
xifeval_en	prompt_level_strict_acc	90.7	89.0	94.9	93.9	91.1	90.0	87.9	89.3	88.3	87.2
xmgsm_native_cot_cs	exact_match [flexible-extract]	82.4	82.4	82.4	80.4	74.4	70.4	83.2	82.4	72.8	77.2
xmgsm_native_cot_cs	exact_match [strict-match]	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
xmgsm_native_cot_en	exact_match [flexible-extract]	87.6	92.0	87.6	85.6	79.6	78.0	89.6	87.2	86.4	86.4
xmgsm_native_cot_en	exact_match [strict-match]	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
nexus_cs	accuracy	64.5	56.6	78.3	76.1	75.2	65.7	67.9	73.3	51.9	34.9
nexus_en	accuracy	70.1	61.3	80.8	78.0	78.3	69.2	74.8	79.2	59.1	40.6
humaneval_cs	pass@1(base)	96.3	93.3	97.6	93.9	78.7	86.0	84.8	97.0	84.8	96.3
humaneval_cs	pass@1(plus)	90.9	88.4	93.3	91.5	75.6	82.3	80.5	92.7	83.5	91.5
humaneval_en	pass@1(base)	97.6	95.1	99.4	97.6	86.0	90.9	84.1	97.6	95.7	95.7
humaneval_en	pass@1(plus)	93.9	89.6	95.1	95.1	81.7	84.8	79.9	92.7	91.5	90.9
lcb_cs	pass@1	62.4	76.3	84.7	85.6	71.8	66.8	70.3	79.5	66.5	78.7
lcb_en	pass@1	63.0	76.9	84.0	87.0	73.9	67.6	69.3	71.4	70.3	80.2
flores cs-en	ChrF	72.3	72.5	53.5	71.6	67.1	19.5	71.6	69.4	71.1	69.1
flores cs-en	spBLEU	46.1	46.1	31.8	45.7	40.7	1.2	45.9	41.4	45.2	40.5
flores en-cs	ChrF	66.1	66.0	60.2	63.5	59.1	12.4	63.1	62.2	62.9	61.9
flores en-cs	spBLEU	45.0	44.8	39.2	40.9	35.1	0.0	41.6	38.3	41.3	37.6
general_translation cs-en	ChrF	66.8	66.9	47.4	64.8	60.0	18.4	66.1	64.0	65.6	63.6
general_translation cs-en	spBLEU	38.3	38.6	24.5	37.1	31.8	0.8	37.6	34.8	36.9	34.2
general_translation en-cs	ChrF	61.9	62.2	49.1	53.9	53.1	9.9	58.6	58.0	58.2	58.0
general_translation en-cs	spBLEU	38.4	38.6	28.2	25.8	27.7	0.1	34.6	31.4	34.1	31.4
ted cs-en	ChrF	58.8	58.9	38.8	55.1	49.7	17.0	58.2	56.3	57.8	55.6
ted cs-en	spBLEU	26.0	26.2	14.7	24.1	19.4	0.0	25.7	23.4	24.9	22.8
ted en-cs	ChrF	55.7	55.9	48.7	51.9	48.4	11.9	53.1	51.8	53.3	52.1
ted en-cs	spBLEU	26.6	26.4	21.9	21.4	17.5	0.0	23.5	20.2	23.5	20.8
wmt24 en-cs	ChrF	61.8	62.4	41.2	49.0	51.0	7.1	58.0	58.0	57.0	57.9
wmt24 en-cs	spBLEU	39.0	39.6	22.7	20.5	26.3	0.1	34.1	31.5	33.3	31.7

reasoning length

Median generated tokens per request — the thinking trace and the answer together, from results/<model>/telemetry/. Bars are the language the model generated in, so a translation direction counts toward its target side.

CzechEnglisheach benchmark scaled to its own maximum

average over benchmarksscale 0–7,380 tok

thinking

qwen3.6-27b

cs7,380

en6,040

qwen3.6-35b-a3b

cs7,304

en4,812

gemma-4-31b

cs2,043

en2,149

gemma-4-e4b

cs1,609

en1,427

deepseek-v4-flash

cs1,228

en1,128

non-thinking

qwen3.6-27b

cs789

en687

qwen3.6-35b-a3b

cs670

en628

gemma-4-31b

cs353

en295

gemma-4-e4b

cs786

en687

deepseek-v4-flash

cs286

en214

GPQAscale 0–15,698 tok

thinking

qwen3.6-27b

cs13,032

en4,477

qwen3.6-35b-a3b

cs15,698

en4,985

gemma-4-31b

cs2,704

en4,142

gemma-4-e4b

cs2,755

en2,445

deepseek-v4-flash

cs2,659

en3,415

non-thinking

qwen3.6-27b

cs2,147

en1,980

qwen3.6-35b-a3b

cs2,135

en1,897

gemma-4-31b

cs924

en734

gemma-4-e4b

cs1,692

en1,483

deepseek-v4-flash

cs727

en415

LiveCodeBenchscale 0–22,968 tok

thinking

qwen3.6-27b

cs22,289

en22,968

qwen3.6-35b-a3b

cs19,434

en19,926

gemma-4-31b

cs7,448

en7,517

gemma-4-e4b

cs3,964

en3,760

deepseek-v4-flash

cs2,317

en1,982

non-thinking

qwen3.6-27b

cs2,464

en2,006

qwen3.6-35b-a3b

cs1,545

en1,647

gemma-4-31b

cs505

en487

gemma-4-e4b

cs1,993

en1,872

deepseek-v4-flash

cs176

en173

HumanEval+scale 0–9,270 tok

thinking

qwen3.6-27b

cs8,801

en9,270

qwen3.6-35b-a3b

cs8,194

en2,922

gemma-4-31b

cs1,167

en1,167

gemma-4-e4b

cs1,649

en1,547

deepseek-v4-flash

cs862

en841

non-thinking

qwen3.6-27b

cs259

en230

qwen3.6-35b-a3b

cs270

en231

gemma-4-31b

cs432

en390

gemma-4-e4b

cs719

en653

deepseek-v4-flash

cs236

en209

MGSMscale 0–1,564 tok

thinking

qwen3.6-27b

cs824

en1,353

qwen3.6-35b-a3b

cs1,564

en1,333

gemma-4-31b

cs701

en513

gemma-4-e4b

cs808

en652

deepseek-v4-flash

cs700

en258

non-thinking

qwen3.6-27b

cs294

en300

qwen3.6-35b-a3b

cs347

en308

gemma-4-31b

cs261

en216

gemma-4-e4b

cs394

en299

deepseek-v4-flash

cs192

en131

IFEvalscale 0–3,054 tok

thinking

qwen3.6-27b

cs3,054

en2,182

qwen3.6-35b-a3b

cs2,919

en2,409

gemma-4-31b

cs1,353

en1,062

gemma-4-e4b

cs1,005

en705

deepseek-v4-flash

cs1,646

en1,038

non-thinking

qwen3.6-27b

cs265

en214

qwen3.6-35b-a3b

cs298

en223

gemma-4-31b

cs261

en175

gemma-4-e4b

cs400

en296

deepseek-v4-flash

cs523

en433

Nexusscale 0–884 tok

thinking

qwen3.6-27b

cs884

en423

qwen3.6-35b-a3b

cs528

en421

gemma-4-31b

cs187

en175

gemma-4-e4b

cs612

en553

deepseek-v4-flash

cs176

en171

non-thinking

qwen3.6-27b

cs59

en58

qwen3.6-35b-a3b

cs60

en69

gemma-4-31b

cs41

en40

gemma-4-e4b

cs43

en42

deepseek-v4-flash

cs110

en118

Translationscale 0–2,788 tok

thinking

qwen3.6-27b

cs2,773

en1,606

qwen3.6-35b-a3b

cs2,788

en1,685

gemma-4-31b

cs739

en470

gemma-4-e4b

cs469

en328

deepseek-v4-flash

cs234

en193

non-thinking

qwen3.6-27b

cs36

en22

qwen3.6-35b-a3b

cs37

en22

gemma-4-31b

cs46

en24

gemma-4-e4b

cs259

en162

deepseek-v4-flash

cs39

en22

full telemetry numbers

task	lang	model	n	median tok	p90	max	total tok	truncated	empty
evalplus	cs	deepseek-v4-flash	164	862	2,070	32,768	276,673	1.8%	1.8%
evalplus	cs	deepseek-v4-flash-nothink	164	236	425	1,008	41,810	0.0%	0.0%
evalplus	cs	gemma-4-31b	164	1,167	1,971	7,472	229,829	0.0%	0.0%
evalplus	cs	gemma-4-31b-nothink	164	432	635	1,037	70,615	0.0%	0.0%
evalplus	cs	gemma-4-e4b	164	1,649	2,357	3,423	272,704	0.0%	0.0%
evalplus	cs	gemma-4-e4b-nothink	164	719	1,471	2,575	137,185	0.0%	0.0%
evalplus	cs	qwen3.6-27b	164	8,801	32,768	32,768	1,943,702	13.4%	13.4%
evalplus	cs	qwen3.6-27b-nothink	164	259	495	2,463	53,207	0.0%	0.0%
evalplus	cs	qwen3.6-35b-a3b	164	8,194	32,768	32,768	1,797,453	14.0%	14.0%
evalplus	cs	qwen3.6-35b-a3b-nothink	164	270	494	2,002	52,069	0.0%	0.0%
evalplus	en	deepseek-v4-flash	164	841	1,879	32,768	299,539	2.4%	1.8%
evalplus	en	deepseek-v4-flash-nothink	164	209	344	828	36,484	0.0%	0.0%
evalplus	en	gemma-4-31b	164	1,167	1,983	10,432	238,732	0.0%	0.0%
evalplus	en	gemma-4-31b-nothink	164	390	544	1,018	63,603	0.0%	0.0%
evalplus	en	gemma-4-e4b	164	1,547	2,297	3,907	260,776	0.0%	0.0%
evalplus	en	gemma-4-e4b-nothink	164	653	1,301	2,204	124,066	0.0%	0.0%
evalplus	en	qwen3.6-27b	164	9,270	32,768	32,768	2,050,344	15.9%	15.9%
evalplus	en	qwen3.6-27b-nothink	164	230	443	1,716	46,074	0.0%	0.0%
evalplus	en	qwen3.6-35b-a3b	164	2,922	9,868	32,768	784,996	3.7%	3.7%
evalplus	en	qwen3.6-35b-a3b-nothink	164	231	435	2,802	50,276	0.0%	0.0%
gpqa	cs	deepseek-v4-flash	448	2,659	8,192	8,192	1,734,680	25.9%	25.4%
gpqa	cs	deepseek-v4-flash-nothink	448	727	2,315	8,192	575,928	4.9%	0.0%
gpqa	cs	gemma-4-31b	448	2,704	8,157	65,536	1,909,200	0.4%	0.4%
gpqa	cs	gemma-4-31b-nothink	448	924	1,350	4,343	450,149	0.0%	0.0%
gpqa	cs	gemma-4-e4b	448	2,755	4,603	10,102	1,346,885	0.0%	0.0%
gpqa	cs	gemma-4-e4b-nothink	448	1,692	3,029	7,551	856,910	0.0%	0.0%
gpqa	cs	qwen3.6-27b	448	13,032	48,592	65,536	8,388,888	9.8%	9.8%
gpqa	cs	qwen3.6-27b-nothink	448	2,147	6,972	65,536	1,561,713	0.7%	0.0%
gpqa	cs	qwen3.6-35b-a3b	448	15,698	65,536	65,536	11,449,287	19.6%	19.6%
gpqa	cs	qwen3.6-35b-a3b-nothink	448	2,135	7,461	65,536	1,771,217	1.3%	0.0%
gpqa	en	deepseek-v4-flash	448	3,415	8,192	8,192	1,873,593	25.4%	24.6%
gpqa	en	deepseek-v4-flash-nothink	448	415	1,005	8,192	258,412	0.4%	0.0%
gpqa	en	gemma-4-31b	448	4,142	10,857	65,536	2,641,102	0.7%	0.7%
gpqa	en	gemma-4-31b-nothink	448	734	1,108	8,191	366,854	0.0%	0.0%
gpqa	en	gemma-4-e4b	448	2,445	4,887	65,536	1,334,792	0.2%	0.2%
gpqa	en	gemma-4-e4b-nothink	448	1,483	2,675	5,099	740,318	0.0%	0.0%
gpqa	en	qwen3.6-27b	448	4,477	10,889	65,536	2,769,764	0.2%	0.2%
gpqa	en	qwen3.6-27b-nothink	448	1,980	7,535	65,536	1,503,525	0.4%	0.0%
gpqa	en	qwen3.6-35b-a3b	448	4,985	12,936	65,536	3,563,127	2.5%	2.5%
gpqa	en	qwen3.6-35b-a3b-nothink	448	1,897	7,704	65,536	1,458,872	0.2%	0.0%
ifeval	cs	deepseek-v4-flash	429	1,646	8,192	8,192	1,075,371	12.1%	11.2%
ifeval	cs	deepseek-v4-flash-nothink	429	523	1,645	6,551	323,354	0.0%	0.0%
ifeval	cs	gemma-4-31b	429	1,353	3,624	16,384	1,046,546	5.8%	5.8%
ifeval	cs	gemma-4-31b-nothink	429	261	892	3,094	163,447	0.0%	0.0%
ifeval	cs	gemma-4-e4b	429	1,005	1,762	16,384	499,148	0.5%	0.0%
ifeval	cs	gemma-4-e4b-nothink	429	400	1,089	16,384	226,691	0.2%	0.0%
ifeval	cs	qwen3.6-27b	429	3,054	7,345	16,384	1,686,478	4.0%	4.0%
ifeval	cs	qwen3.6-27b-nothink	429	265	1,052	6,748	190,290	0.0%	0.0%
ifeval	cs	qwen3.6-35b-a3b	429	2,919	8,099	16,384	1,788,991	7.0%	7.0%
ifeval	cs	qwen3.6-35b-a3b-nothink	429	298	1,004	16,384	255,155	0.9%	0.0%
ifeval	en	deepseek-v4-flash	429	1,038	3,317	8,192	702,632	5.1%	4.9%
ifeval	en	deepseek-v4-flash-nothink	429	433	1,435	7,715	275,111	0.0%	0.0%
ifeval	en	gemma-4-31b	429	1,062	2,749	16,384	651,066	0.9%	0.9%
ifeval	en	gemma-4-31b-nothink	429	175	657	2,148	118,659	0.0%	0.0%
ifeval	en	gemma-4-e4b	429	705	1,316	16,384	348,587	0.2%	0.0%
ifeval	en	gemma-4-e4b-nothink	429	296	870	3,426	172,481	0.0%	0.0%
ifeval	en	qwen3.6-27b	429	2,182	4,463	16,384	1,205,708	2.6%	2.6%
ifeval	en	qwen3.6-27b-nothink	429	214	820	16,384	217,204	0.9%	0.0%
ifeval	en	qwen3.6-35b-a3b	429	2,409	4,870	16,384	1,362,590	3.7%	3.7%
ifeval	en	qwen3.6-35b-a3b-nothink	429	223	848	16,384	188,849	0.5%	0.0%
lcb	cs	deepseek-v4-flash	713	2,317	8,192	8,192	2,619,702	27.6%	27.5%
lcb	cs	deepseek-v4-flash-nothink	713	176	647	8,192	260,864	0.7%	0.0%
lcb	cs	gemma-4-31b	713	7,448	12,878	32,768	6,086,886	6.3%	6.3%
lcb	cs	gemma-4-31b-nothink	713	505	1,476	32,768	638,876	0.3%	0.0%
lcb	cs	gemma-4-e4b	713	3,964	9,720	20,623	3,465,562	0.0%	0.0%
lcb	cs	gemma-4-e4b-nothink	713	1,993	5,521	9,755	1,757,766	0.0%	0.0%
lcb	cs	qwen3.6-27b	713	22,289	32,768	32,768	15,005,568	27.2%	26.1%
lcb	cs	qwen3.6-27b-nothink	713	2,464	11,603	32,768	3,485,985	2.2%	0.0%
lcb	cs	qwen3.6-35b-a3b	713	19,434	32,768	32,768	13,885,411	28.6%	27.9%
lcb	cs	qwen3.6-35b-a3b-nothink	713	1,545	10,313	32,768	2,866,279	1.5%	0.0%
lcb	en	deepseek-v4-flash	713	1,982	8,192	8,192	2,480,961	25.7%	25.2%
lcb	en	deepseek-v4-flash-nothink	713	173	593	8,192	286,612	1.7%	0.0%
lcb	en	gemma-4-31b	713	7,517	13,840	32,768	6,386,533	7.6%	7.4%
lcb	en	gemma-4-31b-nothink	713	487	1,281	11,652	524,916	0.0%	0.0%
lcb	en	gemma-4-e4b	713	3,760	9,423	20,804	3,336,861	0.0%	0.0%
lcb	en	gemma-4-e4b-nothink	713	1,872	5,461	12,145	1,732,956	0.0%	0.0%
lcb	en	qwen3.6-27b	713	22,968	32,768	32,768	15,950,090	28.3%	26.8%
lcb	en	qwen3.6-27b-nothink	713	2,006	10,903	32,768	3,349,966	1.5%	0.0%
lcb	en	qwen3.6-35b-a3b	713	19,926	32,768	32,768	13,731,989	26.1%	25.1%
lcb	en	qwen3.6-35b-a3b-nothink	713	1,647	10,920	32,768	3,075,266	1.1%	0.0%
mgsm	cs	deepseek-v4-flash	250	700	2,183	8,192	308,240	3.6%	3.6%
mgsm	cs	deepseek-v4-flash-nothink	250	192	338	710	53,581	0.0%	0.0%
mgsm	cs	gemma-4-31b	250	701	1,056	9,116	204,939	0.0%	0.0%
mgsm	cs	gemma-4-31b-nothink	250	261	422	625	71,744	0.0%	0.0%
mgsm	cs	gemma-4-e4b	250	808	1,244	3,467	219,315	0.0%	0.0%
mgsm	cs	gemma-4-e4b-nothink	250	394	700	1,450	112,754	0.0%	0.0%
mgsm	cs	qwen3.6-27b	250	824	2,111	16,384	318,117	0.4%	0.4%
mgsm	cs	qwen3.6-27b-nothink	250	294	532	3,350	88,423	0.0%	0.0%
mgsm	cs	qwen3.6-35b-a3b	250	1,564	8,325	16,384	825,261	6.4%	6.4%
mgsm	cs	qwen3.6-35b-a3b-nothink	250	347	638	2,291	101,657	0.0%	0.0%
mgsm	en	deepseek-v4-flash	250	258	962	4,952	118,471	0.0%	0.0%
mgsm	en	deepseek-v4-flash-nothink	250	131	231	368	35,783	0.0%	0.0%
mgsm	en	gemma-4-31b	250	513	799	2,886	144,281	0.0%	0.0%
mgsm	en	gemma-4-31b-nothink	250	216	338	540	56,682	0.0%	0.0%
mgsm	en	gemma-4-e4b	250	652	986	1,907	172,237	0.0%	0.0%
mgsm	en	gemma-4-e4b-nothink	250	299	488	1,356	81,367	0.0%	0.0%
mgsm	en	qwen3.6-27b	250	1,353	2,084	5,197	366,880	0.0%	0.0%
mgsm	en	qwen3.6-27b-nothink	250	300	473	1,888	85,476	0.0%	0.0%
mgsm	en	qwen3.6-35b-a3b	250	1,333	2,050	5,613	349,693	0.0%	0.0%
mgsm	en	qwen3.6-35b-a3b-nothink	250	308	506	1,521	87,125	0.0%	0.0%
nexus	cs	deepseek-v4-flash	318	176	304	903	63,007	0.0%	0.0%
nexus	cs	deepseek-v4-flash-nothink	318	110	180	348	37,223	0.0%	0.0%
nexus	cs	gemma-4-31b	318	187	314	1,210	68,911	0.0%	0.0%
nexus	cs	gemma-4-31b-nothink	318	41	65	303	14,224	0.0%	0.0%
nexus	cs	gemma-4-e4b	318	612	877	1,200	187,013	0.0%	0.0%
nexus	cs	gemma-4-e4b-nothink	318	43	71	1,201	18,247	0.0%	0.0%
nexus	cs	qwen3.6-27b	318	884	1,488	3,511	302,347	0.0%	0.0%
nexus	cs	qwen3.6-27b-nothink	318	59	92	186	19,898	0.0%	0.0%
nexus	cs	qwen3.6-35b-a3b	318	528	1,576	32,768	363,818	1.3%	1.3%
nexus	cs	qwen3.6-35b-a3b-nothink	318	60	98	390	21,719	0.0%	0.0%
nexus	en	deepseek-v4-flash	318	171	298	594	61,014	0.0%	0.0%
nexus	en	deepseek-v4-flash-nothink	318	118	171	487	38,769	0.0%	0.0%
nexus	en	gemma-4-31b	318	175	287	817	64,053	0.0%	0.0%
nexus	en	gemma-4-31b-nothink	318	40	63	102	13,488	0.0%	0.0%
nexus	en	gemma-4-e4b	318	553	819	1,216	174,404	0.0%	0.0%
nexus	en	gemma-4-e4b-nothink	318	42	69	265	15,635	0.0%	0.0%
nexus	en	qwen3.6-27b	318	423	1,219	4,534	195,276	0.0%	0.0%
nexus	en	qwen3.6-27b-nothink	318	58	88	184	19,693	0.0%	0.0%
nexus	en	qwen3.6-35b-a3b	318	421	1,138	32,768	218,058	0.3%	0.3%
nexus	en	qwen3.6-35b-a3b-nothink	318	69	110	390	23,470	0.0%	0.0%
translation-flores	cs-en	deepseek-v4-flash	1012	196	244	319	200,287	0.0%	0.0%
translation-flores	cs-en	deepseek-v4-flash-nothink	1012	27	40	75	28,795	0.0%	0.0%
translation-flores	cs-en	gemma-4-31b	1012	501	794	16,384	595,043	0.3%	0.3%
translation-flores	cs-en	gemma-4-31b-nothink	1012	27	42	320	30,343	0.0%	0.0%
translation-flores	cs-en	gemma-4-e4b	1012	299	447	923	310,834	0.0%	0.0%
translation-flores	cs-en	gemma-4-e4b-nothink	1012	179	302	434	200,199	0.0%	0.0%
translation-flores	cs-en	qwen3.6-27b	1012	1,659	2,649	16,384	1,757,107	0.1%	0.1%
translation-flores	cs-en	qwen3.6-27b-nothink	1012	27	42	84	29,202	0.0%	0.0%
translation-flores	cs-en	qwen3.6-35b-a3b-nothink	1012	27	42	77	29,322	0.0%	0.0%
translation-flores	en-cs	deepseek-v4-flash	1012	232	288	2,219	240,488	0.0%	0.0%
translation-flores	en-cs	deepseek-v4-flash-nothink	1012	45	66	130	47,536	0.0%	0.0%
translation-flores	en-cs	gemma-4-31b	1012	687	983	16,384	846,460	0.8%	0.8%
translation-flores	en-cs	gemma-4-31b-nothink	1012	40	61	16,384	59,867	0.1%	0.0%
translation-flores	en-cs	gemma-4-e4b	1012	462	618	1,052	468,725	0.0%	0.0%
translation-flores	en-cs	gemma-4-e4b-nothink	1012	284	361	16,384	302,904	0.1%	0.0%
translation-flores	en-cs	qwen3.6-27b	1012	2,799	4,052	16,384	3,022,517	1.2%	1.2%
translation-flores	en-cs	qwen3.6-27b-nothink	1012	42	63	127	44,894	0.0%	0.0%
translation-flores	en-cs	qwen3.6-35b-a3b-nothink	1012	42	63	132	45,121	0.0%	0.0%
translation-ted	cs-en	deepseek-v4-flash	1000	190	242	376	192,281	0.0%	0.0%
translation-ted	cs-en	deepseek-v4-flash-nothink	1000	17	38	140	20,350	0.0%	0.0%
translation-ted	cs-en	gemma-4-31b	1000	439	778	16,384	493,222	0.1%	0.1%
translation-ted	cs-en	gemma-4-31b-nothink	1000	20	90	16,384	52,502	0.1%	0.0%
translation-ted	cs-en	gemma-4-e4b	1000	358	589	1,039	380,019	0.0%	0.0%
translation-ted	cs-en	gemma-4-e4b-nothink	1000	144	279	443	159,539	0.0%	0.0%
translation-ted	cs-en	qwen3.6-27b	1000	1,552	2,668	5,248	1,568,867	0.0%	0.0%
translation-ted	cs-en	qwen3.6-27b-nothink	1000	17	37	140	20,304	0.0%	0.0%
translation-ted	cs-en	qwen3.6-35b-a3b	1000	1,685	2,723	5,017	1,685,159	0.0%	0.0%
translation-ted	cs-en	qwen3.6-35b-a3b-nothink	1000	17	37	138	20,352	0.0%	0.0%
translation-ted	en-cs	deepseek-v4-flash	1000	214	272	442	215,492	0.0%	0.2%
translation-ted	en-cs	deepseek-v4-flash-nothink	1000	26	61	212	32,170	0.0%	0.0%
translation-ted	en-cs	gemma-4-31b	1000	616	1,033	16,384	774,447	0.7%	0.7%
translation-ted	en-cs	gemma-4-31b-nothink	1000	27	79	16,384	58,021	0.1%	0.0%
translation-ted	en-cs	gemma-4-e4b	1000	504	721	16,384	526,633	0.1%	0.0%
translation-ted	en-cs	gemma-4-e4b-nothink	1000	238	364	16,384	267,690	0.1%	0.0%
translation-ted	en-cs	qwen3.6-27b	1000	2,438	4,084	16,384	2,613,971	0.6%	0.6%
translation-ted	en-cs	qwen3.6-27b-nothink	1000	24	56	201	30,019	0.0%	0.0%
translation-ted	en-cs	qwen3.6-35b-a3b	1000	2,426	4,028	16,384	2,561,908	0.5%	0.5%
translation-ted	en-cs	qwen3.6-35b-a3b-nothink	1000	25	57	206	30,098	0.0%	0.0%
translation-wmt24	en-cs	deepseek-v4-flash	997	255	396	2,138	275,506	0.0%	0.0%
translation-wmt24	en-cs	deepseek-v4-flash-nothink	997	46	160	366	68,336	0.0%	0.0%
translation-wmt24	en-cs	gemma-4-31b	997	914	1,649	16,384	1,093,008	0.9%	0.9%
translation-wmt24	en-cs	gemma-4-31b-nothink	997	71	211	16,384	125,623	0.2%	0.0%
translation-wmt24	en-cs	gemma-4-e4b	997	441	710	1,056	464,798	0.0%	0.0%
translation-wmt24	en-cs	gemma-4-e4b-nothink	997	256	424	16,384	296,726	0.2%	0.0%
translation-wmt24	en-cs	qwen3.6-27b	997	3,082	5,469	16,384	3,311,069	1.3%	1.2%
translation-wmt24	en-cs	qwen3.6-27b-nothink	997	43	152	344	65,134	0.0%	0.0%
translation-wmt24	en-cs	qwen3.6-35b-a3b	997	3,151	5,417	16,384	3,387,615	1.8%	1.8%
translation-wmt24	en-cs	qwen3.6-35b-a3b-nothink	997	44	153	346	65,298	0.0%	0.0%

IFEval

MGSM

GPQA

Nexus (tool use)

The answer is scored by executing the predicted call against the reference arguments, so a formatting difference alone is not a miss.

HumanEval+

✓ means the program passed both the base and the extended test suite, which is what HumanEval+ counts.

LiveCodeBench

The extracted program is shown, not the whole response. Both Qwen models truncate over a quarter of this task mid-CoT (README §3.1), which shows up here as an empty program.

Translation

chrF beside each hypothesis is a per-segment score computed by the viewer; the strip above is the corpus spBLEU/chrF from the run.