Benchmark · CzTauBench

airline

pass^2 · 5 of ours, 4 published

EnglishCzech (L2 interaction)EN→CS gappublished tau2-bench (English)

deepseek-v4-flash-fp4-think-on
trajectories ›

0.827

0.760Δ -0.067

qwen3.6-35b-a3b-think-on
trajectories ›

0.813

0.687Δ -0.127

gemma-4-31b-it-think-on
trajectories ›

0.767

0.767Δ +0.000

qwen3.6-27b-think-on
trajectories ›

0.727

0.687Δ -0.040

gemma-4-e4b-it-think-on
trajectories ›

0.540

0.487Δ -0.053

GPT-5.2
OpenAI · reasoning high

0.783

Claude Opus 4.5
Anthropic · reasoning high

0.777

Qwen3.5-397B-A17B
Alibaba Cloud · reasoning enabled

0.757

Gemini 3 Pro
Google · reasoning high

0.747

retail

pass^2 · 5 of ours, 4 published

EnglishCzech (L2 interaction)EN→CS gappublished tau2-bench (English)

deepseek-v4-flash-fp4-think-on
trajectories ›

0.845

0.798Δ -0.047

qwen3.6-35b-a3b-think-on
trajectories ›

0.798

0.749Δ -0.050

qwen3.6-27b-think-on
trajectories ›

0.778

0.696Δ -0.082

gemma-4-31b-it-think-on
trajectories ›

0.766

0.769Δ +0.003

gemma-4-e4b-it-think-on
trajectories ›

0.579

0.418Δ -0.161

Qwen3.5-397B-A17B
Alibaba Cloud · reasoning enabled

0.744

GPT-5.2
OpenAI · reasoning high

0.696

Claude Opus 4.5
Anthropic · reasoning high

0.674

Gemini 3 Pro
Google · reasoning high

0.635

On the published bars. Taken from taubench.com (fetched 2026-08-06), English only — there is no Czech reference to compare against. They are indicative, not like-for-like: those runs drive the user simulator with gpt-5.2 while ours uses kimi-k3, and the simulator is a large part of what a τ² score measures. The site's headline core number is also not shown here — it averages airline, retail and telecom, and telecom (which we do not run) is the easiest of the three, so it sits well above the per-domain values.

run	cell	done	pass^1	pass^2	pass^3	ρ₃	avg reward	lang ok
deepseek-v4-flash-fp4-think-on	english_airline	150/150	0.880	0.827	0.800	0.909	0.880	1.000	compare EN/CS
deepseek-v4-flash-fp4-think-on	english_retail	342/342	0.895	0.845	0.816	0.912	0.895	1.000	compare EN/CS
deepseek-v4-flash-fp4-think-on	l2_interaction_airline_cs	150/150	0.807	0.760	0.760	0.942	0.807	1.000	compare EN/CS
deepseek-v4-flash-fp4-think-on	l2_interaction_retail_cs	342/342	0.865	0.798	0.746	0.861	0.865	1.000	compare EN/CS
gemma-4-31b-it-think-on	english_airline	150/150	0.847	0.767	0.720	0.850	0.847	0.999	compare EN/CS
gemma-4-31b-it-think-on	english_retail	342/342	0.822	0.766	0.737	0.897	0.822	1.000	compare EN/CS
gemma-4-31b-it-think-on	l2_interaction_airline_cs	150/150	0.800	0.767	0.740	0.925	0.800	0.952	compare EN/CS
gemma-4-31b-it-think-on	l2_interaction_retail_cs	342/342	0.833	0.769	0.737	0.884	0.833	0.979	compare EN/CS
gemma-4-e4b-it-think-on	english_airline	150/150	0.607	0.540	0.500	0.824	0.607	1.000	compare EN/CS
gemma-4-e4b-it-think-on	english_retail	342/342	0.711	0.579	0.491	0.691	0.711	1.000	compare EN/CS
gemma-4-e4b-it-think-on	l2_interaction_airline_cs	150/150	0.567	0.487	0.440	0.776	0.567	0.998	compare EN/CS
gemma-4-e4b-it-think-on	l2_interaction_retail_cs	342/342	0.553	0.418	0.360	0.651	0.553	0.999	compare EN/CS
qwen3.6-27b-think-on	english_airline	150/150	0.807	0.727	0.680	0.843	0.807	1.000	compare EN/CS
qwen3.6-27b-think-on	english_retail	342/342	0.842	0.778	0.746	0.885	0.842	1.000	compare EN/CS
qwen3.6-27b-think-on	l2_interaction_airline_cs	150/150	0.753	0.687	0.640	0.850	0.753	0.993	compare EN/CS
qwen3.6-27b-think-on	l2_interaction_retail_cs	342/342	0.810	0.696	0.605	0.747	0.810	1.000	compare EN/CS
qwen3.6-35b-a3b-think-on	english_airline	150/150	0.880	0.813	0.780	0.886	0.880	1.000	compare EN/CS
qwen3.6-35b-a3b-think-on	english_retail	342/342	0.851	0.798	0.754	0.887	0.851	1.000	compare EN/CS
qwen3.6-35b-a3b-think-on	l2_interaction_airline_cs	150/150	0.780	0.687	0.620	0.795	0.780	0.998	compare EN/CS
qwen3.6-35b-a3b-think-on	l2_interaction_retail_cs	342/342	0.845	0.749	0.675	0.799	0.845	0.997	compare EN/CS

pass^k: all k trials of a task succeed, averaged over tasks
ρ₃: pass^3 / pass^1 — how much of the single-shot score survives three attempts
lang ok: share of agent turns fastText detected as the target language — not whether the Czech is any good
dimmed, part: cell unfinished; pass^k covers only the tasks done so far and is not yet representative
infra: simulations that never ran — excluded throughout, and they cap k at the thinnest task

Leaderboard

airline

retail

all cells