Benchmark · CzTauBench

airline

pass^3 · 5 of ours, 4 published

EnglishCzech (L2 interaction)EN→CS gappublished tau2-bench (English)

deepseek-v4-flash-fp4-think-on
trajectories ›

0.800

0.760Δ -0.040

qwen3.6-35b-a3b-think-on
trajectories ›

0.780

0.620Δ -0.160

gemma-4-31b-it-think-on
trajectories ›

0.720

0.740Δ +0.020

qwen3.6-27b-think-on
trajectories ›

0.680

0.640Δ -0.040

gemma-4-e4b-it-think-on
trajectories ›

0.500

0.440Δ -0.060

GPT-5.2
OpenAI · reasoning high

0.750

Claude Opus 4.5
Anthropic · reasoning high

0.735

Qwen3.5-397B-A17B
Alibaba Cloud · reasoning enabled

0.715

Gemini 3 Pro
Google · reasoning high

0.700

retail

pass^3 · 5 of ours, 4 published

EnglishCzech (L2 interaction)EN→CS gappublished tau2-bench (English)

deepseek-v4-flash-fp4-think-on
trajectories ›

0.816

0.746Δ -0.070

qwen3.6-35b-a3b-think-on
trajectories ›

0.754

0.675Δ -0.079

qwen3.6-27b-think-on
trajectories ›

0.746

0.605Δ -0.140

gemma-4-31b-it-think-on
trajectories ›

0.737

0.737Δ +0.000

gemma-4-e4b-it-think-on
trajectories ›

0.491

0.360Δ -0.132

Qwen3.5-397B-A17B
Alibaba Cloud · reasoning enabled

0.664

GPT-5.2
OpenAI · reasoning high

0.599

Claude Opus 4.5
Anthropic · reasoning high

0.588

Gemini 3 Pro
Google · reasoning high

0.544

On the published bars. Taken from taubench.com (fetched 2026-08-06), English only — there is no Czech reference to compare against. They are indicative, not like-for-like: those runs drive the user simulator with gpt-5.2 while ours uses kimi-k3, and the simulator is a large part of what a τ² score measures. The site's headline core number is also not shown here — it averages airline, retail and telecom, and telecom (which we do not run) is the easiest of the three, so it sits well above the per-domain values.

run	cell	done	pass^1	pass^2	pass^3	ρ₃	avg reward	lang ok
deepseek-v4-flash-fp4-think-on	english_airline	150/150	0.880	0.827	0.800	0.909	0.880	1.000	compare EN/CS
deepseek-v4-flash-fp4-think-on	english_retail	342/342	0.895	0.845	0.816	0.912	0.895	1.000	compare EN/CS
deepseek-v4-flash-fp4-think-on	l2_interaction_airline_cs	150/150	0.807	0.760	0.760	0.942	0.807	1.000	compare EN/CS
deepseek-v4-flash-fp4-think-on	l2_interaction_retail_cs	342/342	0.865	0.798	0.746	0.861	0.865	1.000	compare EN/CS
gemma-4-31b-it-think-on	english_airline	150/150	0.847	0.767	0.720	0.850	0.847	0.999	compare EN/CS
gemma-4-31b-it-think-on	english_retail	342/342	0.822	0.766	0.737	0.897	0.822	1.000	compare EN/CS
gemma-4-31b-it-think-on	l2_interaction_airline_cs	150/150	0.800	0.767	0.740	0.925	0.800	0.952	compare EN/CS
gemma-4-31b-it-think-on	l2_interaction_retail_cs	342/342	0.833	0.769	0.737	0.884	0.833	0.979	compare EN/CS
gemma-4-e4b-it-think-on	english_airline	150/150	0.607	0.540	0.500	0.824	0.607	1.000	compare EN/CS
gemma-4-e4b-it-think-on	english_retail	342/342	0.711	0.579	0.491	0.691	0.711	1.000	compare EN/CS
gemma-4-e4b-it-think-on	l2_interaction_airline_cs	150/150	0.567	0.487	0.440	0.776	0.567	0.998	compare EN/CS
gemma-4-e4b-it-think-on	l2_interaction_retail_cs	342/342	0.553	0.418	0.360	0.651	0.553	0.999	compare EN/CS
qwen3.6-27b-think-on	english_airline	150/150	0.807	0.727	0.680	0.843	0.807	1.000	compare EN/CS
qwen3.6-27b-think-on	english_retail	342/342	0.842	0.778	0.746	0.885	0.842	1.000	compare EN/CS
qwen3.6-27b-think-on	l2_interaction_airline_cs	150/150	0.753	0.687	0.640	0.850	0.753	0.993	compare EN/CS
qwen3.6-27b-think-on	l2_interaction_retail_cs	342/342	0.810	0.696	0.605	0.747	0.810	1.000	compare EN/CS
qwen3.6-35b-a3b-think-on	english_airline	150/150	0.880	0.813	0.780	0.886	0.880	1.000	compare EN/CS
qwen3.6-35b-a3b-think-on	english_retail	342/342	0.851	0.798	0.754	0.887	0.851	1.000	compare EN/CS
qwen3.6-35b-a3b-think-on	l2_interaction_airline_cs	150/150	0.780	0.687	0.620	0.795	0.780	0.998	compare EN/CS
qwen3.6-35b-a3b-think-on	l2_interaction_retail_cs	342/342	0.845	0.749	0.675	0.799	0.845	0.997	compare EN/CS

pass^k: all k trials of a task succeed, averaged over tasks
ρ₃: pass^3 / pass^1 — how much of the single-shot score survives three attempts
lang ok: share of agent turns fastText detected as the target language — not whether the Czech is any good
dimmed, part: cell unfinished; pass^k covers only the tasks done so far and is not yet representative
infra: simulations that never ran — excluded throughout, and they cap k at the thinnest task

Leaderboard

airline

retail

all cells