krino

krino

Typed decision models that output calibrated probabilities instead of free text. Open replication of TypeSafe's Jev.

View BenchmarksGitHub
10.4–18.6 pp
Judgement over continuation
15 of 16 checkpoints; 10 win by more than 10 pp
Sources ↓
61.9–64.9%
Post-trained 4B judgement
base 53.7%; Qwen/Qwen3-4B, Qwen/Qwen3-Reranker-4B, microsoft/Phi-4-mini-instruct
Sources ↓
9.3 pp / 5–6 pp
Instrument resolution
JevBench 231 / 812-row dev set at 80% power
Sources ↓
57
Instrumented rows
each names its readout and source file
Sources ↓

What we have established

  • On JevBench public, the per-option judgement readout beats option-text continuation on 15 of 16 checkpoints; on the 10 checkpoints where it wins by more than 10 pp the gap is 10.4–18.6 pp, largest on the original tier. openbmb/MiniCPM3-4B (instruct) is the exception (-12.1 pp).
  • Post-training that moves the objective toward the readout helps: a 4B base scores 53.7%, while the three-checkpoint 4B judgement band (Qwen/Qwen3-4B, Qwen/Qwen3-Reranker-4B, microsoft/Phi-4-mini-instruct) scores 61.9–64.9%.
  • Reading one code token per option at a single answer position (panel 2) beats per-option judgement on 8 of 8 checkpoints where both exist, by 5.2–10.4 pp, and it costs one forward per item instead of one per option. The gain is largest on choice (8.6 pp across the set); noul and score move too, by 6.2 and 6.2 pp.
  • JevBench 231 cannot resolve differences below about 9.3 pp, which is why its top four checkpoints (Qwen/Qwen3-32B, Qwen/Qwen3-Reranker-4B, microsoft/Phi-4-mini-instruct, Qwen/Qwen3-Reranker-8B) are mutually indistinguishable within 5.2 pp. The 812-row development set resolves 5–6 pp and separates Qwen/Qwen3-Reranker-4B vs Qwen/Qwen3-Reranker-8B (6.03 pp, p = 0.0005) where JevBench has them 1 item apart.

In-domain single-task result

Banking77, trained heads: 95.2% (Ettin-150m r128). Jev's reference is 75.0%. This is an in-domain single-task result; Jev's number is zero-shot across 77 classes, so the comparison is not like-for-like.

Sources for the figures above

Every figure on this page is computed from one of these committed files.

  • data/benchmarks/dev/dev_800.json
  • model/evaluation/paired_power.py
  • model/experiments/jev_api.json
  • model/experiments/logit_prov/dev_judgement_Qwen_Qwen3-Reranker-0.6B.json
  • model/experiments/logit_prov/dev_judgement_Qwen_Qwen3-Reranker-4B.json
  • model/experiments/logit_prov/dev_judgement_Qwen_Qwen3-Reranker-8B.json
  • model/experiments/logit_prov/dev_judgement_zeroentropy_zerank-2-reranker.json
  • model/experiments/logit_prov/dev_logit_Qwen_Qwen3-Reranker-0.6B.json
  • model/experiments/logit_prov/dev_logit_Qwen_Qwen3-Reranker-4B.json
  • model/experiments/logit_prov/dev_logit_Qwen_Qwen3-Reranker-8B.json
  • model/experiments/logit_prov/dev_logit_zeroentropy_zerank-2-reranker.json
  • model/experiments/logit_prov/jevbench_judgement_Qwen_Qwen2.5-1.5B.json
  • model/experiments/logit_prov/jevbench_judgement_Qwen_Qwen2.5-3B.json
  • model/experiments/logit_prov/jevbench_judgement_Qwen_Qwen3-4B-Base.json
  • model/experiments/logit_prov/jevbench_judgement_Qwen_Qwen3-4B-Instruct-2507.json
  • model/experiments/logit_prov/jevbench_judgement_Qwen_Qwen3-4B-Thinking-2507.json
  • model/experiments/logit_prov/jevbench_judgement_Qwen_Qwen3-4B.json
  • model/experiments/logit_prov/jevbench_judgement_Qwen_Qwen3-Reranker-0.6B.json
  • model/experiments/logit_prov/jevbench_judgement_Qwen_Qwen3-Reranker-4B.json
  • model/experiments/logit_prov/jevbench_judgement_Qwen_Qwen3-Reranker-8B.json
  • model/experiments/logit_prov/jevbench_judgement_microsoft_Phi-4-mini-instruct.json
  • model/experiments/logit_prov/jevbench_judgement_mistralai_Ministral-3-3B-Base-2512.json
  • model/experiments/logit_prov/jevbench_judgement_openbmb_MiniCPM3-4B.json
  • model/experiments/logit_prov/jevbench_judgement_openbmb_MiniCPM5-2B-Base.json
  • model/experiments/logit_prov/jevbench_judgement_zeroentropy_zerank-2-reranker.json
  • model/experiments/logit_prov/jevbench_logit_Qwen_Qwen2.5-1.5B.json
  • model/experiments/logit_prov/jevbench_logit_Qwen_Qwen2.5-3B.json
  • model/experiments/logit_prov/jevbench_logit_Qwen_Qwen3-4B-Base.json
  • model/experiments/logit_prov/jevbench_logit_Qwen_Qwen3-4B-Instruct-2507.json
  • model/experiments/logit_prov/jevbench_logit_Qwen_Qwen3-4B-Thinking-2507.json
  • model/experiments/logit_prov/jevbench_logit_Qwen_Qwen3-4B.json
  • model/experiments/logit_prov/jevbench_logit_Qwen_Qwen3-Reranker-0.6B.json
  • model/experiments/logit_prov/jevbench_logit_Qwen_Qwen3-Reranker-4B.json
  • model/experiments/logit_prov/jevbench_logit_Qwen_Qwen3-Reranker-8B.json
  • model/experiments/logit_prov/jevbench_logit_microsoft_Phi-4-mini-instruct.json
  • model/experiments/logit_prov/jevbench_logit_zeroentropy_zerank-2-reranker.json
  • model/experiments/resolution_inputs.json
  • model/experiments/train_ettin150m_r128.json
  • results/jevbench/continuation_Qwen_Qwen3-14B.json
  • results/jevbench/continuation_Qwen_Qwen3-32B.json
  • results/jevbench/judgement_Qwen_Qwen3-14B.json
  • results/jevbench/judgement_Qwen_Qwen3-32B.json
  • results/jevbench/logit_mistralai_Ministral-3-3B-Base-2512.json
  • results/jevbench/logit_openbmb_MiniCPM3-4B.json
  • results/jevbench/logit_openbmb_MiniCPM5-2B-Base.json
  • results/jevbench/panel2_Qwen_Qwen3-14B.json
  • results/jevbench/panel2_Qwen_Qwen3-32B.json
  • results/jevbench/panel2_Qwen_Qwen3-4B-Base.json
  • results/jevbench/panel2_Qwen_Qwen3-4B-Instruct-2507.json
  • results/jevbench/panel2_Qwen_Qwen3-4B.json
  • results/jevbench/panel2_Qwen_Qwen3-Reranker-0.6B.json
  • results/jevbench/panel2_Qwen_Qwen3-Reranker-4B.json
  • results/jevbench/panel2_Qwen_Qwen3-Reranker-8B.json
  • web/public/results/eval_results.json

View the benchmarks tables →