JevBench Results
Krino trained models evaluated on JevBench (231 curated items across 3 difficulty tiers). Sorted by overall accuracy.
| Model | Params | Overall ▼ | Choice | Noul | Score | Easy | Hard | Original | Latency |
|---|---|---|---|---|---|---|---|---|---|
| rerankerQwen3-reranker-4B (zero-shot) | 4B | 62.8% | 60.4% | 62.2% | 83.3% | 100.0% | 44.1% | 66.7% | — |
| rerankerQwen3-reranker-4B (trained) | 4B | 42.9% | 38.1% | 52.7% | 38.9% | 66.7% | 36.0% | 37.5% | 8594ms |
| rerankerEttin-150m (trained) | 150M | 36.8% | 32.4% | 44.6% | 38.9% | 56.3% | 31.5% | 31.9% | 278ms |
| rerankerQwen3-reranker-0.6B (trained) | 0.6B | 36.4% | 31.6% | 50.0% | 16.7% | 47.9% | 32.4% | 34.7% | 1327ms |
| causalQwen3.5-4B (trained) | 4B | 34.6% | 21.6% | 59.5% | 33.3% | 31.3% | 34.2% | 37.5% | 8408ms |
| causalQwen3-0.6B (trained) | 0.6B | 34.2% | 27.3% | 47.3% | 33.3% | 50.0% | 29.7% | 30.6% | 1315ms |
| encoderModernBERT-base (trained) | 149M | 32.5% | 20.1% | 52.7% | 44.4% | 33.3% | 31.5% | 33.3% | 283ms |
19-Benchmark Sweep
Zero-shot and trained evaluation results across 19 benchmarks. Click a row to expand per-benchmark details.
Columns:
| Model | Params | Method | BMs | Avg Acc ▼ | agnews | arc | banking77 | race | sst2 |
|---|---|---|---|---|---|---|---|---|---|
| apiJev (TypeSafe API) | unknown | jev_api | 18 | 87.0% | 79.0% | 99.0% | 75.0% | 95.5% | 90.5% |
| causalOpenJev Qwen3.5-4B | 4B | openjev | 1 | 79.2% | — | — | — | — | — |
| causalQwen3.5-9B | 9B | logit | 19 | 67.3% | 69.0% | 71.5% | 0.0% | 68.0% | 91.0% |
| causalQwen2.5-7B-Instruct(Phase A) | 7B | logit | 19 | 67.1% | 70.5% | 65.5% | 2.0% | 64.0% | 87.0% |
| causalQwen3-4B | 4B | logit | 19 | 66.3% | 58.5% | 68.0% | 14.0% | 62.0% | 85.5% |
| causalQwen3-8B | 8B | logit | 19 | 65.3% | 73.0% | 72.5% | 5.0% | 64.0% | 83.5% |
| causalPhi-4-mini-instruct(Phase A) | 3.8B | logit | 19 | 64.4% | 64.5% | 73.5% | 2.5% | 65.5% | 83.5% |
| moeQwen3-30B-A3B | 30B (3B active) | logit | 19 | 60.5% | 75.5% | 58.5% | 0.0% | 66.5% | 90.0% |
| rerankerEttin-400m(Phase A) | 400M | reranker | 19 | 59.0% | 73.5% | 73.5% | 49.0% | 62.0% | 57.0% |
| causalQwen2.5-1.5B(Phase A) | 1.5B | logit | 19 | 58.4% | 74.5% | 57.5% | 0.0% | 60.0% | 87.5% |
| rerankerEttin-400m | 400M | reranker | 5 | 55.6% | 73.5% | — | 49.0% | — | 57.0% |
| causalQwen3-1.7B(Phase A) | 1.7B | logit | 19 | 53.6% | 65.5% | 54.5% | 0.0% | 52.5% | 73.0% |
| rerankerEttin-150m(Phase A) | 150M | reranker | 19 | 53.2% | 73.5% | 68.5% | 21.5% | 59.5% | 52.0% |
| causalQwen2.5-0.5B(Phase A) | 0.5B | logit | 19 | 52.4% | 77.5% | 50.0% | 0.0% | 52.5% | 64.5% |
| causalQwen2.5-7B-Instruct | 7B | logit | 5 | 52.1% | 70.5% | — | 1.5% | — | 87.0% |
| causalPhi-4-mini-instruct | 3.8B | logit | 5 | 51.6% | 64.5% | — | 3.0% | — | 83.5% |
| causalSmolLM2-1.7B(Phase A) | 1.7B | logit | 19 | 50.7% | 80.0% | 60.0% | 0.0% | 57.0% | 60.5% |
| rerankerEttin-68m(Phase A) | 68M | reranker | 19 | 50.1% | 69.5% | 54.5% | 27.0% | 55.0% | 54.0% |
| causalQwen2.5-1.5B | 1.5B | logit | 5 | 49.2% | 74.5% | — | 0.0% | — | 87.5% |
| causalQwen3.5-4B | 4B | logit | 5 | 48.6% | 53.0% | — | 2.5% | — | 84.5% |
| rerankerEttin-150m | 150M | reranker | 5 | 48.0% | 73.5% | — | 21.5% | — | 52.0% |
| causalQwen3-0.6B(Phase A) | 0.6B | logit | 19 | 47.7% | 67.0% | 43.5% | 0.0% | 49.0% | 52.5% |
| rerankerEttin-68m | 68M | reranker | 5 | 47.4% | 69.5% | — | 27.0% | — | 54.0% |
| causalOpenJev Qwen3-0.6B | 0.6B | openjev | 1 | 47.2% | — | — | — | — | — |
| causalQwen3-1.7B | 1.7B | logit | 5 | 46.0% | 65.5% | — | 0.0% | — | 73.0% |
| causalSmolLM2-1.7B | 1.7B | logit | 5 | 45.7% | 80.0% | — | 0.0% | — | 60.5% |
| causalQwen2.5-0.5B | 0.5B | logit | 5 | 42.5% | 79.0% | — | 0.0% | — | 64.5% |
| causalQwen cross-encoder | 0.6B | cross-encoder | 3 | 40.5% | — | — | — | — | 48.2% |
| encoderBERT cross-encoder | 110M | cross-encoder | 3 | 40.0% | — | — | — | — | 51.8% |
| causalQwen3-0.6B | 0.6B | logit | 5 | 39.8% | 66.0% | — | 0.0% | — | 52.5% |
| causalQwen2.5-0.5B (baseline) | 0.5B | logit | 3 | 24.8% | — | — | 9.5% | — | 52.5% |
| encoderModernBERT-large | 395M | encoder | 4 | 23.9% | — | — | 6.0% | — | 46.0% |
| encoderModernBERT-base | 149M | encoder | 4 | 22.4% | — | — | 0.0% | — | 49.0% |
Showing 33 of 33 runs