JevBench Results
JevBench (231 curated items across 3 difficulty tiers), sorted by overall accuracy. Every row names the readout that produced it: trained heads, option-text continuation, per-option judgement and native reranker scores are separate instruments and are not directly comparable. H2O-Lightning-4B is a third-party trained model scored through its own serving stack; its native decision rows are not a zero-shot readout cell. Hover a row for the recorded readout and model id. Trained-head rows also carry the project generation: v1 heads predate the v4 line, so a v1 head on a Qwen3.5-4B backbone is not a v4 result.
| Model | Readout | Params | Overall ▼ | Choice | Noul | Score | Easy | Hard | Original | Latency |
|---|---|---|---|---|---|---|---|---|---|---|
| causalH2O-Lightning-4B (third-party) | native decision | 4B | 89.2% | 90.6% | 86.5% | 88.9% | 100.0% | 77.5% | 100.0% | 39ms |
| causalH2O-Lightning-4B (third-party) | native decision, one-way | 4B | 87.4% | 89.2% | 83.8% | 88.9% | 100.0% | 75.7% | 97.2% | 39ms |
| unknownQwen/Qwen3-32B | vocab code | — | 77.1% | 75.5% | 81.1% | 72.2% | 100.0% | 58.6% | 90.3% | — |
| unknownQwen/Qwen3-Reranker-8B | vocab code | — | 71.0% | 72.7% | 66.2% | 77.8% | 100.0% | 50.4% | 83.3% | — |
| unknownQwen/Qwen3-14B | vocab code | — | 70.1% | 69.1% | 71.6% | 72.2% | 100.0% | 49.5% | 81.9% | — |
| unknownQwen/Qwen3-Reranker-4B | vocab code | — | 70.1% | 73.4% | 64.9% | 66.7% | 100.0% | 46.9% | 86.1% | — |
| unknownQwen/Qwen3-32B | judgement | — | 69.7% | 66.9% | 75.7% | 66.7% | 100.0% | 51.3% | 77.8% | 1745ms |
| unknownQwen/Qwen3-4B | vocab code | — | 69.7% | 73.4% | 64.9% | 61.1% | 100.0% | 47.8% | 83.3% | — |
| unknownQwen/Qwen3-4B-Instruct-2507 | vocab code | — | 68.4% | 68.3% | 67.6% | 72.2% | 100.0% | 46.0% | 81.9% | — |
| unknownQwen/Qwen3-Reranker-4B | judgement | — | 64.9% | 64.8% | 63.5% | 72.2% | 100.0% | 45.1% | 72.2% | — |
| unknownmicrosoft/Phi-4-mini-instruct (instruct) | judgement | — | 64.9% | 67.6% | 60.8% | 61.1% | 97.9% | 44.1% | 75.0% | — |
| unknownQwen/Qwen3-Reranker-8B | judgement | — | 64.5% | 66.9% | 59.5% | 66.7% | 97.9% | 45.1% | 72.2% | 1345ms |
| unknownQwen/Qwen3-Reranker-8B | judgement | — | 64.5% | 66.9% | 59.5% | 66.7% | 97.9% | 45.1% | 72.2% | — |
| rerankerQwen/Qwen3-Reranker-4B (zero-shot) | native reranker | 4B | 62.8% | 60.4% | 62.2% | 83.3% | 100.0% | 44.1% | 66.7% | — |
| unknownQwen/Qwen3-4B-Base | vocab code | — | 62.3% | 64.8% | 62.2% | 44.4% | 100.0% | 43.2% | 66.7% | — |
| unknownzeroentropy/zerank-2-reranker | judgement | — | 62.3% | 67.6% | 52.7% | 61.1% | 97.9% | 46.0% | 63.9% | — |
| unknownQwen/Qwen3-4B | judgement | — | 61.9% | 64.0% | 59.5% | 55.6% | 100.0% | 44.1% | 63.9% | 1266ms |
| unknownQwen/Qwen3-4B (instruct) | judgement | — | 61.9% | 64.0% | 59.5% | 55.6% | 100.0% | 44.1% | 63.9% | — |
| unknownQwen/Qwen3-14B | judgement | — | 59.7% | 56.8% | 64.9% | 61.1% | 97.9% | 44.1% | 58.3% | 1375ms |
| unknownQwen/Qwen3-4B-Instruct-2507 (instruct) | judgement | — | 58.9% | 57.6% | 59.5% | 66.7% | 100.0% | 42.3% | 56.9% | — |
| unknownQwen/Qwen2.5-3B (base) | judgement | — | 57.1% | 57.6% | 55.4% | 61.1% | 97.9% | 39.6% | 56.9% | — |
| unknownmistralai/Ministral-3-3B-Base-2512 (base) | judgement | — | 55.8% | 55.4% | 58.1% | 50.0% | 93.8% | 38.7% | 56.9% | — |
| unknownQwen/Qwen3-4B-Thinking-2507 (instruct) | judgement | — | 55.4% | 57.6% | 54.0% | 44.4% | 97.9% | 39.6% | 51.4% | — |
| unknownQwen/Qwen3-32B | continuation | — | 54.1% | 52.5% | 52.7% | 72.2% | 87.5% | 43.2% | 48.6% | 960ms |
| unknownQwen/Qwen3-4B-Base (base) | judgement | — | 53.7% | 56.8% | 50.0% | 44.4% | 89.6% | 36.9% | 55.6% | — |
| unknownQwen/Qwen3-14B | continuation | — | 50.2% | 47.5% | 50.0% | 72.2% | 83.3% | 36.0% | 50.0% | 879ms |
| unknownmicrosoft/Phi-4-mini-instruct (instruct) | continuation | — | 50.2% | 47.5% | 51.3% | 66.7% | 79.2% | 36.9% | 51.4% | — |
| unknownQwen/Qwen3-4B-Base (base) | continuation | — | 48.5% | 47.5% | 51.3% | 44.4% | 83.3% | 39.6% | 38.9% | — |
| unknownQwen/Qwen3-4B | continuation | — | 48.1% | 43.2% | 58.1% | 44.4% | 87.5% | 34.2% | 43.1% | 849ms |
| unknownQwen/Qwen3-Reranker-4B | continuation | — | 48.1% | 43.2% | 54.0% | 61.1% | 87.5% | 32.4% | 45.8% | 856ms |
| unknownQwen/Qwen3-Reranker-8B | continuation | — | 48.1% | 39.6% | 56.8% | 77.8% | 79.2% | 34.2% | 48.6% | 869ms |
| unknownQwen/Qwen3-4B (instruct) | continuation | — | 48.1% | 43.2% | 58.1% | 44.4% | 87.5% | 34.2% | 43.1% | — |
| unknownQwen/Qwen3-Reranker-4B | continuation | — | 48.1% | 43.2% | 54.0% | 61.1% | 87.5% | 32.4% | 45.8% | — |
| unknownQwen/Qwen3-Reranker-8B | continuation | — | 48.1% | 39.6% | 56.8% | 77.8% | 79.2% | 34.2% | 48.6% | — |
| unknownQwen/Qwen3-Reranker-0.6B | vocab code | — | 47.6% | 44.6% | 56.8% | 33.3% | 70.8% | 36.9% | 48.6% | — |
| unknownQwen/Qwen2.5-1.5B (base) | judgement | — | 47.6% | 45.3% | 54.0% | 38.9% | 91.7% | 34.2% | 38.9% | — |
| unknownQwen/Qwen3-4B-Instruct-2507 (instruct) | continuation | — | 47.6% | 46.0% | 52.7% | 38.9% | 83.3% | 36.0% | 41.7% | — |
| unknownopenbmb/MiniCPM5-2B-Base (base) | judgement | — | 47.2% | 45.3% | 50.0% | 50.0% | 70.8% | 37.8% | 45.8% | — |
| unknownQwen/Qwen2.5-1.5B (base) | continuation | — | 47.2% | 42.4% | 56.8% | 44.4% | 77.1% | 36.9% | 43.1% | — |
| unknownQwen/Qwen3-1.7B-Base (base) | judgement | — | 46.8% | 48.9% | 43.2% | 44.4% | 79.2% | 35.1% | 43.1% | — |
| unknownopenbmb/MiniCPM3-4B (instruct) | continuation | — | 45.9% | 45.3% | 50.0% | 33.3% | 72.9% | 38.7% | 38.9% | — |
| unknownmistralai/Ministral-3-3B-Base-2512 (base) | continuation | — | 45.5% | 43.9% | 52.7% | 27.8% | 72.9% | 36.0% | 41.7% | — |
| unknownQwen/Qwen2.5-3B (base) | continuation | — | 44.6% | 40.3% | 50.0% | 55.6% | 72.9% | 33.3% | 43.1% | — |
| unknownQwen/Qwen3-4B-Thinking-2507 (instruct) | continuation | — | 44.6% | 42.4% | 50.0% | 38.9% | 77.1% | 34.2% | 38.9% | — |
| unknownzeroentropy/zerank-2-reranker | continuation | — | 43.7% | 36.7% | 52.7% | 61.1% | 77.1% | 31.5% | 40.3% | — |
| rerankerQwen3-reranker-4B (trained, v1) | trained heads | 4B | 42.9% | 38.1% | 52.7% | 38.9% | 66.7% | 36.0% | 37.5% | 8594ms |
| unknownopenbmb/MiniCPM5-2B-Base (base) | continuation | — | 41.6% | 36.7% | 50.0% | 44.4% | 72.9% | 30.6% | 37.5% | — |
| unknownQwen/Qwen3-Reranker-0.6B | judgement | — | 41.6% | 38.9% | 52.7% | 16.7% | 66.7% | 32.4% | 38.9% | — |
| unknownQwen/Qwen3-1.7B (instruct) | judgement | — | 41.1% | 37.4% | 51.3% | 27.8% | 66.7% | 32.4% | 37.5% | — |
| unknownQwen/Qwen3-Reranker-0.6B | continuation | — | 39.4% | 30.2% | 58.1% | 33.3% | 56.3% | 32.4% | 38.9% | 848ms |
| unknownQwen/Qwen3-Reranker-0.6B | continuation | — | 39.4% | 30.2% | 58.1% | 33.3% | 56.3% | 32.4% | 38.9% | — |
| rerankerEttin-150m (trained, v1) | trained heads | 150M | 36.8% | 32.4% | 44.6% | 38.9% | 56.3% | 31.5% | 31.9% | 278ms |
| rerankerQwen3-reranker-0.6B (trained, v1) | trained heads | 0.6B | 36.4% | 31.6% | 50.0% | 16.7% | 47.9% | 32.4% | 34.7% | 1327ms |
| causalQwen3.5-4B (trained, v1) | trained heads | 4B | 34.6% | 21.6% | 59.5% | 33.3% | 31.3% | 34.2% | 37.5% | 8408ms |
| causalQwen3-0.6B (trained, v1) | trained heads | 0.6B | 34.2% | 27.3% | 47.3% | 33.3% | 50.0% | 29.7% | 30.6% | 1315ms |
| unknownopenbmb/MiniCPM3-4B (instruct) | judgement | — | 33.8% | 28.1% | 47.3% | 22.2% | 35.4% | 36.0% | 29.2% | — |
| encoderModernBERT-base (trained, v1) | trained heads | 149M | 32.5% | 20.1% | 52.7% | 44.4% | 33.3% | 31.5% | 33.3% | 283ms |
19-Benchmark Sweep
Zero-shot and trained evaluation results across 19 benchmarks. Click a row to expand per-benchmark details.
Columns:
| Model | Params | Method | BMs | Avg Acc ▼ | agnews | arc | banking77 | race | sst2 |
|---|---|---|---|---|---|---|---|---|---|
| apiJev (TypeSafe API) | unknown | jev_api | 18 | 87.0% | 79.0% | 99.0% | 75.0% | 95.5% | 90.5% |
| causalOpenJev Qwen3.5-4B | 4B | openjev | 1 | 79.2% | — | — | — | — | — |
| causalQwen3.5-9B | 9B | logit | 19 | 67.3% | 69.0% | 71.5% | 0.0% | 68.0% | 91.0% |
| causalQwen2.5-7B-Instruct(Phase A) | 7B | logit | 19 | 67.1% | 70.5% | 65.5% | 2.0% | 64.0% | 87.0% |
| causalQwen3-4B | 4B | logit | 19 | 66.3% | 58.5% | 68.0% | 14.0% | 62.0% | 85.5% |
| causalQwen3-8B | 8B | logit | 19 | 65.3% | 73.0% | 72.5% | 5.0% | 64.0% | 83.5% |
| causalPhi-4-mini-instruct(Phase A) | 3.8B | logit | 19 | 64.4% | 64.5% | 73.5% | 2.5% | 65.5% | 83.5% |
| moeQwen3-30B-A3B | 30B (3B active) | logit | 19 | 60.5% | 75.5% | 58.5% | 0.0% | 66.5% | 90.0% |
| rerankerEttin-400m(Phase A) | 400M | reranker | 19 | 59.0% | 73.5% | 73.5% | 49.0% | 62.0% | 57.0% |
| causalQwen2.5-1.5B(Phase A) | 1.5B | logit | 19 | 58.4% | 74.5% | 57.5% | 0.0% | 60.0% | 87.5% |
| rerankerEttin-400m | 400M | reranker | 5 | 55.6% | 73.5% | — | 49.0% | — | 57.0% |
| causalMiniCPM5-2B-Base | 2B | logit | 19 | 55.3% | 73.5% | 65.0% | 0.5% | 60.0% | 74.0% |
| causalQwen3-1.7B(Phase A) | 1.7B | logit | 19 | 53.6% | 65.5% | 54.5% | 0.0% | 52.5% | 73.0% |
| causalMinistral-3-3B-Base-2512 | 3B | logit | 19 | 53.5% | 79.5% | 74.5% | 0.0% | 62.0% | 67.5% |
| rerankerEttin-150m(Phase A) | 150M | reranker | 19 | 53.2% | 73.5% | 68.5% | 21.5% | 59.5% | 52.0% |
| causalMiniCPM3-4B (instruct) | 4B | logit | 19 | 53.1% | 80.5% | 51.0% | 0.0% | 64.0% | 63.0% |
| causalQwen2.5-0.5B(Phase A) | 0.5B | logit | 19 | 52.4% | 77.5% | 50.0% | 0.0% | 52.5% | 64.5% |
| causalQwen2.5-7B-Instruct | 7B | logit | 5 | 52.1% | 70.5% | — | 1.5% | — | 87.0% |
| causalPhi-4-mini-instruct | 3.8B | logit | 5 | 51.6% | 64.5% | — | 3.0% | — | 83.5% |
| causalSmolLM2-1.7B(Phase A) | 1.7B | logit | 19 | 50.7% | 80.0% | 60.0% | 0.0% | 57.0% | 60.5% |
| rerankerEttin-68m(Phase A) | 68M | reranker | 19 | 50.1% | 69.5% | 54.5% | 27.0% | 55.0% | 54.0% |
| causalQwen2.5-1.5B | 1.5B | logit | 5 | 49.2% | 74.5% | — | 0.0% | — | 87.5% |
| causalQwen3.5-4B | 4B | logit | 5 | 48.6% | 53.0% | — | 2.5% | — | 84.5% |
| rerankerEttin-150m | 150M | reranker | 5 | 48.0% | 73.5% | — | 21.5% | — | 52.0% |
| causalQwen3-0.6B(Phase A) | 0.6B | logit | 19 | 47.7% | 67.0% | 43.5% | 0.0% | 49.0% | 52.5% |
| rerankerEttin-68m | 68M | reranker | 5 | 47.4% | 69.5% | — | 27.0% | — | 54.0% |
| causalOpenJev Qwen3-0.6B | 0.6B | openjev | 1 | 47.2% | — | — | — | — | — |
| causalQwen3-1.7B | 1.7B | logit | 5 | 46.0% | 65.5% | — | 0.0% | — | 73.0% |
| causalSmolLM2-1.7B | 1.7B | logit | 5 | 45.7% | 80.0% | — | 0.0% | — | 60.5% |
| causalQwen2.5-0.5B | 0.5B | logit | 5 | 42.5% | 79.0% | — | 0.0% | — | 64.5% |
| causalQwen cross-encoder | 0.6B | cross-encoder | 3 | 40.5% | — | — | — | — | 48.2% |
| encoderBERT cross-encoder | 110M | cross-encoder | 3 | 40.0% | — | — | — | — | 51.8% |
| causalQwen3-0.6B | 0.6B | logit | 5 | 39.8% | 66.0% | — | 0.0% | — | 52.5% |
| causalQwen2.5-0.5B (baseline) | 0.5B | logit | 3 | 24.8% | — | — | 9.5% | — | 52.5% |
| encoderModernBERT-large | 395M | encoder | 4 | 23.9% | — | — | 6.0% | — | 46.0% |
| encoderModernBERT-base | 149M | encoder | 4 | 22.4% | — | — | 0.0% | — | 49.0% |
Showing 36 of 36 runs