VNTL Leaderboard
Leaderboard for Japanese visual-novel translation into English, ranking LLMs and translation systems by semantic similarity accuracy over 256 translation samples, with chrF reported as an auxiliary metric.
87rows
accuracyprimary metric
2026-05-06sampled
Metadata
Metrics
Accuracy, Accuracy 95% CI (lower is better), chrF Mean
| Rank | Subject | Accuracy | Model Match | Provenance | Sampled |
|---|---|---|---|---|---|
| 1 | openai/gpt-4o-2024-05-13 | 75.16 |  GPT-4o openai-gpt-4o | Imported | 2026-05-06 |
| 1 | openai/gpt-4o-2024-08-06 | 74.97 | GPT-4o openai-gpt-4o | Imported | 2026-05-06 |
| 1 | anthropic/claude-3-opus | 74.59 | — | Imported | 2026-05-06 |
| 1 | anthropic/claude-3.5-sonnet-20240620 | 74.40 | — | Imported | 2026-05-06 |
| 1 | deepseek-ai/deepseek-v3-chat | 74.24 | — | Imported | 2026-05-06 |
| 6 | anthropic/claude-3.5-sonnet-20241022 | 72.80 |  Claude 3.5 Sonnet anthropic-claude-3.5-sonnet | Imported | 2026-05-06 |
| 6 | nvidia/nemotron-4-340b-instruct | 72.79 | — | Imported | 2026-05-06 |
| 6 | openai/gpt-4o-mini-2024-07-18 | 72.23 | GPT-4o-mini openai-gpt-4o-mini | Imported | 2026-05-06 |
| 6 | x-ai/grok-2-1212 | 71.60 | — | Imported | 2026-05-06 |
| 6 | x-ai/grok-beta | 71.27 | — | Imported | 2026-05-06 |
| 6 | deepseek-ai/deepseek-v2.5 | 71.14 | — | Imported | 2026-05-06 |
| 12 | qwen/qwen-2.5-72b-instruct | 70.79 |  Qwen2.5 72B Instruct qwen-qwen-2.5-72b-instruct | Imported | 2026-05-06 |
| 12 | lmg-anon/vntl-gemma2-27b | 70.67 | — | Imported | 2026-05-06 |
| 12 | qwen/qwen-2.5-32b-instruct | 70.66 | — | Imported | 2026-05-06 |
| 12 | qwen/qwen-2-72b-instruct | 70.20 | — | Imported | 2026-05-06 |
| 12 | openai/gpt-3.5-turbo-1106 | 69.98 | GPT-3.5 Turbo openai-gpt-3.5-turbo | Imported | 2026-05-06 |
| 12 | meta-llama/llama-3.1-70b-instruct | 69.79 |  Llama 3.1 70B Instruct meta-llama-llama-3.1-70b-instruct | Imported | 2026-05-06 |
| 12 | lmg-anon/vntl-llama3-8b-v2 | 69.52 | — | Imported | 2026-05-06 |
| 12 | meta-llama/llama-3.1-405b-instruct | 69.46 | — | Imported | 2026-05-06 |
| 12 | openai/gpt-4-0613 | 69.28 | GPT-4 openai-gpt-4 | Imported | 2026-05-06 |
| 12 | lmg-anon/vntl-llama3-8b | 69.22 | — | Imported | 2026-05-06 |
| 22 | nvidia/llama-3.1-nemotron-70b-instruct | 69.04 |  Llama 3.1 Nemotron 70B Instruct nvidia-llama-3.1-nemotron-70b-instruct | Imported | 2026-05-06 |
| 22 | anthropic/claude-3.5-haiku-20241022 | 68.94 | — | Imported | 2026-05-06 |
| 22 | qwen/qwen-2-72b-instruct | 68.87 | — | Imported | 2026-05-06 |
| 22 | meta-llama/llama-3.3-70b-instruct | 68.81 | Llama 3.3 70B Instruct meta-llama-llama-3.3-70b-instruct | Imported | 2026-05-06 |
| 22 | qwen/qwq-preview | 68.65 | — | Imported | 2026-05-06 |
| 22 | microsoft/phi-4 (unofficial) | 68.60 |  Phi 4 microsoft-phi-4 | Imported | 2026-05-06 |
| 22 | cohere/command-r-plus-08-2024 | 68.53 | Command R (08-2024) cohere-command-r-08-2024 | Imported | 2026-05-06 |
| 22 | mistralai/mixtral-8x22b-instruct | 68.46 |  Mistral: Mixtral 8x22B Instruct mistralai-mixtral-8x22b-instruct | Imported | 2026-05-06 |
| 22 | mistralai/mistral-large | 67.94 | Mistral Large mistralai-mistral-large | Imported | 2026-05-06 |
| 22 | google/gemma-2-27b-it | 67.93 |  Gemma 2 27B google-gemma-2-27b-it | Imported | 2026-05-06 |
| 22 | anthropic/claude-3-sonnet | 67.72 | — | Imported | 2026-05-06 |
| 22 | cohere/aya-23-35B | 67.71 | — | Imported | 2026-05-06 |
| 22 | rinna/llama-3-youko-70b | 67.65 | — | Imported | 2026-05-06 |
| 22 | webbigdata/C3TR-Adapter | 67.56 | — | Imported | 2026-05-06 |
| 22 | mistralai/Mistral-Nemo-Instruct-2407 | 67.38 | — | Imported | 2026-05-06 |
| 22 | cohere/command-r-plus | 67.19 | — | Imported | 2026-05-06 |
| 22 | anthropic/claude-3-haiku | 67.19 | Claude 3 Haiku anthropic-claude-3-haiku | Imported | 2026-05-06 |
| 39 | meta-llama/llama-3-70b-instruct | 66.91 | Llama 3 70B Instruct meta-llama-llama-3-70b-instruct | Imported | 2026-05-06 |
| 39 | google/gemma-2-27b | 66.74 | — | Imported | 2026-05-06 |
| 39 | qwen/qwen-2.5-14b-instruct | 66.48 | — | Imported | 2026-05-06 |
| 39 | google/gemini-flash-1.5 | 66.20 | — | Imported | 2026-05-06 |
| 39 | cyberagent/Llama-3.1-70B-Japanese-Instruct-2407 | 66.10 | — | Imported | 2026-05-06 |
| 39 | meta-llama/llama-3-70b-instruct | 65.94 | Llama 3 70B Instruct meta-llama-llama-3-70b-instruct | Imported | 2026-05-06 |
| 39 | google/gemini-flash-1.5-8b | 65.93 | — | Imported | 2026-05-06 |
| 39 | qwen/qwen-2.5-14b | 65.92 | — | Imported | 2026-05-06 |
| 39 | google/gemini-pro | 65.89 | — | Imported | 2026-05-06 |
| 39 | lmg-anon/vntl-gemma2-2b | 65.72 | — | Imported | 2026-05-06 |
| 39 | cohere/aya-expanse-32b | 65.50 | — | Imported | 2026-05-06 |
| 39 | rinna/nekomata-14b | 65.39 | — | Imported | 2026-05-06 |
| 39 | cohere/command-r-08-2024 | 65.20 | Command R (08-2024) cohere-command-r-08-2024 | Imported | 2026-05-06 |
| 39 | qwen/qwen-2.5-7b-instruct | 65.18 | Qwen2.5 7B Instruct qwen-qwen-2.5-7b-instruct | Imported | 2026-05-06 |
| 39 | lmg-anon/vntl-13b-v0.2 | 65.02 | — | Imported | 2026-05-06 |
| 54 | cyberagent/calm3-22b-chat | 64.80 | — | Imported | 2026-05-06 |
| 54 | google/gemma-2-9b-it-SPPO-Iter3 | 64.47 | — | Imported | 2026-05-06 |
| 54 | mistralai/mistral-small | 64.41 | — | Imported | 2026-05-06 |
| 54 | google/gemini-pro-1.5 | 64.36 | — | Imported | 2026-05-06 |
| 54 | BeaverAI/Cydonia-22B-v2p-GGUF | 64.10 | — | Imported | 2026-05-06 |
| 54 | rinna/llama-3-youko-8b | 63.95 | — | Imported | 2026-05-06 |
| 54 | rinna/llama-3-youko-70b-instruct | 63.55 | — | Imported | 2026-05-06 |
| 54 | meta-llama/llama-3-70b-instruct | 63.30 | Llama 3 70B Instruct meta-llama-llama-3-70b-instruct | Imported | 2026-05-06 |
| 54 | mistralai/Ministral-8B-Instruct-2410 | 63.25 | — | Imported | 2026-05-06 |
| 54 | lmg-anon/vntl-7b-v0.3.1 | 63.04 | — | Imported | 2026-05-06 |
| 64 | rakuten/rakutenai-7b-instruct | 62.71 | — | Imported | 2026-05-06 |
| 64 | mistralai/mixtral-8x7b-instruct | 62.08 | Mistral: Mixtral 8x7B Instruct mistralai-mixtral-8x7b-instruct | Imported | 2026-05-06 |
| 64 | google/gemma-2-9b-it | 61.94 | — | Imported | 2026-05-06 |
| 64 | cohere/aya-expanse-8b | 61.91 | — | Imported | 2026-05-06 |
| 64 | microsoft/phi-3-medium-4k-instruct | 61.21 | — | Imported | 2026-05-06 |
| 64 | qwen/qwen-2-7b-instruct | 61.13 | — | Imported | 2026-05-06 |
| 64 | cohere/command-r | 61.03 | — | Imported | 2026-05-06 |
| 64 | rinna/gemma-2-baku-2b | 60.77 | — | Imported | 2026-05-06 |
| 72 | meta-llama/llama-3-8b-instruct | 60.19 | Llama 3 8B Instruct meta-llama-llama-3-8b-instruct | Imported | 2026-05-06 |
| 72 | rinna/nekomata-14b-instruction | 60.07 | — | Imported | 2026-05-06 |
| 72 | openchat/openchat-8b | 59.86 | — | Imported | 2026-05-06 |
| 72 | cohere/aya-23-8b | 59.62 | — | Imported | 2026-05-06 |
| 72 | qwen/qwen-2.5-7b | 59.62 | — | Imported | 2026-05-06 |
| 72 | mistralai/Mistral-Nemo-Base-2407 | 58.77 | — | Imported | 2026-05-06 |
| 78 | LLaMAX/LLaMAX3-8B | 57.38 | — | Imported | 2026-05-06 |
| 78 | elyza/Llama-3-ELYZA-JP-8B | 57.15 | — | Imported | 2026-05-06 |
| 78 | mistralai/mistral-7b-instruct-v0.3 | 56.03 | — | Imported | 2026-05-06 |
| 78 | 01-ai/yi-1.5-34b-chat | 55.94 | — | Imported | 2026-05-06 |
| 82 | LLaMAX/LLaMAX3-8B-Alpaca | 55.16 | — | Imported | 2026-05-06 |
| 82 | meta-llama/llama-3-8b-instruct | 55.03 | Llama 3 8B Instruct meta-llama-llama-3-8b-instruct | Imported | 2026-05-06 |
| 82 | nitky/Oumuamua-7b-instruct-v2 | 54.88 | — | Imported | 2026-05-06 |
| 82 | lightblue/qarasu-14b-chat-plus-unleashed | 53.09 | — | Imported | 2026-05-06 |
| 86 | meta-llama/llama-2-13b-chat | 50.24 | — | Imported | 2026-05-06 |
| 87 | 01-ai/yi-1.5-9b-chat | 47.59 | — | Imported | 2026-05-06 |
No matching rows.