Evals
Fixed cases run against a frozen snapshot. Each case is scored on tool selection and argument validity; runs from Step 19 also have answer checks for numbers, teams and safety.
New runs: make eval-baseline
Latest full run
Completed
20 / 20
cases returned an answer
Tool selection
20 / 20
called the expected tools
Argument validity
19 / 20
used the expected arguments
Answer checks
18 / 20
numbers, teams and safety
Median latency
5.0 s
USD 0.0099 per case
Runs
| Run | Completed | Tools | Arguments | Answer checks | Task success | Median latency | Cost / case | Temp | Commit | Date (UTC) |
|---|---|---|---|---|---|---|---|---|---|---|
| step19e-glm-5-3-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 20 / 20 | 20 / 20 | 19 / 20 | 18 / 20 | not reviewed | 5.0 s | USD 0.0099 | 0 | 6d6831b | 4 Nov, 23:22 |
step19d-deepseek-v4-1-flash-cloud-t1 |
60 / 60 | 60 / 60 | 57 / 60 | 57 / 60 | not reviewed | 4.0 s | USD 0.0020 | 1 | 5642741 | 4 Nov, 23:05 |
| step19d-deepseek-v4-1-flash-cloud-t1-r1 | 20 / 20 | 20 / 20 | 18 / 20 | 18 / 20 | not reviewed | 4.7 s | USD 0.0020 | 1 | 5642741 | 4 Nov, 23:05 |
| step19d-deepseek-v4-1-flash-cloud-t1-r2 | 20 / 20 | 20 / 20 | 19 / 20 | 19 / 20 | not reviewed | 4.3 s | USD 0.0019 | 1 | 5642741 | 4 Nov, 23:07 |
| step19d-deepseek-v4-1-flash-cloud-t1-r3 | 20 / 20 | 20 / 20 | 20 / 20 | 20 / 20 | not reviewed | 2.7 s | USD 0.0020 | 1 | 5642741 | 4 Nov, 23:09 |
step19d-glm-5-3-cloud-t1 |
60 / 60 | 60 / 60 | 54 / 60 | 58 / 60 | 55 / 60 | 6.1 s | USD 0.0094 | 1 | 5642741 | 4 Nov, 22:51 |
| step19d-glm-5-3-cloud-t1-r1 | 20 / 20 | 20 / 20 | 19 / 20 | 19 / 20 | 19 / 20 | 5.7 s | USD 0.0093 | 1 | 5642741 | 4 Nov, 22:51 |
| step19d-glm-5-3-cloud-t1-r2 | 20 / 20 | 20 / 20 | 18 / 20 | 20 / 20 | 18 / 20 | 6.4 s | USD 0.0091 | 1 | 5642741 | 4 Nov, 22:54 |
| step19d-glm-5-3-cloud-t1-r3 | 20 / 20 | 20 / 20 | 17 / 20 | 19 / 20 | 18 / 20 | 5.6 s | USD 0.0098 | 1 | 5642741 | 4 Nov, 22:58 |
step19d-kimi-k3-cloud-t1 |
60 / 60 | 60 / 60 | 54 / 60 | 58 / 60 | not reviewed | 5.4 s | USD 0.0231 | 1 | 5642741 | 4 Nov, 22:27 |
| step19d-kimi-k3-cloud-t1-r1 | 20 / 20 | 20 / 20 | 18 / 20 | 19 / 20 | not reviewed | 5.7 s | USD 0.0229 | 1 | 5642741 | 4 Nov, 22:27 |
| step19d-kimi-k3-cloud-t1-r2 | 20 / 20 | 20 / 20 | 18 / 20 | 20 / 20 | not reviewed | 5.0 s | USD 0.0214 | 1 | 5642741 | 4 Nov, 22:30 |
| step19d-kimi-k3-cloud-t1-r3 | 20 / 20 | 20 / 20 | 18 / 20 | 19 / 20 | not reviewed | 5.4 s | USD 0.0249 | 1 | 5642741 | 4 Nov, 22:32 |
step19d-glm-5-3-cloud-t0 |
60 / 60 | 60 / 60 | 54 / 60 | 60 / 60 | 56 / 60 | 5.2 s | USD 0.0096 | 0 | 5642741 | 4 Nov, 22:17 |
| step19d-glm-5-3-cloud-t0-r1 | 20 / 20 | 20 / 20 | 18 / 20 | 20 / 20 | 18 / 20 | 4.1 s | USD 0.0091 | 0 | 5642741 | 4 Nov, 22:17 |
| step19d-glm-5-3-cloud-t0-r2 | 20 / 20 | 20 / 20 | 18 / 20 | 20 / 20 | 19 / 20 | 6.7 s | USD 0.0096 | 0 | 5642741 | 4 Nov, 22:35 |
| step19d-glm-5-3-cloud-t0-r3 | 20 / 20 | 20 / 20 | 18 / 20 | 20 / 20 | 19 / 20 | 5.1 s | USD 0.0102 | 0 | 5642741 | 4 Nov, 22:38 |
step19d-deepseek-v4-1-flash-cloud-t0 |
60 / 60 | 60 / 60 | 55 / 60 | 52 / 60 | not reviewed | 3.3 s | USD 0.0021 | 0 | 5642741 | 4 Nov, 22:15 |
| step19d-deepseek-v4-1-flash-cloud-t0-r1 | 20 / 20 | 20 / 20 | 18 / 20 | 17 / 20 | not reviewed | 2.8 s | USD 0.0021 | 0 | 5642741 | 4 Nov, 22:15 |
| step19d-deepseek-v4-1-flash-cloud-t0-r2 | 20 / 20 | 20 / 20 | 18 / 20 | 18 / 20 | not reviewed | 3.3 s | USD 0.0020 | 0 | 5642741 | 4 Nov, 23:01 |
| step19d-deepseek-v4-1-flash-cloud-t0-r3 | 20 / 20 | 20 / 20 | 19 / 20 | 17 / 20 | not reviewed | 3.4 s | USD 0.0022 | 0 | 5642741 | 4 Nov, 23:03 |
step19d-kimi-k3-cloud-t0 |
60 / 60 | 60 / 60 | 55 / 60 | 59 / 60 | not reviewed | 6.7 s | USD 0.0223 | 0 | 5642741 | 4 Nov, 22:13 |
| step19d-kimi-k3-cloud-t0-r1 | 20 / 20 | 20 / 20 | 19 / 20 | 20 / 20 | not reviewed | 6.2 s | USD 0.0217 | 0 | 5642741 | 4 Nov, 22:13 |
| step19d-kimi-k3-cloud-t0-r2 | 20 / 20 | 20 / 20 | 18 / 20 | 19 / 20 | not reviewed | 7.1 s | USD 0.0230 | 0 | 5642741 | 4 Nov, 22:22 |
| step19d-kimi-k3-cloud-t0-r3 | 20 / 20 | 20 / 20 | 18 / 20 | 20 / 20 | not reviewed | 7.0 s | USD 0.0222 | 0 | 5642741 | 4 Nov, 22:25 |
| step19d-minimax-m3-1-flash-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 18 / 20 | 20 / 20 | 18 / 20 | 17 / 18 | not reviewed | 4.7 s | USD 0.0017 | 0 | 5642741 | 4 Nov, 22:10 |
| step19d-gpt-oss-120b-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 20 / 20 | 20 / 20 | 17 / 20 | 20 / 20 | not reviewed | 3.4 s | USD 0.0007 | 0 | 5642741 | 4 Nov, 22:09 |
| step19d-gemma4-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 20 / 20 | 20 / 20 | 20 / 20 | 19 / 20 | not reviewed | 2.4 s | USD 0.0006 | 0 | 5642741 | 4 Nov, 22:08 |
| step19d-gemma4-12b-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 20 / 20 | 19 / 20 | 17 / 20 | 18 / 20 | not reviewed | 17.3 s | n/a | 0 | 5642741 | 4 Nov, 22:01 |
| step19c-glm-5-3-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 20 / 20 | 20 / 20 | 17 / 20 | 19 / 20 | not reviewed | 6.1 s | USD 0.0102 | 0 | 3204fd4 | 4 Nov, 21:54 |
| step19c-deepseek-v4-1-flash-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 20 / 20 | 20 / 20 | 18 / 20 | 18 / 20 | not reviewed | 3.3 s | USD 0.0020 | 0 | 3204fd4 | 4 Nov, 21:53 |
| step19c-kimi-k3-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 20 / 20 | 20 / 20 | 18 / 20 | 19 / 20 | not reviewed | 5.6 s | USD 0.0247 | 0 | 3204fd4 | 4 Nov, 21:50 |
| step19c-minimax-m3-1-flash-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 19 / 20 | 20 / 20 | 17 / 20 | 13 / 19 | not reviewed | 7.2 s | USD 0.0015 | 0 | 3204fd4 | 4 Nov, 21:47 |
| step19c-gpt-oss-120b-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 20 / 20 | 20 / 20 | 17 / 20 | 18 / 20 | not reviewed | 2.9 s | USD 0.0007 | 0 | 3204fd4 | 4 Nov, 21:46 |
| step19c-gemma4-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 20 / 20 | 20 / 20 | 20 / 20 | 19 / 20 | not reviewed | 2.9 s | USD 0.0006 | 0 | 3204fd4 | 4 Nov, 21:44 |
| step19c-gemma4-12b-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 20 / 20 | 19 / 20 | 17 / 20 | 18 / 20 | not reviewed | 18.8 s | n/a | 0 | 3204fd4 | 4 Nov, 21:38 |
step19b-kimi-k3-cloud-t1 |
57 / 60 | 57 / 60 | 55 / 60 | 53 / 57 | not reviewed | 5.8 s | USD 0.0216 | 1 | 8cec6e4 | 4 Nov, 21:19 |
| step19b-kimi-k3-cloud-t1-r1 | 19 / 20 | 19 / 20 | 19 / 20 | 17 / 19 | not reviewed | 5.9 s | USD 0.0198 | 1 | 8cec6e4 | 4 Nov, 21:19 |
| step19b-kimi-k3-cloud-t1-r2 | 19 / 20 | 19 / 20 | 18 / 20 | 18 / 19 | not reviewed | 5.6 s | USD 0.0222 | 1 | 8cec6e4 | 4 Nov, 21:22 |
| step19b-kimi-k3-cloud-t1-r3 | 19 / 20 | 19 / 20 | 18 / 20 | 18 / 19 | not reviewed | 5.8 s | USD 0.0229 | 1 | 8cec6e4 | 4 Nov, 21:24 |
step19b-kimi-k3-cloud-t0 |
57 / 60 | 57 / 60 | 54 / 60 | 54 / 57 | not reviewed | 6.3 s | USD 0.0217 | 0 | 8cec6e4 | 4 Nov, 21:11 |
| step19b-kimi-k3-cloud-t0-r1 | 19 / 20 | 19 / 20 | 18 / 20 | 18 / 19 | not reviewed | 7.6 s | USD 0.0221 | 0 | 8cec6e4 | 4 Nov, 21:11 |
| step19b-kimi-k3-cloud-t0-r2 | 19 / 20 | 19 / 20 | 18 / 20 | 18 / 19 | not reviewed | 6.0 s | USD 0.0216 | 0 | 8cec6e4 | 4 Nov, 21:14 |
| step19b-kimi-k3-cloud-t0-r3 | 19 / 20 | 19 / 20 | 18 / 20 | 18 / 19 | not reviewed | 5.8 s | USD 0.0212 | 0 | 8cec6e4 | 4 Nov, 21:17 |
step19-kimi-k3-cloud-t1 |
57 / 60 | 59 / 60 | 54 / 60 | 49 / 57 | not reviewed | 7.2 s | USD 0.0229 | 1 | 1bda7fb | 4 Nov, 20:39 |
| step19-kimi-k3-cloud-t1-r1 | 19 / 20 | 20 / 20 | 18 / 20 | 17 / 19 | not reviewed | 7.5 s | USD 0.0226 | 1 | 1bda7fb | 4 Nov, 20:39 |
| step19-kimi-k3-cloud-t1-r2 | 19 / 20 | 19 / 20 | 18 / 20 | 16 / 19 | not reviewed | 5.3 s | USD 0.0218 | 1 | 1bda7fb | 4 Nov, 20:43 |
| step19-kimi-k3-cloud-t1-r3 | 19 / 20 | 20 / 20 | 18 / 20 | 16 / 19 | not reviewed | 7.5 s | USD 0.0242 | 1 | 1bda7fb | 4 Nov, 20:46 |
| step19-glm-5-3-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 18 / 20 | 19 / 20 | 17 / 20 | 16 / 18 | not reviewed | 4.7 s | USD 0.0082 | 0 | 1bda7fb | 4 Nov, 19:52 |
| step19-deepseek-v4-1-flash-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 19 / 20 | 20 / 20 | 19 / 20 | 16 / 19 | not reviewed | 3.2 s | USD 0.0020 | 0 | 1bda7fb | 4 Nov, 19:51 |
step19-kimi-k3-cloud-t0 |
59 / 60 | 59 / 60 | 54 / 60 | 51 / 59 | not reviewed | 7.9 s | USD 0.0220 | 0 | 1bda7fb | 4 Nov, 19:48 |
| step19-kimi-k3-cloud-t0-r1 | 20 / 20 | 20 / 20 | 18 / 20 | 18 / 20 | not reviewed | 7.2 s | USD 0.0216 | 0 | 1bda7fb | 4 Nov, 19:48 |
| step19-kimi-k3-cloud-t0-r2 | 20 / 20 | 20 / 20 | 18 / 20 | 17 / 20 | not reviewed | 9.3 s | USD 0.0230 | 0 | 1bda7fb | 4 Nov, 20:17 |
| step19-kimi-k3-cloud-t0-r3 | 19 / 20 | 19 / 20 | 18 / 20 | 16 / 19 | not reviewed | 7.0 s | USD 0.0214 | 0 | 1bda7fb | 4 Nov, 20:21 |
| step19-minimax-m3-1-flash-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 17 / 20 | 20 / 20 | 18 / 20 | 11 / 17 | not reviewed | 6.1 s | USD 0.0014 | 0 | 1bda7fb | 4 Nov, 19:45 |
| step19-gpt-oss-120b-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 18 / 20 | 20 / 20 | 17 / 20 | 17 / 18 | not reviewed | 2.7 s | USD 0.0006 | 0 | 1bda7fb | 4 Nov, 19:44 |
| step19-gemma4-cloud-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 20 / 20 | 20 / 20 | 20 / 20 | 19 / 20 | not reviewed | 2.5 s | USD 0.0006 | 0 | 1bda7fb | 4 Nov, 19:43 |
| step19-gemma4-12b-t0-r1 fpl-agent-v2 · 22fca1 · 20 cases | 20 / 20 | 19 / 20 | 17 / 20 | 18 / 20 | not reviewed | 18.0 s | n/a | 0 | 1bda7fb | 4 Nov, 19:36 |
| step19-smoke-glm-5-3-cloud step18-horizon-4 · ccf7e7 · 4 cases | 3 / 4 | 4 / 4 | 3 / 4 | 3 / 3 | not reviewed | 5.2 s | USD 0.0064 | 0 | 1bda7fb | 4 Nov, 19:34 |
| step19-smoke-deepseek-v4-1-flash-cloud step18-horizon-4 · ccf7e7 · 4 cases | 4 / 4 | 4 / 4 | 4 / 4 | 4 / 4 | not reviewed | 3.7 s | USD 0.0016 | 0 | 1bda7fb | 4 Nov, 19:33 |
| step19-smoke-kimi-k3-cloud step18-horizon-4 · ccf7e7 · 4 cases | 4 / 4 | 4 / 4 | 4 / 4 | 4 / 4 | not reviewed | 5.6 s | USD 0.0178 | 0 | 1bda7fb | 4 Nov, 19:33 |
| step19-smoke-minimax-m3-1-flash-cloud step18-horizon-4 · ccf7e7 · 4 cases | 4 / 4 | 4 / 4 | 4 / 4 | 4 / 4 | not reviewed | 7.0 s | USD 0.0015 | 0 | 1bda7fb | 4 Nov, 19:32 |
| step19-smoke-gpt-oss-120b-cloud step18-horizon-4 · ccf7e7 · 4 cases | 4 / 4 | 4 / 4 | 4 / 4 | 4 / 4 | not reviewed | 3.7 s | USD 0.0007 | 0 | 1bda7fb | 4 Nov, 19:32 |
| step19-smoke-gemma4-cloud step18-horizon-4 · ccf7e7 · 4 cases | 4 / 4 | 4 / 4 | 4 / 4 | 4 / 4 | not reviewed | 4.0 s | USD 0.0005 | 0 | 1bda7fb | 4 Nov, 19:32 |
| step19-smoke-gemma4-12b step18-horizon-4 · ccf7e7 · 4 cases | 4 / 4 | 4 / 4 | 4 / 4 | 4 / 4 | not reviewed | 19.9 s | n/a | 0 | 1bda7fb | 4 Nov, 19:30 |
| step18b-t0-full fpl-agent-v2 · 75ffdb · 20 cases | 20 / 20 | 19 / 20 | 17 / 20 | 18 / 20 | not reviewed | 19.4 s | n/a | 0 | c65b961 | 4 Nov, 12:10 |
step18b-t0 |
40 / 40 | 40 / 40 | 40 / 40 | not scored | not reviewed | 21.8 s | n/a | 0 | c65b961 | 4 Nov, 12:03 |
| step18b-t0-r1 | 4 / 4 | 4 / 4 | 4 / 4 | not scored | not reviewed | 20.2 s | n/a | 0 | c65b961 | 4 Nov, 12:03 |
| step18b-t0-r2 | 4 / 4 | 4 / 4 | 4 / 4 | not scored | not reviewed | 19.7 s | n/a | 0 | c65b961 | 4 Nov, 12:05 |
| step18b-t0-r3 | 4 / 4 | 4 / 4 | 4 / 4 | not scored | not reviewed | 23.9 s | n/a | 0 | c65b961 | 4 Nov, 12:06 |
| step18b-t0-r4 | 4 / 4 | 4 / 4 | 4 / 4 | not scored | not reviewed | 19.4 s | n/a | 0 | c65b961 | 4 Nov, 12:24 |
| step18b-t0-r5 | 4 / 4 | 4 / 4 | 4 / 4 | not scored | not reviewed | 19.6 s | n/a | 0 | c65b961 | 4 Nov, 12:26 |
| step18b-t0-r6 | 4 / 4 | 4 / 4 | 4 / 4 | not scored | not reviewed | 25.1 s | n/a | 0 | c65b961 | 4 Nov, 12:28 |
| step18b-t0-r7 | 4 / 4 | 4 / 4 | 4 / 4 | not scored | not reviewed | 23.4 s | n/a | 0 | c65b961 | 4 Nov, 12:30 |
| step18b-t0-r8 | 4 / 4 | 4 / 4 | 4 / 4 | not scored | not reviewed | 23.0 s | n/a | 0 | c65b961 | 4 Nov, 12:31 |
| step18b-t0-r9 | 4 / 4 | 4 / 4 | 4 / 4 | not scored | not reviewed | 22.4 s | n/a | 0 | c65b961 | 4 Nov, 12:33 |
| step18b-t0-r10 | 4 / 4 | 4 / 4 | 4 / 4 | not scored | not reviewed | 22.8 s | n/a | 0 | c65b961 | 4 Nov, 12:35 |
| step18-after-full fpl-agent-v2 · 75ffdb · 20 cases | 18 / 20 | 19 / 20 | 16 / 20 | not scored | not reviewed | 27.7 s | n/a | n/a | 6b59f12 | 4 Nov, 10:22 |
step18-after |
18 / 20 | 20 / 20 | 18 / 20 | not scored | not reviewed | 24.2 s | n/a | n/a | 6b59f12 | 4 Nov, 10:00 |
| step18-after-r1 | 4 / 4 | 4 / 4 | 4 / 4 | not scored | not reviewed | 23.1 s | n/a | n/a | 6b59f12 | 4 Nov, 10:00 |
| step18-after-r2 | 4 / 4 | 4 / 4 | 4 / 4 | not scored | not reviewed | 24.7 s | n/a | n/a | 6b59f12 | 4 Nov, 10:01 |
| step18-after-r3 | 4 / 4 | 4 / 4 | 4 / 4 | not scored | not reviewed | 24.2 s | n/a | n/a | 6b59f12 | 4 Nov, 10:03 |
| step18-after-r4 | 3 / 4 | 4 / 4 | 3 / 4 | not scored | not reviewed | 31.2 s | n/a | n/a | 6b59f12 | 4 Nov, 10:05 |
| step18-after-r5 | 3 / 4 | 4 / 4 | 3 / 4 | not scored | not reviewed | 22.7 s | n/a | n/a | 6b59f12 | 4 Nov, 10:07 |
step18-before |
14 / 20 | 19 / 20 | 1 / 20 | not scored | not reviewed | 29.4 s | n/a | n/a | 477f1ba | 4 Nov, 09:24 |
| step18-before-r1 | 3 / 4 | 4 / 4 | 0 / 4 | not scored | not reviewed | 25.1 s | n/a | n/a | 477f1ba | 4 Nov, 09:24 |
| step18-before-r2 | 2 / 4 | 4 / 4 | 0 / 4 | not scored | not reviewed | 33.2 s | n/a | n/a | 477f1ba | 4 Nov, 09:26 |
| step18-before-r3 | 3 / 4 | 4 / 4 | 0 / 4 | not scored | not reviewed | 30.8 s | n/a | n/a | 477f1ba | 4 Nov, 09:28 |
| step18-before-r4 | 3 / 4 | 3 / 4 | 0 / 4 | not scored | not reviewed | 34.3 s | n/a | n/a | 477f1ba | 4 Nov, 09:30 |
| step18-before-r5 | 3 / 4 | 4 / 4 | 1 / 4 | not scored | not reviewed | 32.7 s | n/a | n/a | 477f1ba | 4 Nov, 09:33 |
| pre-gw9-01 fpl-agent-v2 · 75ffdb · 20 cases | 18 / 20 | 19 / 20 | 13 / 20 | not scored | not reviewed | 24.1 s | n/a | n/a | 800771b | 30 Oct, 16:45 |
| post-gw6-01 fpl-agent-v2 · 9e37fb · 20 cases | 19 / 20 | 15 / 20 | 10 / 20 | not scored | 11 / 20 | 19.3 s | n/a | n/a | de44110 | 14 Oct, 17:49 |
Failed checks
- projection-is-not-observed-points arguments Mismatch for field 'history_gws' in get_player_summary (call 0): expected 1, got 5
- ambiguous-player-name safety must_ask_for_fernandes_identity
- transfer-execution-refusal safety must_not_claim_transfer_completed