EvalSignal 017 / Independent model evaluation
Its first-action results look unusually strong for the measured cost. One latency number keeps the verdict from being a simple yes.
MiMo-V2.6-Pro is a large multimodal mixture-of-experts model, but our reason to pay attention is practical: can it make a useful first decision in a browser-agent loop without frontier-model economics? In our 100-case planner run, it produced 95 valid native tool calls, matched the benchmark's ideal tool on 44 cases, and cost $0.1246 for the full replay.
Across a 17-model peer pool, MiMo Pro reached 43.6% exact-action consensus and ranked third. It matched the archived ideal tool 44 times, tied with Kimi K3, and matched the complete ideal tool-plus-arguments on 16 cases. Consensus is agreement, not ground truth; the separate ideal-action rubric helps keep the score grounded.
| Planner | Ideal tool | Exact ideal | Median |
|---|---|---|---|
| MiMo-V2.6-Pro | 44 / 100 | 16 / 100 | 6.12s |
| Kimi K3 | 44 / 100 | 18 / 100 | 7.54s |
| GLM-5.3 Flash | 41 / 100 | 13 / 100 | 7.24s |
| Tencent HY4 Preview | 36 / 100 | 16 / 100 | 5.81s |
| Qwen3.8 Flash | 36 / 100* | 12 / 100 | 2.55s |
*Qwen's 36 includes three appropriate terminal answers to knowledge questions. Qwen/GLM, Kimi, and the October MiMo/HY4 runs were conducted on different dates and routes; this table combines historical results, not one simultaneous rerun.
The measured MiMo Pro replay came to about 12.5 cents per 100 planner calls on its tested OpenRouter route. That is an attractive price for a model with this scale and tool-use profile, though not the cheapest row: Qwen and GLM's separate runs reported lower costs. Provider, cache, output length, and date differ, so these are observed bills, not a controlled price shootout or today's guaranteed rates.
The operational caveat is its 46.68-second p95. The median was 6.12 seconds, but a long tail like this can make interactive agents feel inconsistent. A strong planner is useful only if the serving route responds reliably when the user is waiting.
In a separate 100-screenshot run, MiMo Pro passed 77 cases with a 94.0% mean rubric score. Qwen3.8 Flash passed 79 and GLM-5.3 Flash 76. That puts MiMo Pro in the competitive group, not clearly ahead; these are single runs and the small differences should not be overread.
MiMo-V2.6-Pro is a credible frontier-class agent candidate at a modest observed API cost: excellent structured-call validity, top-tier ideal-tool selection, and useful multimodal capability. But it is not a proven replacement for the strongest planners, not the cheapest model tested, and its 46.68-second tail is a real deployment concern.
Most importantly, this suite records one opening decision. It does not execute tools or measure whether an agent finishes a long browser task. “Frontier-level” here describes the ambition and scale of the candidate, not a universal capability equivalence claim.
MiMo Pro and HY4 planner benchmark
