EvalSignal 004 / Frontier models
A fixed WebBrain browser-planner evaluation ran 13 models through 100 cases: 1,300 comparable calls using identical page state, instructions, tool definitions, and output limits.
Rather than appointing Claude or another model as judge, the analysis used peer consensus. Each model's first action was compared with the other twelve models on the same case.
Exact consensus means choosing the same normalized tool and arguments. Tool consensus ignores argument differences. Ideal tool measures agreement with the benchmark's deterministic expected tool.
| Model | Exact | Valid | Ideal | Median |
|---|---|---|---|---|
| DeepSeek V4 Flash | 51.8% | 90/90 | 39 | 1.56s |
| Tencent HY3 | 48.3% | 90/90 | 41 | 5.12s |
| Claude Sonnet 5 | 44.0% | 98/98 | 47 | 4.06s |
| Gemini 3.6 Flash | 43.5% | 100/100 | 36 | 1.96s |
| GLM-5.2 | 43.3% | 86/89 | 39 | 1.73s |
| Kimi K3 | 41.0% | 97/97 | 44 | 7.54s |
Kimi K3 and Sonnet 5 selected the same tool family on 86 of 100 cases. Kimi produced 97 valid calls and selected 44 ideal tools, compared with Sonnet's 98 and 47.
That is no longer a categorical capability gap. Kimi behaves like a frontier multimodal peer with a less attractive serving profile: it was slower, produced a 26.15-second p95, and carried a higher published token price at test time. Its lower observed replay bill depended heavily on prompt caching.
DeepSeek and HY3 have a critical browser-agent limitation.
Both tested routes are text-only. Accessibility trees and extracted text cannot reliably represent charts, canvas applications, visual editors, malformed interfaces, image-based controls, or pages with poor semantic structure. When the decisive state exists only in a screenshot, neither model can inspect it.
DeepSeek and HY3 are compelling planners when another component handles vision. They are not complete browser-agent models by themselves.
The tested GLM-5.2 route was also text-only, but GLM-5.2 Vision NVFP4 attaches a MoonViT-3d vision tower to the frozen GLM text backbone through a newly trained projector. This makes it a credible visual companion or alternative to the text route.
It was not scored in this benchmark and requires infrastructure-scale hardware, reportedly four to eight B200 GPUs depending on context length. It is not a local workstation option, but it gives GLM a concrete vision story that DeepSeek V4 Flash and HY3 do not yet have.
This was a first-action planner test. Models did not browse for twenty turns, recover from mistakes, inspect screenshots, or revise their plans after receiving tool results.
The result is narrower than "best AI model." It shows that Chinese open-weight models have become peers in structured planning, while still carrying important differences in modality, latency, infrastructure, and reliability.
Bottom line: "Chinese models are cheaper but clearly one tier behind" is no longer a useful default. The remaining gap is increasingly about deployment quality and multimodality, not basic planning capability.
Benchmark source: WebBrain's open-source browser-agent test suite. Inspect the reproducible suite and raw results.