EvalSignal field note

What makes a local browser agent choose the right tool?

Qwen and Gemma take different architectural and post-training routes to browser-agent planning.

WebBrain is an open-source browser-agent library for Chrome and Firefox. It lets an AI model inspect pages and operate the browser using tools for navigation, clicking, typing, scrolling, reading page structure, and completing multi-step workflows.

Giving a model browser access creates a specific evaluation problem. The model must do more than understand the instruction: it must choose the correct first action from a 41-tool browser-control schema. It must also recognize when it should not touch the browser yet.

WebBrain's public evaluation uses a frozen 100-case suite built around that decision. Every model receives the same browser states, user instructions, system prompt, and tool definitions. The benchmark then records what the model tries to do first.

EvalSignal independently reviewed and compared the published results. The underlying benchmark, harness, and model runs are WebBrain's; the complete public test directory is linked below for inspection and reproduction.

This issue compares three architectural families:

• Qwen 3.6 27B: Alibaba's full 27B hybrid-attention model.
• Qwen 3.5 35B-A3B MoE lineage: roughly 35B total parameters, with about 3B activated for each token.
• Gemma 4 31B: Google DeepMind's dense, multimodal 31B model.

How to read the results

Parsed calls measure whether the response could be converted into a machine-readable browser tool call. A model may understand the task but still fail here by answering in prose or producing malformed arguments.

Exact is the strictest result. The selected tool and its arguments must match the hand-written ideal first action. Small argument differences can make an otherwise reasonable answer fail this metric.

Ideal tool ignores argument differences and asks whether the model selected the expected tool name. For example, choosing navigate is credited even if the generated URL differs slightly.

Sonnet match asks whether the model made the same first routing decision as Claude Sonnet 4.6, including decisions not to call a tool. This measures similarity to a strong production reference, not general intelligence or universal correctness.

Median latency is the middle first-response time across the 100 cases. It includes the tested quantization and local serving configuration, so it should not be treated as a hardware-independent architecture benchmark.

The three practical family leaders

Best Green cells mark the strongest result in each column; ties are highlighted.

The table makes Gemma look like the straightforward winner. The family-level results are more interesting because each intervention changed a different aspect of planner behavior.

Qwen 3.6 27B: speed versus restraint

Qwen 3.6 27B is a full-parameter model using a hybrid of Gated DeltaNet and gated attention. The published WebBrain results compare the original saved row with two derivatives: NVIDIA's NVFP4 deployment and BottleCap AI's ThinkingCap fine-tune.

Variant	Parsed	Exact	Ideal	Sonnet	Median
Qwen 3.6 27B	92	18	37	77%	10.18s
Qwen 3.6 27B NVFP4	96	18	38	74%	1.76s
ThinkingCap 27B INT4	91	19	35	77%	2.25s

NVIDIA NVFP4: faster, but more eager

NVFP4 is not a behavioral fine-tune. NVIDIA used its Model Optimizer to compress linear weights and activations into a 4-bit floating-point serving format.

Median latency fell from 10.18s to 1.76s, parsed calls increased from 92 to 96, and ideal-tool matches rose from 37 to 38. But all-case Sonnet alignment fell from 77% to 74%.

The loss came from boundary cases. Sonnet used no tool in eight cases. The original Qwen matched six of those decisions; NVFP4 matched only three. Quantization made this deployment dramatically faster and highly parseable, but also more likely to act on an ambiguous instruction.

BottleCap AI ThinkingCap: fewer thinking tokens, better boundaries

ThinkingCap was created by BottleCap AI. It is a minimally invasive Qwen 3.6 27B fine-tune intended to retain the base model's capabilities while reducing thinking-token usage. The evaluated checkpoint was an INT4 AutoRound deployment.

ThinkingCap tied the original Qwen at 77% Sonnet alignment, raised exact matches from 18 to 19, and reduced median latency to 2.25s. Most importantly, it matched seven of Sonnet's eight no-tool decisions.

Qwen 27B verdict: NVFP4 is the throughput option; ThinkingCap is the safer autonomous planner.

Qwen 35B-A3B MoE: two kinds of agent training

A Mixture-of-Experts model contains many parameter blocks but activates only a small subset for each token. The 35B-A3B naming means approximately 35B total parameters with around 3B active during inference.

The published baseline is Qwen 3.6 35B-A3B. Agents-A1 and Ornith use the same broad Qwen 3.5 MoE architectural lineage, but they are independently post-trained agent models rather than simple Qwen quantizations.

Variant	Parsed	Exact	Ideal	Sonnet	Median
Qwen 3.6 35B-A3B	90	18	38	70%	10.29s
Agents-A1 AWQ INT4	88	15	33	75%	1.66s
Ornith 1.0 35B NVFP4	88	21	36	71%	2.38s

InternScience Agents-A1: broad agentic post-training

Agents-A1 was developed by InternScience for long-horizon search, engineering, scientific research, instruction following, and tool use.

Its training combines full-domain supervised fine-tuning, specialized domain-teacher models, and multi-domain on-policy distillation. The evaluated checkpoint used AWQ INT4; AWQ is the deployment compression layer, while the behavioral changes come from InternScience's agentic training.

Against Qwen 3.6 35B-A3B, Agents-A1 raised Sonnet alignment from 70% to 75% and cut median latency from 10.29s to 1.66s. Exact matches fell from 18 to 15 and ideal-tool matches from 38 to 33. Its training transferred to browser routing, but not uniformly.

DeepReinforce Ornith: reinforcement learning for coding agents

Ornith 1.0 35B was developed by DeepReinforce. It is post-trained with reinforcement learning for agentic coding and learns both solution trajectories and the scaffolding used to produce them. The evaluated checkpoint used NVFP4.

Ornith produced the highest strict exact score in this comparison: 21/100. But its coding-agent specialization did not transfer completely to browser routing; Sonnet alignment reached 71%, below Agents-A1's 75%.

35B MoE verdict: Agents-A1 is the better general browser router; Ornith is the stronger strict-action specialist.

Gemma 4 31B: AutoRound versus QAT

Gemma 4 31B is a dense 30.7B-parameter model with local sliding-window and periodic global attention. It is also natively multimodal. The published results compare two 4-bit deployments:

Variant	Parsed	Exact	Ideal	Sonnet	Median
Intel Gemma 4 31B AutoRound INT4	88	14	34	74%	0.63s
Gemma 4 31B QAT w4a16	95	19	37	77%	0.55s

Intel AutoRound: compressing an already-trained model

AutoRound is Intel's post-training quantization toolkit for LLMs and vision-language models. It uses a calibration set and sign-gradient optimization to find better low-bit rounding choices without retraining the full model.

The Intel AutoRound Gemma checkpoint was already fast, but it retained less planner behavior during compression: 88 parsed calls, 14 exact matches, and 34 ideal-tool matches.

Google QAT: teaching the model about quantization

Gemma 4 31B QAT takes a different approach. Quantization-Aware Training exposes the model to simulated low-precision behavior during training, allowing it to adapt before export. The tested w4a16 checkpoint uses 4-bit weights with 16-bit activations.

Against Intel's AutoRound result, QAT added seven parsed calls, five exact matches, three ideal-tool matches, and three Sonnet-alignment points. It also reduced no-tool responses from 12 to 5 while slightly improving median and tail latency.

Final ranking

Best planner balance: Gemma 4 31B QAT. It combined 95 parsed calls, top-tier Sonnet alignment, competitive exact accuracy, and 0.55s median latency.

Best Qwen 27B planner: ThinkingCap. BottleCap AI's fine-tune preserved the base model's alignment while substantially improving speed and no-tool restraint.

Best high-throughput Qwen: NVFP4. It was fast and highly parseable, but needs stronger external safeguards for ambiguous instructions.

Best general 35B MoE agent: Agents-A1. InternScience's broad agent training transferred better to browser routing than Ornith's coding-focused specialization.

Best strict-action specialist: Ornith. It produced the most exact first actions but was less aligned with the reference on the complete suite.

The broader lesson is that parameter count alone did not determine the result. QAT improved Gemma's low-precision planner quality, ThinkingCap changed Qwen's decision boundaries, and agent-specific post-training moved the 35B MoE family in different directions depending on the training objective.

Reproduce the evaluation

The complete WebBrain LLM evaluation directory is public. It contains the runner scripts, frozen suite, browser tool schemas, scoring utilities, and result conventions used for these comparisons.

Open the full test/llm suite on GitHub

Sources

Detailed WebBrain evaluations

Qwen 27B NVFP4  |  ThinkingCap 27B  |  Agents-A1  |  Ornith 35B  |  Gemma 31B QAT

Primary model and quantization references

BottleCap AI ThinkingCap  |  InternScience Agents-A1  |  DeepReinforce Ornith  |  Intel AutoRound

All rows use the same frozen 100-case suite. Sonnet match measures first-action similarity to Claude Sonnet 4.6, not general intelligence. Latency reflects the tested checkpoint, quantization, hardware, and serving configuration.

EvalSignal field note

Which local vision model actually understands a browser screenshot?

We compared eight local vision-model variants on one realistic browser-agent screenshot: a Google password screen with small account text, a focused empty password field, a red validation state, an error icon, a checkbox, a secondary link, and a primary button.

The target was not generic image description. Browser agents need exact OCR, UI affordance detection, state extraction, and calibrated uncertainty. The last point mattered more than expected.

The test screenshot: small identifier text, dropdown affordance, empty password field, validation message, and visible error state.

Result

Qwen 3.6-35B-A3B is the best default. It was fast for its quality class, multilingual, read visible strings correctly, surfaced the red error state, and was the only model that used the unknowns section honestly instead of pretending every ambiguous visual state was certain.

Qwen 3.5-9B-int4 is the budget surprise. Around the 6 GB VRAM bracket, it correctly treated the account chip as a dropdown and captured all visible strings. It missed the red border and icon, and also failed the uncertainty test, but it is stronger than its size suggests.

Nemotron Omni 30B is interesting but narrow. It had cheaper image tokens and the best explicit dropdown classification, but was slower in this setup, English-only, and overconfident on ambiguity. MiMo V2.5 had strong OCR/state extraction but was too slow at the tested quant. Gemma remained weak for identifier OCR.

Comparison

Model	Best read	Main issue
Qwen 3.6-35B-A3B	Best overall; only honest uncertainty result	Needs roughly 22 GB VRAM
Qwen 3.6-27B	Strong smaller option with visual-state fidelity	Less robust than 35B-A3B on uncertainty
Qwen 3.5-9B-int4	Best budget option; good OCR + dropdown structure	Missed visual error cues
Nemotron Omni 30B	Cheaper tokens; explicit dropdown classification	English-only; overconfident; slower here
MiMo V2.5 IQ3_S	Strong OCR and state extraction	Latency class was impractical in this test
Gemma 4 family	Low image-token cost	Identifier OCR failures

Why this matters

A browser agent can recover from many perception errors by checking the DOM or accessibility tree. What it cannot recover from is a vision caption that confidently reports a guessed state as fact. The useful model is not just the one with the best OCR or the most parameters. It is the one that knows when a visual cue is ambiguous and should be verified before action.

A practical routing policy now looks simple: use Qwen 3.6-35B-A3B if you have the VRAM headroom, Qwen 3.6-27B if you need a smaller strong model with visual-state fidelity, and Qwen 3.5-9B-int4 if you are operating in the 6-8 GB range and can tolerate missing some color/error cues.

Source notes

Round 1   Round 2   Round 3   Round 4