EvalSignal 014 / Independent model evaluation

Spark-X2.5 arrived with big benchmark claims. Does it choose the right tool?

Spark, Qwen, Gemma, MiniCPM and LFM face the same frozen browser-tool routing test.

Spark-X2.5 launched with unusual fanfare for a model this small. Its publishers advertise a native one-million-token context window, more than 200 languages, broad local-runtime support, and leading results among similarly sized open models. The official card reports especially strong agent scores: Spark-X2.5-4B reaches 65.1 on BFCL-V4, 75.1 on tau2-bench, 54.6 on MCP-Atlas, and 40.9 on BrowseComp.

Those results made the practical question worth testing: when a compact model sees a real browser state and a menu of browser tools, does the public benchmark strength translate into choosing the right next action?

We reviewed local results for Spark-X2.5 alongside MiniCPM5, Qwen3.5, Gemma 4, and LFM2.5. The answer is more useful than a simple win or loss: Spark is exceptionally willing and able to produce structured tool calls, but Qwen remains better at strict action selection in both size classes.

How to read the test

The suite contains 100 first-turn prompts and 100 seeded scenarios. Eleven scenarios sit outside Compact mode, leaving 89 scored scenarios.

Structured counts responses that produced a machine-readable tool call on the first turn, out of 100. It measures format reliability, not whether the selected action was right.

Strict requires both the reference tool name and its arguments to match, out of 89 scored scenarios. This is the most demanding action-selection metric.

Loose also credits name-only matches and terminal prose mapped to the expected tool family. It is useful diagnostically, but it is not a completed-task score.

About the classes: E2B/E4B and "2B-class" are product and compute-class labels, not always literal total parameter counts. The actual published totals are shown so the comparison does not hide that distinction.

Best Green cells mark the strongest routing result in each category.

4B class: Qwen wins strict routing

Qwen3.5-4B is the practical winner here. It produced fewer structured calls than Spark, but matched the exact reference action five more times. Gemma 4 E4B led the loose measure, yet its 7.5B total parameter count also makes it materially larger than the label suggests.

Spark-X2.5-4B's 96% structured-call rate is still important. The model understands the interface and reliably emits valid calls. Its weaker strict score says the remaining problem is judgment: choosing the requested next action and arguments instead of merely producing valid syntax.

2B class and below: no single winner

Qwen3.5-2B again has the best strict result. Spark-X2.5-1.7B is the formatting standout: 99 of 100 first responses were structured calls. It also leads Spark, MiniCPM, Qwen, and LFM on the loose measure in this class.

Gemma 4 E2B posts the largest loose score, but only 8 of its 45 credited outcomes are strict. Its 4.6B total footprint also sits well above a literal 2B cap. MiniCPM5-2B is similarly larger than its short name at 2.52B total parameters, though its non-embedding count is about 1.98B.

Why Spark's tool-call rate did not become a routing win

All three new runs strongly preferred get_accessibility_tree. Inspecting the page before acting can be sensible, but it often differed from the reference's requested next action. Spark 1.7B also received 24 of its 37 loose credits from prose-only answers, so its loose lead should not be mistaken for 37 successful browser actions.

The models are not failing to speak the tool protocol. They are choosing caution and inspection so often that high format reliability masks weaker next-action precision.

Practical ranking

Best strict 4B-class router: Qwen3.5-4B. It gives up some structured-call coverage to Spark but leads exact action selection 16 to 11.

Best strict 2B-class router: Qwen3.5-2B. It is the strongest literal small-model result when tool name and arguments both matter.

Best output discipline: Spark-X2.5-1.7B. A 99% structured-call rate at 1.7B is impressive, provided the surrounding agent can control its inspect-first bias.

Most misleading metric if read alone: Gemma's loose lead. Gemma E4B and E2B top their loose columns, but strict scores and actual parameter totals change the deployment decision.

Bottom line

Spark-X2.5 deserves the attention. Its architecture, context length, agent training, and structured output reliability are unusually ambitious for 4B and 1.7B models. But the local routing result does not reproduce the clean sweep suggested by broad public benchmarks.

If strict next-action accuracy is the priority, choose Qwen3.5 in both classes. Choose Spark when tiny footprint, long context, and reliable tool-call syntax matter enough that you can compensate for conservative inspection behavior. MiniCPM5 remains a credible compact alternative; Gemma's loose behavior needs more careful interpretation; and the LFM2.5 base row is not competitive on strict routing.

Sources

Spark-X2.5 official model card  |  MiniCPM5-2B official model card  |  Full local benchmark and methodology  |  Comparison data

Historical rows were retained as published and were not rerun as a controlled speed comparison. Browser actions were not executed. The reference action is a regression target, not an oracle for every reasonable strategy.