EvalSignal 018 / Independent model evaluation
New US open-weight releases are meaningful. The latest independent scores still put them just behind the leaders.
American AI had a busy week. Whether it had a good one is another question.
Reflection released Beam. Meta released Muse Spark Open. NVIDIA released Nemotron 3.5 Super VL alongside its smaller Bolt models. These are meaningful additions to the US open-weight ecosystem. But the latest Artificial Analysis Intelligence Index scores give little reason to declare a comeback.
| Model | Index |
|---|---|
| DeepSeek V4.1 Flash 1017 | 45 |
| Qwen3.8 Flash-Next | 40 |
| Meta Muse Spark Open | 39 |
| Reflection Beam | 30 |
| NVIDIA Nemotron 3.5 Super VL | 26 |
Scores are from the current Artificial Analysis Intelligence Index snapshot. It is a broad composite, not a complete verdict on every capability or deployment.
The three US releases scored 39, 30, and 26. Meta is plainly competitive: a one-point gap from Qwen deserves perspective. But on this composite, the new US models have not taken the lead.
Open weights do not automatically mean open source. Muse Spark Open is released under Meta's Muse Model License, with its own terms; developers should read the license before building a product around it. More broadly, downloadable weights can support private deployment and adaptation, but practical rights differ by model. A capable open model can still become the foundation for many products, tools, and deployment practices, with influence beyond the lab that trained it.
DeepSeek says its October 17 V4.1 Flash 1017 update improved coding and agent benchmarks, shortened outputs, and reduced task runtimes. Artificial Analysis independently scores the updated model at 45, up from 39 for the earlier version. If quality improves while using fewer tokens and less time, that matters commercially. The vendor's benchmark claims and the independent index are different kinds of evidence, so they should not be conflated.
There is a second DeepSeek story beyond the model: DeepSeek open-sourced DeepInfer, the inference framework it says powers its large-scale production services. The initial release supports V4.1 Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, with optimized kernels, distributed inference features, and DSpark decoding. It is not a ready-made one-command server: DeepSeek notes that a KV-cache system, prefill/decode disaggregation, and load balancing are still outside the release. Even with that caveat, opening up the serving stack is significant. Inference infrastructure is part of how a model becomes fast and affordable, not just an implementation detail.
MiniMax M3.1 Flash is another signal. In its October 19 announcement, MiniMax reports 73.7% on DeepSWE v1.1 and 61.7 points on OSWorld V2, along with nearly three-times-faster online inference and a 70% reduction in usage costs versus its predecessor. These are company-reported results, not an independent comparison. If reproduced, the combination would make agent work cheaper as well as more capable.
The pressure extends down to small models. Spark-X2.6-4B targets on-device coding, tool use, and agent workflows. Its developers call it best-in-class for its size and claim it can match or exceed larger 9B models, including Qwen3.5-9B, on several benchmarks. That is the publisher's comparison, not a result we have independently reproduced. If it holds under a controlled rerun, a 4B model approaching a strong 9B model would be a meaningful on-device efficiency gain. See the project results and model card.
Flint is Recursive's first model. Recursive, co-founded by AI researcher Richard Socher, is pursuing a larger idea called the Eureka Machine: an automated research loop that links ideation, implementation, and validation, with the goal of improving AI research recursively. Recursive says that system shaped each stage of Flint's development. Its early research results are an encouraging proof point, not evidence that self-improving AI is already solved.
As a model, Flint has 39.8B total parameters, with about 2.5B active per token. Recursive's published comparisons put it close to Qwen3.8-27B on SWE-bench Verified, while showing it behind on Terminal-Bench 2.1 and LiveCodeBench. That is a promising but mixed profile, not a universal win.
Recursive also reports more than 125 tokens per second on an M4 Pro MacBook Pro and more than 150 on a 64GB RTX Spark at short contexts using its optimized engine. Those are vendor measurements. If they hold in independent use, Flint could offer a useful balance of coding capability, latency, and local control.
Could it fit on an RTX 5090? Flint's NVFP4 checkpoint is listed at about 22.9GB, while the 5090 has 32GB of VRAM. That makes a fully GPU-resident deployment plausible, but leaves a finite budget for runtime overhead and the KV cache. Long contexts and concurrent requests can consume that headroom quickly. Activating only 2.5B parameters per token reduces computation; it does not remove the need to store the full model. Actual 5090 throughput still needs measurement.
Flint is worth watching because responsiveness, privacy, and accessible hardware can matter as much as a top-line benchmark score. Qwen 4 release rumors remain unconfirmed. Qwen3.8 is not unbeatable, but its continued competitiveness makes the next generation an important watch item.
Artificial Analysis measured Beam at 48.5% on AutomationBench-AA, but 3.0% on Terminal-Bench 4.0. Muse Spark Open scored about 8% on Terminal-Bench 4.0. These are demanding evaluations, not deployment success rates. Still, the gap is a reminder that strong demonstrations do not yet guarantee dependable autonomous execution.
Security adds another constraint. OpenCode disclosed a breach affecting 1.4% of Console users, where third-party AI provider credentials had been stored and may have been exposed. API keys, hosted consoles, integrations, and permissions all become part of an agent product's security boundary.
WebBrain, a browser-agent project, reported a separate investigation that found malicious JavaScript disguised as a font file in two development checkouts, paired with a VS Code task configured to run when a project folder opened. The payload attempted to collect credentials and establish persistence. The investigation did not establish successful execution, credential theft, or the attacker's identity, and found no established connection to the OpenCode breach.
Repositories, editor settings, credentials, and integrations deserve the same scrutiny as the model itself. A capable agent cannot compensate for exposed credentials elsewhere in its stack.
One week cannot settle the global AI race. But this one offers a useful snapshot: Chinese labs are shipping capable open-weight models and pushing down the cost of agent work; US teams have credible releases and promising work like Flint, but have not established a lead across the board. The next contest is about model quality, deployment efficiency, developer adoption, and trust.
Artificial Analysis: Reflection Beam and the US open-weight comparison
Artificial Analysis: Muse Spark Open vs. Qwen3.8 Flash-Next
Artificial Analysis: DeepSeek V4.1 Flash 1017
DeepSeek DeepInfer inference framework
MiniMax M3.1 Flash announcement
Recursive and the Eureka Machine
Richard Socher on Recursive's research system
