DeepSeek V4 Flash tops leaderboards but completes only 54% of real agent tasks

Full text

DeepSeek's V4 Flash has topped model leaderboards and been hailed by developers as a "total monster" since its rollout. But in real-world testing, it completed just 53.8% of a batch of complex agent tasks.

Composio ran the model through eight different agent harnesses , including Claude Code, Codex, and OpenCode, on 30 deliberately difficult, multi-step tasks spanning live tools like Gmail, GitHub, Slack, and Google Sheets. Of 240 total runs, 129 passed — and only six of the 30 workflows were completed successfully by every harness tested.

The gap illustrates why orchestration, not raw model capability, may decide whether the model succeeds in enterprise settings: the same model produced substantially different results depending on the harness, tool configuration, caching behavior, retries, and provider stack it ran on.

DeepSeek said it will be hiking the prices for V4 Flash and Pro , models that have quickly become favorites among developers building coding assistants and agents.

While it seems the move might undercut its very appeal — strikingly capable models at ultra-low pricing that frontier providers simply can't match — it also moves the story beyond the now-clichéd "cheap Chinese model" narrative, as early use cases emerge and enterprises figure out where different models fit into their tech stacks and what workflows they should be aimed at.

" Insane" adoption numbers as DeepSeek flips the cost structure

DeepSeek rolled out V4 Flash to public beta July 31, and made V4 Pro generally available on August 13. The 284-billion-parameter Flash is built for volume and speed, the 1.6 trillion-parameter Pro for more complex workflows.

Both models have flexible reasoning capabilities (low, high, max) and ‘thinking modes’ applying chain-of-thought (CoT) reasoning to improve answer accuracy.

Users were immediately impressed by Flash’s capabilities. It has dominated OpenRouter's usage leaderboard since its rollout, currently the most-used model on the platform by weekly token volume.

“The adoption numbers of the initial DeepSeek V4 Flash were insane,” ML researcher Nathan Lambert posted to X , adding that the new version "scored the same as GLM 5.2," making it a "total monster" that will be used extensively.

DeepSeek switching the cost model adds an interesting dimension.

V4 API rates are going up by as much as 1,100% depending on the model, token type and time of use. The new pricing structure:

Flash will be 22 cents per million input tokens and 66 cents per million output tokens off-peak; and 44 cents per million input tokens and $1.32 per million output tokens at peak. This represents a 57% to 371% increase.

Pro will be 66 cents per million input tokens and $1.98 per million output tokens off-peak; and $1.32 per million input tokens and $3.96 per million output tokens at peak. This shows a 51% to 355% jump.

Cache hits, meanwhile (when models reuse prompts rather than starting from scratch), are going up between 52% and 1,100%.

DeepSeek says offering 50% lower off-peak usage is intended to encourage “more flexible workload scheduling." Seventeen of every 24 hours stay at half price, and the new structure actually prices the company's home market the highest.

"This is not a simple price rise," said Sanchit vir Gogia of Greyhound Research . "It is a pricing architecture that makes the timing of inference an economic variable."

Work that can wait — such as batch evaluation, synthetic-data generation, and overnight development runs — moves into the cheap hours; interactive agents and live operations cannot. Gogia said irritation among developers and enterprises is genuine and vocal, and that DeepSeek's past low pricing doesn't obligate it to stay cheap forever.

At first glance, it does look like a “suicidal move from a platform still looking for credibility against more established AI model vendors,” said tech analyst Carmi Levy . The increases will certainly eat into DeepSeek's price advantage and force customers to weigh concerns around the company's Chinese origins more heavily.

Still, DeepSeek remains far cheaper by all pricing measures relative to competing models from OpenAI, Anthropic, Google, Cohere, xAI, and others, he said.

So, while the move will force DeepSeek to emphasize performance and security over cost, it hardly wipes out its already-notable price-performance advantage, and still gives customers ample wiggle room to justify its use for specific workloads, Levy said. The math will just have to be more tightly calculated.

“The advantage will likely erode over time as DeepSeek inevitably continues to align pricing with market realities, but for now it's still easy to make the business case,” Levy said.

Where can DeepSeek Flash fit into enterprise environments?

Adoption inside enterprises remains an open question due to cost, capability, reliability, data governance, security, and other factors.

One use case is batch processing, Levy said. This kind of work is typically routine and repetitive rather than demanding top-tier intelligence, so it makes sense to use a cheaper, more efficient model. “It’s a high-performance inference engine that enterprises can consider using for point solution workloads rather than as a wholesale replacement for the incumbent offerings,” Levy said.

Partial adoption will likely involve isolated, non-sensitive workloads with clearly-defined success metrics, strict oversight, permissions controls, and fallback models in case of failures, he noted. Broader deployment will require DeepSeek and its hosting partners to demonstrate strong reliability, security, privacy, auditability, and deployment options. As it adjusts price structures based on demand, DeepSeek also must retain a large enough price-performance advantage to justify any risk, he said.

Expect unsanctioned, smaller-scale use in backroom labs and contained test environments as IT teams get familiar with the new model and figure out when and how to bring it to senior leadership for budget approval.

“DeepSeek has built a well-earned reputation as a global disruptor,” he said, “and it’s clear that its march to broader enterprise adoption will continue to gather momentum.”

Testing DeepSeek in multi-tool workflows

While many use cases are still in the experimental phases, Meta software engineer Naman Ahuja offers one that could translate directly into enterprise environments. In a project unrelated to his employer, he built a home-automation agent with DeepSeek V4 Flash to explore how a lower-cost model performs as the reasoning/orchestration layer for a real multi-tool workflow.

When he leaves home, an agent coordinates several actions across otherwise separate systems: Such as setting a thermostat to “away” to reduce unnecessary energy use, arming a Ring security system, closing and locking doors.

“What interested me was not simply whether the model could understand a command, but whether it could translate intent into a sequence of actions across multiple tools where reliability matters,” Ahuja said.

The biggest lesson was that once a model can take actions, reliability matters as much as intelligence. The system needs structured tool outputs, verification that actions actually succeeded, retry/failure handling, and clear boundaries around what the model is allowed to do.

In the case of enterprise, “the architecture is similar.” Home devices change to ticketing systems, databases, CRM platforms, or infrastructure APIs. The most useful agents will likely orchestrate repetitive workflows across multiple systems, with scoped permissions, auditability, observability, and human approval for higher-risk actions.

“Many valuable AI agents will not be chatbots; they will be background agents coordinating APIs, infrastructure, and business systems in response to events,” he said.

Enterprises need tangible use cases

But Flash's API is still in public beta, Gogia pointed out, and there is not yet an evidence trail of settled enterprise adoption, real-world deployments, and named customers.

“The benchmark story is looser than its retelling, the portfolio story is newer than it looks, and the economics have moved into the system around the model,” he said. Developer mainstreaming is proven; enterprise standardization is not. “The model is mainstream by traffic and still unproven by contract.”

DeepSeek's own integration guidance is an important consideration, he said: Its documentation for at least one popular agent environment states that built-in V4 entries are not sufficient for reliable operation without compatibility overrides.

“Which is a vendor telling the market, accurately, that benchmark performance is not a proxy for production readiness,” Gogia said. “A model can score beautifully and still misbehave once tools, credentials, and state enter the room.”

The serving layer behaves no differently: the same open weights run by different hosts show visible differences in throughput and uptime. "Choosing Flash therefore answers one procurement question and opens three more: Who serves it, where it runs, and which controls surround it," Gogia said.

Prepare for a multi-model future

DeepSeek offers a nuanced case for a multi-model future. V4 Flash is being deployed as the high-volume worker inside diverse estates, Gogia noted: It handles routine generation, retrieval, and background automation, while more difficult or sensitive tasks go elsewhere.

“The question is whether its performance is sufficient for the real-world workflows enterprises actually run, not whether it tops every benchmark,” he said. Enterprises must determine which combination of model, harness, and provider completes the work safely at the lowest cost.

Adam Dalloul, CEO and founder of EmpirioLabs AI, pointed out that bigger isn’t always better; workflows should be task-dependent. For example, his team at EmpirioLabs AI — which hosts 100-plus models on one API, including DeepSeek V4 Flash — were recently working on translating its site into different languages, and there was no need for a large model like GPT 5.6 Sol or Opus 5 to complete the task.

“This is where subagents come in handy,” he said.

His recommended approach: Spawn cheaper subagents and adapt per task. For instance, use Flash variants for day-to-day work, and Pro variants when you need something more powerful. “It depends on the nature of your application.”

Many companies are pivoting towards their own internal benchmarks to route models effectively, Dalloul noted. For example, his team has a workflow that puts a model through various gates and instructions. This helps them identify the model with the speed and accuracy required for the task.

In another example, one of his enterprise clients exclusively wanted access to DeepSeek V4 Flash. They had tested a variety of models and V4 Flash was the only one that met their criteria for speed, cost, and an “appropriate intelligence threshold.”

Meta’s Ahuja agreed that smaller, more efficient models can handle frequent, well-defined agentic tasks, while more expensive frontier models can be reserved for “ambiguous, difficult, or higher-risk decisions.” The relevant metric increasingly becomes cost per successfully completed workflow rather than simply cost per token.

The trade-off, however, is that cheap inference does not automatically mean cheap or safe automation, he said. Once an AI system can take actions, reliability, verification, permissions, failure handling, and security become much more important.

“A failed text response is inconvenient; a failed action in an operational workflow can have real consequences,” Ahuja said.