TrueForge: An Open-Source Alternative to Claude Managed Agents, Up to 75% Cheaper
An open-source, vendor-neutral agent harness benchmarked head-to-head against Claude Managed Agents. Same accuracy, up to 75% lower cost per run.
TrueFoundry has open-sourced TrueForge, a vendor-neutral agent harness you run on your own infrastructure, with any model. We put it up against the two obvious alternatives on the same 14 enterprise tasks, with the same tools and one blind judge: the closed Claude Managed Agents and the open-source deepagents. Here are the results:
Prototyping an agent is easy. Running one in production is where the bill shows up. The tool-calling loop, context management, sub-agents, sandboxed execution, approvals, retries: all of it is tokens, and tokens are money. How many you burn to get a single answer comes down to two choices: which model you run, and which harness you run it on.
Most managed platforms take the first choice away from you. Claude Managed Agents is the clearest example: a genuinely good experience, tied to one model family, so you pay frontier prices even for tasks a cheaper model could handle. At the other end, open-source harnesses like deepagents (LangChain's LangGraph agent) hand you the model choice but leave the efficiency up to you.
TrueForge gives you that managed-agent experience on any model, in your own infrastructure. We ran it head-to-head against both to see what vendor-neutrality and a leaner loop are actually worth
Everything below is reproducible: the same 14 tasks, the same MCP tool servers for every harness, a fresh session per task, and one blind LLM judge that never learns which harness or model wrote an answer.
The benchmark
We used DevRev's Enterprise-Bench: 14 cross-system tasks at varied difficulty that read like real B2B ops work. To answer any of them, an agent has to plan, call MCP tools across three systems (a Salesforce-style CRM, a Jira-style project tracker, and a Drive-style store of documents and call transcripts), join the results correctly, and pitch the answer at the right level of detail.
The setup was identical for every harness. The same 14 tasks, the same three MCP servers, and a fresh session per task.
Grading is blind. Each answer is scored by an LLM judge against the task's required criteria. It sees the answer and the rubric, but never which model or harness produced it. A task counts as solved only if it meets every criterion; there is no partial credit.
How we counted cost. Cost is each run's actual token usage priced at list rates ($5 / $25 per 1M in/out for Opus 4.8; $0.73 / $2.28 for GLM-5.2), and it's cache-aware. Latency is wall-clock per run with every harness hitting the same cloud-hosted MCP servers.
An open-source model, same quality, 75% cheaper
The biggest lever a vendor-neutral harness gives you is the one a single-vendor platform can't touch: send the task to a cheaper model. So we ran the benchmark through TrueForge on GLM-5.2, an open-source model. Here are the results:
| Configuration | Solved | $ / run | Tokens | $ / correct | vs CMA |
|---|---|---|---|---|---|
| Claude Managed Agents (Opus 4.8) | ~11 / 14 | $11.8 | 10 M | ~$1.10 | baseline |
| TrueForge (GLM-5.2) | ~11 / 14 | $2.9 | 3.7 M | ~$0.25 | −75% |
On TrueForge, GLM-5.2 solved the same tasks Opus did on Claude Managed Agents. Same quality, about $3 a run against $12. That's roughly 75% cheaper. Claude Managed Agents can't run this setup at all: it's Anthropic-only, so the cheaper-model lever was never on the table.
Same model, three harnesses
Model choice isn't the only place cost hides. Hold the model fixed (Opus 4.8 for all three harnesses) and the harness alone still moves the bill a lot.
Cost per run (same model, same tasks)
All three on Opus 4.8. Accuracy is a three-way tie (everyone within a task of each other); the cost is not.
| Harness | Solved | $ / run | Tokens / run | Latency | $ / correct |
|---|---|---|---|---|---|
| TrueForge | ~11 / 14 | $8.5 | 3.8 M | 40 min | ~$0.80 |
| Claude Managed Agents | ~11 / 14 | $11.8 | 10 M | 63 min | ~$1.10 |
| deepagents (LangGraph) | ~10 / 14 | $21 | 16.5 M | 64 min | ~$2.10 |
Accuracy is a three-way tie. Everyone lands within a task of each other, because the model sets the ceiling on what's possible. What the harness sets is the price of getting there, and the spread is wide: TrueForge reaches the same answers on about 40% of Claude Managed Agents' tokens, and under a quarter of deepagents'. Against the closed platform that's about 30% cheaper per run; against the open-source harness, about 2.5× cheaper.
Why a harness changes the bill?
Every turn re-processes the system prompt and tool definitions. TrueForge drives from a compact instruction; the others carry heavier scaffolding. deepagents alone adds planning, a virtual filesystem, and sub-agent machinery. Across hundreds of turns, that fixed overhead is real money.
Most of the cost is the loop, not the answer, and every tool round-trip ships the growing context back to the model. TrueForge planned the fewest calls of the three to reach the same result.
TrueForge trims history and large tool responses rather than re-sending everything verbatim. deepagents re-reads its own accumulated context step after step, which is how it lands at three to four times the tokens for the same task.
Because the harness is vendor-neutral, a task that doesn’t need a frontier model isn’t billed like one. That’s the lever behind the cheaper-model section, and routing is a setting rather than a re-platforming project.
What this means
Two levers, and they stack. Vendor-neutrality lets you send a task to a cheaper model when it doesn't need a frontier one. That's the bigger win, and the one a closed platform can't offer at all. A leaner loop then takes cost out of whatever model you do run, with fewer tool calls and fewer tokens.
The model is still the model. What TrueForge changes is everything around it, and on an agent that runs thousands of times a day, that is most ofthe bill.
Get started with TrueForge
It's open source and runs on your own infrastructure, with any model through one OpenAI-compatible API. Bring your own MCP servers and run the same agent loop we benchmarked here.
Prefer it managed? Run it on TrueFoundry AI Gateway, managed MCPs and observability included. See the platform →