> ## Content Index
> Fetch the complete content index at: https://www.truefoundry.com/blog/engineering/llms.txt
> Use this file to discover other available public pages before exploring further.

# TrueForge: An Open-Source Alternative to Claude Managed Agents, Up to 75% Cheaper
- URL: https://www.truefoundry.com/blog/engineering/trueforge-vs-claude-managed-agents-benchmark/
- Published: 2026-08-18T11:20:00.000Z
- Updated: 2026-08-19T11:15:21.000Z
- Description: An open-source, vendor-neutral agent harness benchmarked head-to-head against Claude Managed Agents. Same accuracy, up to 75% lower cost per run.
- Author: Shubham Agarwal
- Tags: Benchmarks, AI Agents, Open Source, Agent Harness, LLM Cost

**TrueFoundry** has open-sourced **TrueForge**, a vendor-neutral agent harness you run on your own infrastructure, with any model. We put it up against the two obvious alternatives on the same 14 enterprise tasks, with the same tools and one blind judge: the closed **Claude Managed Agents** and the open-source **deepagents**. Here are the results:

![](https://storage.ghost.io/c/e5/4d/e54d84e5-5dbd-4c68-9d8b-b5ffa5039da0/content/images/2026/08/Info.png)

Prototyping an agent is easy. Running one in production is where the bill shows up. The tool-calling loop, context management, sub-agents, sandboxed execution, approvals, retries: all of it is tokens, and tokens are money. How many you burn to get a single answer comes down to two choices: which model you run, and **which harness you run it on**.

Most managed platforms take the first choice away from you. **Claude Managed Agents** is the clearest example: a genuinely good experience, tied to one model family, so you pay frontier prices even for tasks a cheaper model could handle. At the other end, open-source harnesses like **deepagents** (LangChain's LangGraph agent) hand you the model choice but leave the efficiency up to you.

**TrueForge** gives you that managed-agent experience on any model, in your own infrastructure. We ran it head-to-head against both to see what vendor-neutrality and a leaner loop are actually worth

> Everything below is reproducible: the same 14 tasks, the same MCP tool servers for every harness, a fresh session per task, and one blind LLM judge that never learns which harness or model wrote an answer.

## The benchmark

We used **DevRev's** [**Enterprise-Bench**](https://devrev.ai/enterprise-bench-methodology?ref=truefoundry.com): 14 cross-system tasks at varied difficulty that read like real B2B ops work. To answer any of them, an agent has to plan, call MCP tools across **three systems** (a Salesforce-style CRM, a Jira-style project tracker, and a Drive-style store of documents and call transcripts), join the results correctly, and pitch the answer at the right level of detail.

**The setup was identical for every harness.** The same 14 tasks, the same three MCP servers, and a fresh session per task.

**Grading is blind.** Each answer is scored by an LLM judge against the task's required criteria. It sees the answer and the rubric, but never which model or harness produced it. A task counts as **solved** only if it meets *every* criterion; there is no partial credit.

**How we counted cost.** Cost is each run's actual token usage priced at list rates ($5 / $25 per 1M in/out for Opus 4.8; $0.73 / $2.28 for GLM-5.2), and it's **cache-aware.** Latency is wall-clock per run with every harness hitting the same cloud-hosted MCP servers.

## An open-source model, same quality, 75% cheaper

The biggest lever a vendor-neutral harness gives you is the one a single-vendor platform can't touch: send the task to a cheaper model. So we ran the benchmark through TrueForge on **GLM-5.2, an open-source model**. Here are the results:

__Open model on TrueForge vs Claude Managed Agents on Opus__
| Configuration                    | Solved    | $ / run  | Tokens | $ / correct | vs CMA   |
| -------------------------------- | --------- | -------- | ------ | ----------- | -------- |
| Claude Managed Agents (Opus 4.8) | \~11 / 14 | $11.8    | 10 M   | \~$1.10     | baseline |
| TrueForge (GLM-5.2)              | \~11 / 14 | **$2.9** | 3.7 M  | **\~$0.25** | −75%     |

On TrueForge, GLM-5.2 solved the same tasks Opus did on Claude Managed Agents. Same quality, about **$3 a run against $12**. That's roughly 75% cheaper. Claude Managed Agents can't run this setup at all: it's Anthropic-only, so the cheaper-model lever was never on the table.

## Same model, three harnesses

Model choice isn't the only place cost hides. Hold the model fixed (**Opus 4.8 for all three harnesses)** and the harness alone still moves the bill a lot.

#### Cost per run (same model, same tasks)

deepagentsLangGraph

$21

\~10 / 14

Claude Managed Agentsclosed

$11.8

\~11 / 14

TrueForgeopen

$8.5

\~11 / 14

All three on Opus 4.8\. Accuracy is a three-way tie (everyone within a task of each other); the cost is not.

__Opus 4.8, three harnesses (mean per run)__
| Harness                | Solved    | $ / run  | Tokens / run | Latency    | $ / correct |
| ---------------------- | --------- | -------- | ------------ | ---------- | ----------- |
| TrueForge              | \~11 / 14 | **$8.5** | **3.8 M**    | **40 min** | **\~$0.80** |
| Claude Managed Agents  | \~11 / 14 | $11.8    | 10 M         | 63 min     | \~$1.10     |
| deepagents (LangGraph) | \~10 / 14 | $21      | 16.5 M       | 64 min     | \~$2.10     |

Accuracy is a three-way tie. Everyone lands within a task of each other, because the model sets the ceiling on what's *possible*. What the harness sets is the price of getting there, and the spread is wide: TrueForge reaches the same answers on about **40% of Claude Managed Agents' tokens, and under a quarter of deepagents'**. Against the closed platform that's about 30% cheaper per run; against the open-source harness, about 2.5× cheaper.

## Why a harness changes the bill?

A leaner starting payload

Every turn re-processes the system prompt and tool definitions. TrueForge drives from a compact instruction; the others carry heavier scaffolding. deepagents alone adds planning, a virtual filesystem, and sub-agent machinery. Across hundreds of turns, that fixed overhead is real money.

Fewer, more targeted tool calls

Most of the cost is the loop, not the answer, and every tool round-trip ships the growing context back to the model. TrueForge planned the fewest calls of the three to reach the same result.

It compacts instead of replaying

TrueForge trims history and large tool responses rather than re-sending everything verbatim. deepagents re-reads its own accumulated context step after step, which is how it lands at three to four times the tokens for the same task.

You pick the model per task

Because the harness is vendor-neutral, a task that doesn’t need a frontier model isn’t billed like one. That’s the lever behind the cheaper-model section, and routing is a setting rather than a re-platforming project.

## What this means

Two levers, and they stack**.** **Vendor-neutrality** lets you send a task to a cheaper model when it doesn't need a frontier one. That's the bigger win, and the one a closed platform can't offer at all. A **leaner loop** then takes cost out of whatever model you do run, with fewer tool calls and fewer tokens.

The model is still the model. What TrueForge changes is everything around it, and on an agent that runs thousands of times a day, that is most ofthe bill.

### Get started with TrueForge

It's open source and runs on your own infrastructure, with any model through one OpenAI-compatible API. Bring your own MCP servers and run the same agent loop we benchmarked here.

[Star TrueForge on GitHub](https://github.com/truefoundry/trueforge?ref=truefoundry.com) [Read the docs](https://trueforge.dev/introduction?ref=truefoundry.com) 

Prefer it managed? Run it on TrueFoundry AI Gateway, managed MCPs and observability included. [See the platform →](https://truefoundry.com/?ref=truefoundry.com)