5 Lessons on Running Agentic AI in Production - From the Fireside chat

Most enterprises now have agents in production before they have a way to see what those agents are doing. That gap is where the cost overruns, the untraceable failures, and the shadow AI show up. We heard the same story from 200 enterprise AI leaders across 18 industries, and again in a live session with Daniel, VP of Engineering at Via, and Meenal, who leads data and enterprise AI at SurveyMonkey.

The through-line in almost every answer was the same idea, even when nobody used the word: you need an AI gateway. Not another observability tool bolted on after the fact, but a single control layer that models, MCP servers, and agents route through, so cost, logging, and access are handled in one place instead of in every codebase. Here is what the leaders we talked to learned getting there.

Watch the detailed fireside chat here - 

Why agentic AI governance matters at scale

The most revealing finding in our report was a paradox. When we asked leaders where their 2026 and 2027 budgets were going, expanding AI use cases ranked first. When we asked separately what they saw as the biggest risk to fund against, the security and governance of that same expansion ranked just as high. The two line items sit almost on top of each other.

That tension is the whole game. Press too hard on expansion and cost and risk get away from you. Move too cautiously and you fall behind. Every leader we spoke with is trying to fund both at once, and the ones doing it well treat governance as the thing that lets them expand faster, not the thing that slows them down. As Meenal put it, every dollar you spend on the right risk and security posture becomes a lever to put more money into expansion at a faster pace.

Lesson 1: The inference bill is the smallest part of the cost

There was a lot of noise about Uber reportedly burning through its AI budget early, and it is easy to read that as an inference-price problem. It usually is not. The bill runs up because of the context you feed the model, not the per-token rate.

Meenal framed it as homework the organization has to do before it scales anything. Context comes from three places: the data in your databases, the decision logic your teams apply, and the institutional and tribal knowledge that lives in people's heads. Assemble that well and the agent gets what it needs in the fewest tokens. Skip it and you pay for the same model to guess repeatedly at the same problem.

She also drew an analogy that stuck with us. This is the on-prem to cloud story again. Companies moved to the cloud, watched costs balloon anyway, and only then learned to manage them. AI is the same curve, compressed. Costs will keep climbing as tools figure out how to price AI and as the underlying compute gets more expensive, so the discipline has to come earlier this time.

Working across enterprises, our co-founder Anurag sees cost control land in three layers, and an AI gateway is where all three get enforced:

• Visibility. If you allow models across Bedrock, OpenAI, and Anthropic, you need one view of where spend goes, by user, team, and application. You cannot manage what you cannot see.
• Control. Once you have visibility, you set limits: budgets and rate limits at the user, team, or application level, matched to the ROI of what each app is worth.
• Routing. The smartest layer. A request sent to a large frontier model can often get the same result from a smaller one. The central layer sees all the traffic, so it can route each request to the right model, and over time the routing logic gets better because it learns from the observability logs.

Here is the difference between running agents without that layer and running them behind one.

Lesson 2: You cannot trace what you did not design to trace

More than half of the enterprises we surveyed could not fully trace the agents they had running in production. Some of those agents were approved. Some were shadow AI, deployed without IT or leadership knowing.

Daniel had the sharpest way of putting the stakes. Imagine handing an employee a laptop with permission to install anything, full network access, and no monitoring. No modern company does that. Agents need the same guardrails, and they need them from the start, because tracing a non-deterministic system after the fact is close to impossible. The agent does not do the same thing twice, so if the audit trail was not designed in, it is not there when you go looking.

At Via, they treat that as table stakes. Good infrastructure to run agents does two things at once: it makes deployment safer, and it makes deployment easy, which means people deploy a lot more of them. That is why agent observability has to be part of the platform, not a project you get to later. One practical technique that worked for them was splitting an agent's prompt into checkpointed pieces, so they could inspect each response in the chain and catch where something went wrong.

Lesson 3: 76% lack unified logging, and it is a governance problem

Seventy-six percent of organizations in our report lacked fully unified logging across all their models and workflows. That is a cost problem and a control problem at once. If you cannot see which model is being used, by whom, in which department, you cannot reason about the cost structure and you cannot control any of it.

SurveyMonkey put LLM observability in first, before scaling, precisely because these systems spiral without it. Their logging now extends past cost into quality: they watch for hallucination, and they use evals and LLM-as-judge to check outputs. Daniel raised a wrinkle that catches a lot of teams off guard. Agent output can contain user PII even when your normal logs never do, so you have to design what gets logged and what gets scrubbed rather than logging everything by default.

Both agreed on the uncomfortable part: observability rarely feels like the first thing to build. The pull is to ship something and show value. That is reasonable, but governance added later is far more expensive than governance designed in, and an AI gateway is where that logging becomes automatic instead of something every developer has to remember.

Lesson 4: Internal versus external decides your risk posture

Where you land on the expansion-versus-risk question depends mostly on one thing: whether the AI faces inward or outward. That was Anurag's read across the enterprises we work with, and both practitioners confirmed it.

Internal productivity tools can move fast, because the cost of something going slightly wrong is low. Via built internal infrastructure so their own people, including non-technical staff, can spin up a dashboard, a small site, or an automation from a ready-made template. Anything customer-facing is a different story. Meenal was direct: external AI faces harder scrutiny because customers want to know how their data is being used, so it goes through a tougher review cycle than anything internal. The companies balancing both well are the ones that invest in the risk posture first and then use that headroom to expand.

Lesson 5: Start with 8 to 10 high-ROI use cases, not 200

The strongest advice from the session was about restraint. Do not boil the ocean. Meenal's rule was to lead with the use case, not the technology, because starting from the problem is what points you at the right tools and the real ROI. Daniel's was to choose the goal wisely: too small and it is meaningless, too far out and it is not reasonable, then run it as an agile POC, build the guardrails, and ship when it holds up.

Anurag has watched this change firsthand. A few years ago, teams walked in with lists of 200 use cases they wanted to solve with AI, and most of them would have cost as much to build as they saved. The organizations at the top of their game today bring 8 to 10, each with solid ROI, and they put the cost controls, governance, and observability in from day zero. That last part matters: teams that skip it ship to production, hit one failure, and have to unwind and rebuild the whole thing. Put the control layer in early and the good use cases get room to grow.

How TrueFoundry runs agentic AI governance through one control plane

We build TrueFoundry as an enterprise AI control plane for agentic AI. The idea maps directly onto the six lessons above: bring models, MCPs, and agents together in a single layer, then put cost, governance, and observability on top of all of it.

The AI gateway is the entry point. It gives you one OpenAI-compatible API to 1,000+ LLMs across providers, so switching models is a name change in the request rather than a rewrite. It adds roughly 3 to 4 ms of overhead and handles 350+ RPS on a single vCPU, which keeps it in the hot path without becoming the bottleneck. Because every request flows through it, spend tracking, budgets, rate limits, and model routing are enforced centrally rather than reimplemented in each service.

The MCP gateway and agent registry handle the N-by-M problem from Lesson 3. MCP servers and agents register in one place with tool-level access control, so you can grant a specific user a specific tool without exposing the rest, and every call is authenticated and tracked. Guardrails apply as policy at this layer, which is the pattern SurveyMonkey described with pre-vetted agent templates: the data, context, and observability controls are baked into the template so teams inherit them by default.

For observability, the gateway is OpenTelemetry-compliant and traces every request from prompt to tool and model execution, then plugs into Grafana, Datadog, or Prometheus. If a CTO shows up with 30 agents, three model providers, and no control layer, the first move is not a rewrite. It is registering everything that those agents consume in one place and pointing their endpoints at the central layer, so unified logging and cost control come without developers touching their code.

Related reading

• What is an LLM Gateway? the architectural primer behind this post
• Best MCP Gateways how to evaluate the layer that governs your MCP servers
• LLM Cost Tracking Solution going deeper on spend visibility
• MCP vs RAG how the two ways of giving agents context compare

Conclusion

Six lessons, one throughline. Cost, tracing, MCP access, logging, risk posture, and use-case selection all come back to whether you have a single place to see and control what your agents do. An AI gateway is that place, and putting it in early is what lets teams expand agentic AI in production instead of unwinding it after the first failure.

See how TrueFoundry's AI gateway handles cost, MCP governance, and observability in one control plane, or book a demo to walk through it with your own stack.