DSPy Through a Gateway: Cost, Rate Limits and Fallback

What DSPy is

DSPy is a framework from Stanford for building LM systems by programming them rather than prompting them. Instead of hand-tuning prompt strings, you declare the shape of a task — a signature like question -> answer — and compose those into modules (ChainOfThought, ReAct, custom dspy.Module classes). Then DSPy’s optimizers tune the actual prompts and few-shot demonstrations against a metric you define, treating the prompt as a learnable parameter.

This is an effective design, and one which has a cost profile that teams often underestimate. Since optimization entails calling your program to execute the process iteratively on a dataset (creation, evaluation, selection), DSPy is capable of converting small chunks of code into thousands of LM calls. This framework does tasks that you would be performing on your own manually, however, through burning tokens.

This is precisely what an ideal workload for a gateway would be like.

How DSPy calls a model

DSPy routes every LM request through LiteLLM, which means it speaks the OpenAI-compatible protocol. To send that traffic through the TrueFoundry AI Gateway, you name the model with the openai/ prefix and point api_base at your gateway URL.

import dspy

lm = dspy.LM(
    model="openai/anthropic-account/claude-4",  # openai/ prefix + TrueFoundry model name
    api_key="your-truefoundry-api-key",
    api_base="{GATEWAY_BASE_URL}",
)
dspy.configure(lm=lm)

That is the entire integration. Every signature, module, and optimizer you write from here calls the gateway underneath — no other code changes. You can also set it once via environment variables:

export OPENAI_API_KEY="your-truefoundry-api-key"
export OPENAI_BASE_URL="{GATEWAY_BASE_URL}"

Your DSPy code talks to dspy.LM; dspy.LM talks to the gateway; the gateway talks to whichever provider you named. Swapping openai/anthropic-account/claude-4 for an OpenAI or Gemini model is a string change, because the gateway normalizes them all.

Why optimization makes a gateway worth it

Here is the number that matters. A single ChainOfThought prediction is one or two LM calls. An optimizer is a different animal.

When you call optimizer.compile(), DSPy runs your program over the training set many times — bootstrapping demonstrations, evaluating candidates, searching for the best configuration. A BootstrapFewShot run can be hundreds of calls; MIPROv2 can be thousands. Run that a few times while you iterate on your metric and you have generated serious, bursty, hard-to-predict spend — the kind that blows through a provider rate limit at the worst moment and shows up on the invoice with no breakdown of which experiment caused it.

Calling the model directly, you have no guardrails on any of this. Through the gateway, you do:

• Cost tracking per optimizer run, model, and team — so you know a MIPROv2 sweep cost $14 before you run forty of them.
• Rate limiting that stops a compile() from exhausting your provider quota and taking down everything else sharing the key.
• Caching of repeated calls during optimization, which happen more than you’d think when the same demonstrations are re-scored.
• Fallback so a provider hiccup mid-compile fails over instead of crashing a two-hour optimization run.
• One-string model switching, so re-optimizing on a cheaper model to compare quality-per-dollar is a config change.

Where teams get this wrong

They optimize against a raw provider key. The first big compile() either trips a rate limit or produces a surprise bill with no attribution. Route it through a gateway before you scale optimization.

They forget caching during iteration. Re-running optimization while tuning a metric repeats many identical calls. Without a cache you pay for every one.

They hardcode the provider. DSPy’s whole value is comparing configurations — but if the model is baked into every module, comparing providers means code edits. The openai/ prefix plus a gateway makes it a string.

They don’t separate dev from prod spend. Optimization runs are experiments; inference is production. Without per-run cost attribution they blur into one line nobody can explain.

Want to see what a DSPy optimization run actually costs?

Book a demo

How to use DSPy with TrueFoundry

Two steps. Install DSPy (pip install -U dspy), then grab your gateway base URL and model name from the unified code snippet in the TrueFoundry playground.

Configure the LM as shown above, then write ordinary DSPy. A signature and module:

class GenerateAnswer(dspy.Signature):
    """Answer questions with short factoid answers."""
    question = dspy.InputField()
    answer = dspy.OutputField(desc="often between 1 and 5 words")

generate_answer = dspy.ChainOfThought(GenerateAnswer)
pred = generate_answer(question="What is the capital of Brazil?")
print(pred.answer)
And an optimizer run — the part that benefits most from the gateway:
from dspy.evaluate import Evaluate

def validate_answer(example, pred, trace=None):
    return example.answer.lower() in pred.answer.lower()

optimizer = dspy.BootstrapFewShot(metric=validate_answer)
optimized = optimizer.compile(qa_module, trainset=trainset)

Every call that compile() makes is now metered, rate-limited, cached, and fallback-protected by the gateway — with zero changes to the DSPy code itself.

A worked example: budgeting an optimization sweep

A team is tuning a RAG module. Each MIPROv2 run makes roughly 1,500 LM calls, averaging 3,000 input and 400 output tokens per call. They plan 20 runs while iterating on the metric.

On a mid-tier model (say Claude Sonnet 5.5 at $2 / $10 per million), one run is about (1,500 × 3,000 × $2 + 1,500 × 400 × $10) / 1e6 ≈ $15 in input-plus-output — call it $15 per run, or $300 across 20 runs. Route the same optimization at a cheaper model for the bootstrapping phase and the number drops sharply — but you only know that, and can prove the quality trade-off, because the gateway tracked cost per run and per model.

Without that attribution, the $300 is a single unexplained line. With it, optimization becomes a budgeted, measurable part of development — and the decision to re-optimize on a cheaper model is backed by numbers, not vibes.

Gotchas worth knowing

• Use the openai/ prefix. DSPy goes through LiteLLM; the prefix tells it to use the OpenAI-compatible route to your gateway. Omit it and routing fails.
• Watch trailing slashes on api_base. A trailing slash can produce a double slash in the request path. Test with a simple call first.
• api_base vs base_url. Most versions use api_base; confirm which your installed DSPy/LiteLLM accepts.
• LiteLLM logs can get noisy when pointed at a custom endpoint. If your terminal fills with provider messages, adjust LiteLLM’s logging.
• Cache during iteration, not just production. The biggest savings in DSPy often come from caching repeated optimization calls.

Related reading

• Multi-model routing on the AI Gateway
• LLM cost tracking: a practical solution
• Cost-and-quality-aware LLM routing
• Rate limiting AI agents: preventing LLM API exhaustion
• What is LLM fallback?
• What is an LLM router?

Conclusion

DSPy changes how you build with LLMs — prompts become parameters, and a compiler does the tuning. The hidden cost of that elegance is traffic: optimizers make far more LM calls than the code suggests, and that spend is bursty, unpredictable, and easy to lose track of.

Pointing DSPy at the TrueFoundry AI Gateway is a two-line change that turns all of it into a governed, measured path — cost per optimization run, rate limits that protect your quota, caching that cuts repeated calls, and fallback that keeps a long compile alive. You keep DSPy’s programming model and gain the operational control production needs.

Put your DSPy optimization runs under one roof

Book a demo