LLM model comparison: MiniMax M3 matched Claude Opus 4.8 on every task. It cost 16x less.

Vendor benchmarks measure what a model can do at its best: full agent harnesses, Docker containers, curated test sets. That's not what most engineers actually need to know. The real question is whether the quality gap justifies the price gap on a production API call.

We ran MiniMax M3, Gemini 3.5 Flash, and Claude Opus 4.8 through TrueFoundry AI Gateway on a 130-line Python payment service with three code-review tasks of increasing difficulty. Total cost for all nine calls: $0.067. What we found was not what we expected.

The setup

The codebase had deliberately planted issues: a missing import time that would crash the retry logic on first rate-limit, a hardcoded production API key, a mutable default argument on the charge function, no idempotency key on retries, and no error handling on two of four public functions.

Three tasks, ordered by difficulty:

• Easy: summarize what the service does and how it is structured
• Medium: find every bug, security issue, and reliability problem
• Hard: full architectural critique, including the three most serious structural problems, a redesigned retry strategy, missing observability, and a refactoring plan

All nine calls went through the same TrueFoundry AI Gateway endpoint. Same base URL, same credentials, different model string. We scored each response manually on a 1-5 scale after reading it in full, with 5 meaning it caught non-obvious issues and gave actionable fixes specific to this codebase. No LLM judge: an LLM-as-judge on a code-review task biases toward whichever model's output style resembles the judge's training data.

The results

Cost and quality

Response time

Gemini is the fastest by a meaningful margin. MiniMax and Opus are comparable on latency.

Model-by-model

MiniMax M3

Easy: 5/5 · Medium: 5/5 · Hard: 5/5

What stood out was the depth of reasoning, not just the number of issues found. On the medium task it didn't just flag individual bugs — it traced how two separate problems compounded each other, turning what looked like a crash into a silent failure that returns the wrong result with no indication something went wrong. On hard it produced a concrete redesign plan, not a list of problems. On easy it spotted a production bug while answering a question about code structure.

Claude Opus 4.8

Easy: 5/5 · Medium: 5/5 · Hard: 5/5

Matched MiniMax across all three tasks. It caught one issue MiniMax missed: a security vulnerability caused by unsanitized user input being passed directly into a URL. Different catch patterns, equivalent quality overall.

Gemini 3.5 Flash

Easy: 3/5 · Medium: 1/5 · Hard: 1/5

The easy response was correct but generic. The medium and hard responses were severely incomplete: medium opened mid-sentence, hard ended mid-phrase with no completed thought. Whether this was streaming truncation or a short model response, your application receives a response with no error but missing most of its content. TrueFoundry Gateway logs output token counts on every call, which is how you catch this before it reaches users.

The routing decision

MiniMax M3 matched Opus at 16x lower cost ($0.00390 vs $0.06422 across nine calls). At 10,000 queries per day on an 800-token context, routing standard tasks to MiniMax instead of Opus saves roughly $4,500/month.

For tasks where a single missed issue has asymmetric consequences (security review, payment logic, compliance), use Opus. Opus's path traversal catch is the concrete example from this run: MiniMax didn't find it, Opus did. On a live payment service, that's not a minor gap.

For Gemini, use it where response completeness is verifiable: classification, intent detection, short-document summarization. Don't use it unsupervised on tasks requiring exhaustive output until the incomplete response behavior is better understood.

Setting this up through TrueFoundry AI Gateway is one configuration change: a virtual model with priority-based rules that classifies the task, routes to the appropriate tier, and falls back automatically if a model is unavailable. The gateway handles LLM routing across 1,000+ models through a single OpenAI-compatible API, adds about 3-4 ms of overhead at 350+ RPS on a single vCPU, and logs cost and output token counts per request.

What we take from it

MiniMax M3 is not a budget model with a benchmark story. On a real codebase it produced output indistinguishable from Opus 4.8 at 16x lower cost. The Gemini finding is equally important: a model that produces incomplete responses silently is more dangerous in production than one that is simply slower, because you can't see the failure in your application logs.

The practical question is not which model wins a benchmark. It's which routing policy extracts the most value from the cost-quality tradeoff across your actual workload. Run the benchmark on your own codebase and build the policy from what you find.

Explore LLM routing on TrueFoundry AI Gateway

Related reading

• Open-Weight Routing at Scale: GLM-5.1 vs Claude Opus 4.7 on TrueFoundry AI Gateway
• Grok 4.3 on Amazon Bedrock: We Routed Four Frontier Models Through One Gateway
• Intelligent LLM Routing: Cost & Quality-Aware Selection
• Multi-Model Routing: Optimize AI Tasks Efficiently
• What is an LLM Gateway?

FAQ

What is an LLM model comparison and how should teams run one?

‍An LLM model comparison evaluates multiple models on the same task under the same conditions: same prompt, same endpoint, same evaluation rubric. The most useful comparisons use tasks from your actual workload. Route all models through a shared gateway to ensure identical infrastructure conditions across every call.

How does MiniMax M3 compare to Claude Opus 4.8 on code review?

‍On the three tasks we ran, MiniMax M3 scored 5/5 on all three and cost 16x less than Opus 4.8 per query. Opus caught one additional issue, a URL path traversal vulnerability, that MiniMax did not. Both models identified the idempotency bug, missing import, PII logging issue, and mutable default argument.

What is LLM routing and when does it matter?

‍LLM routing directs requests to different models based on task type, complexity, cost, or latency requirements rather than sending everything to one model. Routing easy tasks to cheaper models while reserving frontier models for high-stakes work can reduce LLM spend by 60-80% with no quality loss on the low-complexity tier. TrueFoundry AI Gateway supports LLM routing natively: you configure a virtual model with priority-based rules, and the gateway handles model selection, fallbacks, and cost tracking automatically.

What latency does TrueFoundry AI Gateway add?

‍About 3-4 ms of overhead. The gateway handles 350+ RPS on a single vCPU and is designed to sit in the hot path without becoming the bottleneck.

Can I deploy TrueFoundry in my own VPC?

‍Yes. TrueFoundry runs in your VPC, on-prem, air-gapped, hybrid, or across multiple clouds. No data leaves your domain, which is the primary reason regulated enterprises choose it over SaaS-only gateways.

‍