Fine-Tuning vs Prompting: When to Specialize an SLM

Why fine-tuning vs prompting confuses product teams

Four things people collapse into one question

Teams often ask "SLM or LLM?" as if that answers the product. It doesn't. Four different decisions get mashed together:

You can prompt a private LLM. You can fine-tune and still call a public API for the open-ended parts. Hosting and specialization are orthogonal. Mixing them up is how regulated teams stall for a quarter on GPUs when the real blocker was a missing doc index.

For how teams keep model choice and spend under control once traffic is live, see what an LLM gateway is (https://www.truefoundry.com/blog/llm-gateway). The gateway question comes after you know which track you are on.

The three tracks that actually help

Once you separate those four decisions, the roadmap collapses into three tracks:

1. Learn: hosted model, eval set, logging. Prove the workflow before you own a model.
2. Ground: retrieval, tools, schemas. Give the model evidence instead of changing weights.
3. Specialize: fine-tune a small model on a narrow slice after gates flip. Not the whole assistant.

Most "we need an SLM" pitches are really Learn or Ground problems wearing Specialize language.

We built a small fine-tuning vs prompting resolver (https://benchmark-accelerators.ml.tfy-eo.truefoundry.cloud/fine-tuning-vs-slm) around that framing. Answer a handful of product questions and it returns a diagnosis, locked vs current tracks, and a copy-paste memo. Same rubric every time, so PM, eng, and security argue from one sheet.

What to evaluate before you fine-tune

Developer enthusiasm matters. Vendor demos matter less. Neither replaces gates.

1. Name the job, not the model

Is this ticket triage, field extraction, docs Q&A, open chat, or a multi-step agent?

Closed or schema-bound work can specialize later. Open-ended assistants and coding agents almost never should be fine-tuned first. If you can peel off a narrow sub-task (classify, then draft), specialize only that slice.

2. Decide the data path before model size

Internal / low risk can start on a public API with normal vendor review. Customer PII needs a privacy review. Regulated data (PHI, finance residency, air-gap) forces VPC or on-prem.

That decision is hosting. It does not unlock fine-tuning by itself. For spend and latency once volume grows, LLM cost optimization (https://www.truefoundry.com/blog/llm-cost-optimization) is the parallel conversation, not a substitute for labels.

3. Count gold labels, not ticket dumps

Historical tickets and chat logs are not a training set until someone marks the correct output. Dumping a CSV into LoRA mostly memorizes noise.

If someone says "we have some examples," ask whether those are gold or raw. Raw counts as none.

4. Assign a model owner for 6+ months

Fine-tuning without an owner is a frozen prompt you cannot safely update when data or the base model shifts. Evals, drift, and redeploys need a name on the hook. "Maybe later" is Learn, not Specialize.

5. Be honest about next-quarter scale

Pilots rarely pay back GPU plus MLOps. Hundreds of requests per day may still favor APIs. High steady volume is when unit cost and latency start to justify a specialist, and only after the other gates pass.

6. Align eng and security before the training kickoff

Ask eng for the eval set, logging plan, and whether a prompted baseline already exists. Ask security about vendor allowlists, prompt logging, and what happens if a future feature starts sending PII. Skip this and the fine-tune becomes a compliance surprise after the first demo lands.

Six situations where teams pick the wrong fork

Docs Q&A with no trusted index

Someone wants the model to "know our wiki." That is retrieval. Fine-tuning will not invent missing pages or stay fresh when docs change. If nobody owns a corpus, you do not have a RAG problem yet. You have a corpus problem. Ground first.

Ticket triage with lots of labels, an owner, and real volume

This is one of the few places Specialize earns a seat. Keep a prompted baseline on the same eval. Run one narrow LoRA experiment. Only then argue managed private endpoint vs self-host. Classification and extraction are the common wins, not "replace the whole assistant."

Open-ended support chat

Teams try to fine-tune personality into a small model and wonder why quality collapses on edge cases. Keep the open-ended agent on a general model. If there is a closed step before drafting (intent, route, extract), peel that off and specialize only the peel.

"Some labels" that are really raw tickets

The spreadsheet has thousands of rows. Nobody marked correct. Treat it as no labels until review exists. Specialize stays locked for a reason.

Regulated data, so "we must fine-tune"

Privacy can force VPC or on-prem hosting. It does not force weight updates. Many regulated teams should prompt a private LLM (Learn) or retrieve over a private index (Ground) long before they train.

High API spend without an eval harness

Spend pressure is real. Training without a held-out scoreboard is how you ship a specialist that loses to the prompted baseline you never measured. Fix the harness first. For routing routine work to cheaper models while you Learn, an LLM router (https://www.truefoundry.com/blog/what-is-llm-router) is often the cheaper lever than a fine-tune.

A simple way to brief the room

When the debate starts, force three sentences:

1. Diagnosis: what problem this actually is (retrieval, narrow classification, open agent, corpus gap).
2. Not this: the wrong fork ("we are not choosing an SLM vs GPT; we are choosing how answers get grounded").
3. Current track: Learn, Ground, or Specialize, plus why Specialize is locked or unlocked.

Then list the hard gates with severity. No labels is a hard stop. No owner is a soft but real stop. "Some labels" is a sequence issue: enough to measure, not enough to train. Data path is a hosting gate, not a specialize unlock.

We use that structure in the strategy resolver (https://benchmark-accelerators.ml.tfy-eo.truefoundry.cloud/fine-tuning-vs-slm). The output is a short memo PMs can paste into a PRD or Slack thread, with rubric version on top so arguments stay pinned to the same rules.

How teams run Learn, Ground, and Specialize in production

Choosing the track is half the job. Once traffic is live, the same issues show up for any LLM workload.

Model selection stays a settings preference. Fine-tunes ship without a baseline. Spend and quality stay invisible until someone complains.

Teams that stay ahead usually put a control layer in front of model traffic. An AI gateway sits between apps and providers. It sets defaults, gates who can call frontier models, attributes cost by team, and keeps inference in the right network boundary. For the mechanics, see LLM gateway (https://www.truefoundry.com/blog/llm-gateway) and multi-model routing (https://www.truefoundry.com/blog/multi-model-routing).

TrueFoundry's AI Gateway (https://www.truefoundry.com/ai-gateway) is one option built for that job: one OpenAI-compatible API across 1,000+ models, with quotas, RBAC, cost tracking, and routing policies. You can run it in your VPC, on-prem, air-gapped, or hybrid, so the data path decision from earlier stays enforceable. When Specialize finally unlocks, TrueFoundry's training and fine-tuning (https://www.truefoundry.com/training-fine-tuning) path covers the narrow LoRA work without bolting on a separate MLOps stack first.

If model governance, private hosting, or a clean path from prompted baseline to specialist are still open after you pick a track, it's worth a look.

Takeaways

Fine-tuning vs prompting is a budgeting and sequencing decision, not a brand preference. Most teams should Learn or Ground this quarter. Specialize is earned: narrow job, gold labels, named owner, honest volume, and a prompted baseline you can beat.

Map the job before you name a model. Separate hosting from fine-tuning. Count reviewed examples, not ticket dumps. Give finance and security a track and a gate list, not a vibes vote.

We learned most of this after watching "let's fine-tune" projects absorb a quarter and still fail an eval the prompted model already passed. The model catalog will keep growing. The gates will not get simpler. Getting the strategy side right early saves a lot of awkward roadmap resets later.

Related reading

• What Is an LLM Gateway and How Does It Work?
• LLM Cost Optimization: Why an AI Gateway Is the Missing Layer
• What is LLM Router?
• Multi-Model Routing: Optimize AI Tasks Efficiently
• Training and fine-tuning on TrueFoundry