Fine-Tuning vs Prompting: When to Specialize an SLM

Why fine-tuning vs prompting confuses product teams

Four things people collapse into one question

Teams often ask "SLM or LLM?" as if that answers the product. It doesn't. Four different decisions get mashed together:

You can prompt a private LLM. You can fine-tune and still call a public API for the open-ended parts. Hosting and specialization are orthogonal. Mixing them up is how regulated teams stall for a quarter on GPUs when the real blocker was a missing doc index.

For how teams keep model choice and spend under control once traffic is live, see what an LLM gateway is (https://www.truefoundry.com/blog/llm-gateway). The gateway question comes after you know which track you are on.

The three tracks that actually help

Once you separate those four decisions, the roadmap collapses into three tracks:

1. Learn: hosted model, eval set, logging. Prove the workflow before you own a model.
2. Ground: retrieval, tools, schemas. Give the model evidence instead of changing weights.
3. Specialize: fine-tune a small model on a narrow slice after gates flip. Not the whole assistant.

Most "we need an SLM" pitches are really Learn or Ground problems wearing Specialize language.

We built a small fine-tuning vs prompting resolver (https://benchmark-accelerators.ml.tfy-eo.truefoundry.cloud/fine-tuning-vs-slm) around that framing. Answer a handful of product questions and it returns a diagnosis, locked vs current tracks, and a copy-paste memo. Same rubric every time, so PM, eng, and security argue from one sheet.

----------------------------------------

What to evaluate before you fine-tune

Gates before Specialize unlocks

Developer enthusiasm matters. Vendor demos matter less. Neither replaces gates.

1. Name the job, not the model

Is this ticket triage, field extraction, docs Q&A, open chat, or a multi-step agent?

Closed or schema-bound work can specialize later. Open-ended assistants and coding agents almost never should be fine-tuned first. If you can peel off a narrow sub-task (classify, then draft), specialize only that slice.

2. Decide the data path before model size

Internal / low risk can start on a public API with normal vendor review. Customer PII needs a privacy review. Regulated data (PHI, finance residency, air-gap) forces VPC or on-prem.

That decision is hosting. It does not unlock fine-tuning by itself. For spend and latency once volume grows, LLM cost optimization (https://www.truefoundry.com/blog/llm-cost-optimization) is the parallel conversation, not a substitute for labels.

3. Count gold labels, not ticket dumps

Historical tickets and chat logs are not a training set until someone marks the correct output. Dumping a CSV into LoRA mostly memorizes noise.

‍