Agent Skills for Slack MCP: Do They Improve Accuracy?

If you are running Slack through an agent on Opus, here is the uncomfortable number. Sonnet 4.6 at high effort, with a 55-line skill file, passed 56 of 63 tasks. That is 88.9%. Opus 4.8 at high effort with no skill also passed 56 of 63. Sonnet's run cost $4.28. Opus's cost $6.81. The median task was 12% faster on Sonnet too.

So the obvious move is to give Opus the same file and watch it pull ahead. We did. Opus with the skill passed 55 of 63. Fable 5, which started above 90%, finished at 85.7%. On one prompt Opus went from 3 of 3 without the skill to 0 of 3 with it, and the failure is not subtle: it listed channel members, got user ids back, and stopped, because the file said “make one call, then stop.” Sonnet read the same line and went on to look up the names anyway.

That is the shape of the whole result. A skill is worth exactly the gap it fills. On a smaller model the gap is wide and a short index closes most of it, for less money than the bigger model costs with nothing. On a model near the ceiling the gap is narrow, and a rule written for the confused case gets followed most faithfully by the model that least needed it.

The fallacy: more guidance is always safe, and the best model benefits most

Once a team sees a skill lift accuracy on one MCP server, two beliefs tend to follow. First, that guidance has no downside, so write the rules broadly and let every model read them. Second, that the strongest model benefits most, because it follows instructions best. Slack breaks both.

The first belief fails because a skill only helps where the model was missing a distinction. Most of Slack's tools are easy to tell apart. Send, draft, and schedule are distinct in the tool names, and a strong model got those right nearly every time with no help. Where the tool list is already doing the work, a skill cannot add accuracy. It can only add a rule the model did not need, and a rule can be followed too well.

The second belief fails for the same reason read the other way. The model that follows instructions best is the model most likely to obey “make one call, then stop” at the exact moment it was about to do something useful. Opus had already looked up member names on its own. Fable had already passed the DM filter on its own. The skill did not teach either of them anything. It handed them a short general line, and they followed it more literally than they would have on their own.

The money part sits inside that. The question is not whether a skill is worth its tokens. The skilled agent used slightly fewer input tokens and made about a third fewer calls, so the file paid for itself on cost alone. The question is which model you need in the first place. If a 55-line index lets Sonnet match Opus, the skill did not just save tokens on the run. It changed which model the workload needs.

TL;DR: the example and the learning

The example. “Who is a member of mcp-gateway?” The member list returns user ids. A useful answer looks those ids up and gives names. Opus 4.8 at high effort, no skill: passed 3 of 3, listing members and then reading profiles to answer with names. Same model, same prompt, with the skill: passed 0 of 3. One attempt searched for the channel, listed members twice, and stopped. The answer was user ids with no names. Sonnet 4.6 with the same skill passed 3 of 3 and finished the name lookups.

The numbers. 21 Slack tasks, six Claude settings, three runs each. Across all models, accuracy rose from 83.3% to 86.5%. At high effort, Sonnet with the skill went from 66.7% to 88.9%, the same score as Opus with no skill, for $4.28 against $6.81. Opus with the skill was 87.3%. At medium effort, Sonnet rose from 74.6% to 84.1% and Opus stayed at 87.3%. Fable went from above 90% to 85.7% at both settings, a drop of 3 or 4 runs out of 63, so the size is uncertain but the specific failures are real. Tool calls per task fell from 2.52 to 1.71. Full benchmark cost fell from $46.10 to $44.01.

The learning. Match the skill to the model, then prune it. A skill helps where the model was missing a distinction: a capability the server lacks, a search scope, a write the user already spelled out. Where the tool names already make the choice clear, a strong model gets there alone, and a broad rule can cut off work it would have finished. Write each rule narrowly against a specific confusion, and test any new rule on your strongest model before trusting it.

How we benchmarked agent skills on Slack MCP

Same design as the Linear test. One variable changed.

The tasks. 21 Slack prompts written around search scope, threads, drafts and schedules, canvases, lists, uploads, missing capabilities, and consent. “What did we decide?” might be a search across the workspace, the last few messages in one channel, or the replies under a single message. “Send this” might mean now, a draft someone still has to approve, or a message that should go out tomorrow. “Write this up” might be a canvas, a list, or a file. We picked the tasks because they are confusing, so they are harder than ordinary Slack traffic.

The data. Slack responses came from a frozen recording. Both arms saw identical data, and writes never reached a real workspace.

The models. Sonnet 4.6, Opus 4.8, and Fable 5, each at medium and high effort. Six settings, three runs per task, in a fresh workspace each time. 378 runs with the skill and 378 without.

The two arms.

The skill. A 55-line SKILL.md index the agent always reads. It lists the objects, a short way to choose between them, a table of lookalikes, and what the server cannot do. Three reference files, one each for finding, writing, and artifacts, hold the argument details. On a typical task the agent decides where the content lives (a topic with no single home is search, recent messages in one conversation are channel history, replies under one message are a thread), resolves channel names to ids, opens that family's reference file if needed, and makes one call, then stops.

The scoring. A task passed only if the agent chose the right tools in the right order, respected the safety rules, and gave a correct final answer. A separate Opus 4.8 judge graded the answer without knowing whether a skill was present. Three runs per task is a small sample, two of the pass checks are model-judged, and replay removes real API latency.

The tasks, skills, scoring rules, and reproduction steps are in the public mcp-skillbench repository.

Results: agent skills vs. the Slack MCP tool list

Overall accuracy moved a little, 83.3% to 86.5%. The interesting movement is per model.

Per model, per effort.

The average is the wrong number to remember. Sonnet at high effort went from 42 of 63 to 56 of 63, the score Opus posted with no skill, at $4.28 against $6.81 and a median task of 32.6 seconds against 36.9. Opus still won the slow tail. Fable's drop is 3 or 4 runs out of 63 per setting, which near the ceiling can be partly noise. The specific failures below are solid. The exact size of the drop is less certain.

Where the skill paid for itself

Without a skill the agent was already reliable on the obvious pairs. Send versus draft versus schedule, adding a reaction, and listing your own channels were right nearly every time either way. A few tasks were different, and those are where the smaller model gained.

The index rows behind those are short:

Where the skill overrode good judgment

One prompt, four runs. “Who is a member of mcp-gateway?” The member list returns user ids. A useful answer looks those ids up and gives names. High effort throughout. The only change in each row is the skill.

The smaller model still finished the name lookups. The larger model followed “make one call, then stop” and cut the task off.

Two other prompts show the same override, both on Fable at high effort, both from 3 of 3 to 1 of 3. “With my permission, search private channels and DMs for __mcp_skill_eval_no_match__.” Private-inclusive search was the right tool either way. Without a skill, the agent passed a filter naming private channels, group DMs, and DMs. The index says that search “includes private channels and DMs,” and some skilled runs took the sentence literally, left the filter out, then said DMs had been searched. And “Read the whole discussion under [a link to one message].” The first page only holds some of the replies, and agents often presented that page as the full discussion. That was hard either way, and the skill made it worse.

In each case the agent had the right tool, or was one lookup away from a finished answer. It then followed a short general line in the index more literally than it would have on its own.

What it cost

The index is small and the reference files load only when needed, so the skilled agent used slightly fewer input tokens than the unskilled one. Tool calls fell by about a third, from 2.52 per task to 1.71, against a shortest correct path of 1.24. Most of that saving was Sonnet, whose unskilled runs tended to search, read, and search again. The median got a bit faster, 37.6 seconds to 35.1, and the slowest 5% dropped from about 75 seconds to about 65. The full benchmark cost $44.01 with the skill and $46.10 without.

Match the skill to the model, then prune it

Skip a skill for routine reads on a strong model. If the agent runs on Opus or Fable and mostly searches public channels and reads history, the tool list is already doing the work. A skill will save some calls. It may not raise accuracy, and on a model near the ceiling it can cost a few trials.

Add one if a smaller model is on the path, or if the tasks involve search scope, a missing capability, or a write the user has already spelled out. Sonnet at high effort, from 66.7% to 88.9%, is why we wrote one. At that point the skill is not a prompt-engineering detail. It is the difference between needing Opus and not.

Then prune it. Keep the index to the choice, load the details on demand, and read the tasks that got worse. If a rule cuts off work the model was already doing, such as looking up names after listing members, narrow the rule or delete it. A shorter skill is easier to follow. That includes the times it should not have been followed.

Test every new rule on your strongest model before trusting it. The smaller model is where you see the gain. The strongest model is where you see the override.

Managing Slack MCP skills with TrueFoundry

A skill changes which Slack tool gets called. It does not decide who may search private channels, which token the call uses, or what gets logged.

The TrueFoundry Skills Registry keeps a skill like this versioned and shared. A skill lives in a repository alongside prompts and models. The repository decides who can discover it, edit it, and attach it, and keeps the version history and audit log. Each publish is a numbered version, so when you narrow a rule that caused a regression, every agent picks up the same fix, and an agent can be pinned back if a new publish misbehaves. The same version can be attached to a TrueFoundry agent or pulled into Claude Code and Cursor. This Slack skill is an index plus three reference files, so it is published as a bundle through the CLI or through GitOps with tfy apply.

The TrueFoundry MCP Gateway sits in front of the Slack MCP server. It handles authentication, tool-level access control, and a trace of every call. Overhead is about 3 to 4 ms, and it handles 350+ requests per second on one vCPU. This benchmark used a local replay, so it says nothing about the gateway. The skill picks the call. The gateway decides whether that call is allowed.