A production-ready community skill passes 4 tests that most community skills fail: the description field auto-triggers correctly without a slash command, the output contract defines both what it produces and what it doesn't, the skill stays stable on inputs the author didn't test, and evals.json proves the behavior was verified before release. A prompt in a file passes 0 of those 4.
TL;DR: The difference between a production skill and a prompt in a file isn't the word count or the number of steps. It's whether the skill was engineered to work reliably on inputs beyond the happy path. A prompt with no output contract, no description engineering, and no evals breaks when reality shows up.
What's the Actual Difference Between a Prompt File and a Production Skill?
A prompt file tells Claude what to do. A production skill tells Claude what to do, when to do it, what the output must look like, what the output must not look like, and what to do when the input doesn't match expectations.
The distinction is operational, not aesthetic. A well-written paragraph of instructions is better than a badly written 50-step SKILL.md. But a well-written prompt that lacks an output contract, a working description field, and test coverage is still a fair-weather skill: it works when the input is clean and the task is obvious. It produces unpredictable output when edge cases arrive.
Over 700,000 skills are listed on SkillsMP as of 2025. Most of them are prompts in a trenchcoat. At AEM, our Claude Code skill engineering work spans two production-bar attributes that matter most: description-field design for reliable auto-triggering and eval coverage for verified behavior. We've reviewed hundreds for client projects. The 4-test bar check takes 90 seconds and filters out the bottom 70% (AEM bar check data, 2025).
"The single biggest predictor of whether an agent works reliably is whether the instructions are written as a closed spec, not an open suggestion." — Boris Cherny, TypeScript compiler team, Anthropic (2024)
The 4 tests operationalize this. Each one checks whether the skill is a closed spec or an open suggestion.
Does the Description Field Auto-Trigger or Require a Slash Command?
A description field auto-triggers without a slash command if it meets 3 conditions: single line in the YAML, under 1,024 characters, and imperative trigger language. This test eliminates 60-70% of community skills (AEM bar check data, 2025); most fail on the imperative language check.
Passive phrasing such as "This skill helps with code review" activates less reliably than an explicit invoke condition.
Open the SKILL.md and read the description frontmatter field. Check 3 things:
- Single line. The description must be on one line in the YAML. If it wraps to a second line, the YAML parser may truncate it, and Claude receives an incomplete trigger signal.
- Under 1,024 characters. Claude truncates descriptions longer than 1,024 characters. The truncated portion is invisible to the model.
- Imperative trigger language. Descriptions written passively ("This skill helps with code review") activate less reliably than descriptions written imperatively ("Invoke when the user asks for a code review or mentions reviewing a pull request"). In our bar checks, imperative descriptions achieve 100% auto-trigger on matched inputs. Passive descriptions sit at 77% (AEM activation trial data, 2025). An independent study of 650 automated activation trials found directive descriptions were 20x more likely to activate than passive variants (Seleznov, Medium, 2026).
A skill that fails any of these 3 checks is not auto-triggering. It works via /skill-name slash command, which is useful but not the same as a production skill that activates when and only when it should.
The practical test: install the skill, start a fresh Claude Code session, and describe the task naturally without typing the slash command. If the skill activates, the description field works. If Claude answers without using the skill, it doesn't.
For a deeper look at description field mechanics, see How do I write trigger phrases that make my skill activate reliably?.
Does the Skill Define What It Does NOT Produce?
A production-ready output contract defines what a skill produces, what it does not produce, and who consumes the output. Most community skills cover the first part. Far fewer cover the second. The "does not produce" clause prevents scope creep: without it, a code review skill will eventually start rewriting the code it reviews.
A code review skill without a "does not produce" clause will, at some point, start rewriting the code it's reviewing. That's not a bug in Claude's reasoning. It's a gap in the spec. The model saw a code review task and produced the most helpful output it could imagine, which included fixes.
A production-ready output contract has 3 parts:
- Produces: What the skill outputs, in specific format terms. "A 3-section Markdown document: Critical Issues, Warnings, and Style Notes. Each issue includes a code reference and a severity rating (P0, P1, P2)."
- Does not produce: What the skill explicitly declines to generate. "Does not produce: implementation fixes, rewritten code, or architecture recommendations. If issues require architectural changes, the review notes this without providing a redesign."
- Consumed by: Who or what receives the output. "Output is for the developer submitting the pull request, not for automated pipelines."
Skills without all 3 parts have an incomplete output contract. They work consistently on the happy path and scope-creep on everything else.
Does the Skill Handle Inputs It Wasn't Explicitly Tested On?
A production-ready skill handles inputs it wasn't tested on by documenting explicit rules for edge cases: inputs that are too short, in the wrong language, malformed, or outside scope. These rules appear as "if/then" statements in the process steps or a dedicated "failure modes" section. A skill without them breaks on real-world inputs.
A production-ready skill has explicit rules for what happens when the input is:
- Too short to review meaningfully
- In a language the skill wasn't designed for
- Malformed or missing required context
- Edge-case-valid but unusual
These rules appear as explicit "if/then" statements in the SKILL.md process steps or as a dedicated "failure modes" section. A skill that says "If the input file is less than 10 lines, respond with: 'Input too short for a meaningful review. Provide the full file'" is handling edge cases explicitly.
A skill that says nothing about what happens at the edges is a fair-weather skill. It handles the clean, obvious inputs and produces inconsistent output on everything else.
In our commissions, we've seen this fail in 3 of 4 cases where clients installed community skills into production workflows: the skill worked perfectly in testing on clean examples and broke on the first real-world input that was slightly off from the expected format.
Does the Skill Include evals.json?
A production-ready skill includes evals.json containing at least 6 test cases: 2 trigger tests, 2 negative trigger tests, and 2 output quality assertions. This file proves the skill was built and verified before release, not just written and published. Open the skill folder and check. The file either exists or it doesn't.
If yes, the author tested the skill before releasing it and documented what they tested. This is a signal that the skill was built, not just written.
If no, you have no evidence the skill was tested beyond the author's own session history. The skill was probably run a few times on inputs the author controls, declared working, and published.
A minimum evals.json for a production-ready skill contains:
- 2 trigger tests: inputs that should activate the skill
- 2 negative trigger tests: inputs that should NOT activate the skill
- 2 quality tests: assertions about the structure and content of the output
6 test cases total. That's the minimum that gives you confidence the skill behaves as documented.
For a production build, we run 20-30 test cases before declaring a skill ready (AEM production builds, 2025). Community skills with 6+ test cases are in the top 5% of the SkillsMP catalog (SkillsMP catalog analysis, 2025).
For more on how evals work, see What is evaluation-first development?.
What Does a Fair-Weather Skill Look Like Next to a Production Skill?
A fair-weather skill is an open suggestion: a description line that doesn't auto-trigger, no output contract, and no documented edge-case handling. A production skill is a closed spec: it defines what to invoke, what to produce, what to decline, and what to do when input falls outside scope.
Fair-weather version:
---
name: code-reviewer
description: This skill helps review code for quality issues.
---
Review the provided code and identify any issues.
Production version:
---
name: reviewing-typescript-prs
description: >-
Invoke when the user asks for a code review, mentions reviewing a PR,
or pastes TypeScript code and asks for feedback. Do NOT invoke for
general coding questions, architecture decisions, or debugging.
---
## Output Contract
Produces: A structured Markdown review in 3 sections...
Does not produce: Implementation fixes, rewritten code...
## Process Steps
1. Read the provided TypeScript code in full before commenting.
2. Identify critical issues (P0: blocks merge), warnings (P1: should fix),
and style notes (P2: optional improvements).
3. For each issue: cite the exact line, describe the problem, and state severity.
4. If the input is under 10 lines, respond: "Input too short for a structured review."
## Failure Modes
- If no code is provided: ask the user to paste the file or share a code block.
- If the code is not TypeScript: note the language mismatch and ask to confirm scope.
Same task. Completely different reliability profile. The production version is a closed spec. The fair-weather version is an open suggestion.
One limitation worth naming: the 4-test check verifies spec completeness and activation design, not the correctness of the skill's domain logic. A skill can pass all 4 tests and still produce wrong output if the process steps encode bad reasoning. The bar check filters structural failures, not subject-matter errors.
For the related article on where to find and evaluate community skills, see Where can I find community-built Claude Code skills to install?.
FAQ
The 4-test bar check takes 90 seconds and filters out most community skills that fail production requirements. The four tests cover description auto-triggering, output contract completeness, edge-case handling, and eval coverage. Each test is independently verifiable without installing the skill or reading the full SKILL.md body.
How long does the 4-test bar check take?
90 seconds. Open the SKILL.md, read the description field (10 seconds), look for the output contract section (10 seconds), check for evals.json in the folder (5 seconds), and run a fresh session test to check auto-trigger (60 seconds). The fresh session test takes the longest and is non-negotiable.
Can a skill be production-ready without a reference file section?
Yes. Simple skills with fewer than 5 steps and no domain-specific knowledge don't require reference files. A code formatter with a fixed 3-step process and no external domain data doesn't need reference files. A code reviewer with style rules, language-specific conventions, and project context does.
Is a high download count on SkillsMP a quality signal?
Partially. 500+ downloads means the skill is widely used, which means obvious failures would have been flagged. But download count doesn't indicate whether the skill auto-triggers, has an output contract, or handles edge cases. Apply the 4-test check regardless of download count.
Should community skills used in production be pinned to a specific version?
Yes. Install via git submodule and pin to a tagged release rather than tracking main. A skill update that changes the description field changes what auto-triggers the skill, which is a breaking change in production workflows.
What's the difference between a skill being "reliable" and being "production-ready"?
A reliable skill produces consistent output on expected inputs. A production-ready skill produces consistent output on expected inputs AND fails gracefully on unexpected inputs AND triggers correctly without manual invocation AND has documented test coverage. Reliable is necessary. Production-ready requires all 4 tests.
Last updated: 2026-04-25