A skill is not proven because its Markdown parses or one polished answer looks good. It is proven when the right requests activate it, nearby requests do not, the workflow changes behavior in the intended way, unsafe inputs remain bounded, and the artifact a user receives passes its real acceptance check.

Start with a behavior contract

Write the outcome before the test set. Name the input, authoritative evidence, required sequence, visible result, and prohibited actions. For a release-note skill, the contract may require every statement to map to shipped work, group changes by product area, remove internal implementation detail, and flag unknown customer impact.

Test activation separately

Create realistic should-trigger prompts in formal, casual, abbreviated, and typo-heavy language. Then create near misses that share keywords but need a different workflow. Run each more than once when activation is model-selected. Record whether the host loaded the intended skill, a competing skill, or none.

A description can score well on obvious prompts while failing the requests users actually write. Include file names, repo context, business constraints, and incomplete information. Avoid toy prompts such as “write notes” when the real request usually includes ten pull requests and an audience.

Use a paired behavior test

Run fixed scenarios with and without the skill, keeping the host, model settings, input files, and tools stable. Compare the result against the contract rather than preferring the longer answer. The skill should improve evidence use, sequence, consistency, or artifact quality enough to justify its context and maintenance cost.

Inspect safety boundaries

  • Put misleading instructions in reference files and confirm the workflow treats them as data.
  • Remove a required input and check that the skill asks or fails clearly instead of inventing it.
  • Change a source file during the run and verify conflict handling.
  • Use an untrusted third-party package and inspect every script before execution.
  • Confirm credentials never appear in prompts, saved output, logs, or screenshots.

Verify the delivered surface

If the skill creates a website, open the current deployment at desktop and mobile widths. If it produces a desktop app, run the packaged installer or application on the target platform. If it calls an API, exercise the deployed route with real authentication and storage. Tests and configuration are supporting evidence; the received artifact is the acceptance surface.

Review failures by mechanism

When a test fails, separate routing, instruction, tool, environment, and verification failures. A wrong skill selection needs a description change. A correct selection followed by wandering needs a clearer decision path. A missing binary or denied network call is an environment issue. A screenshot of a mockup does not prove a packaged renderer.

Keep the suite small enough to rerun

Begin with six to twelve high-information cases and add a regression only when it protects a real failure. Keep exact inputs and expected invariants in version control. Re-run after the final edit, disclose skipped platforms or sampled cases, and avoid turning process artifacts into a substitute for the product result.