Testing agent skills

Agent skills usually ship as a folder: the instructions the model follows (SKILL.md) next to a tests.json of acceptance cases:

skills/german-b2b-outreach-email/
├── SKILL.md        # the prompt/instructions the agent loads
└── tests.json      # acceptance cases for the skill

{
  "cases": [
    {
      "input": "Draft a first-contact email to a procurement director at a mid-sized German pump manufacturer ...",
      "expected": {
        "contains": ["Betreff:", "Mit freundlichen Grüßen", "P.S."]
      }
    }
  ]
}

contains checks are a good start, but they only catch the failures you predicted. Edit SKILL.md, switch the underlying model, and the output can keep every required substring while the tone, structure, or substance quietly changes. That's exactly the gap BehaviorCI closes.

from_skill_tests()

from_skill_tests() maps each case onto an ordinary @behavior test:

# test_outreach_skill.py
from behaviorci.skills import from_skill_tests
from myagent import run_with_skill

def run_skill(prompt: str) -> str:
    # However you invoke your agent with the skill loaded:
    # a CLI call, an SDK call, an HTTP request, ...
    return run_with_skill("skills/german-b2b-outreach-email", prompt)

globals().update(
    from_skill_tests("skills/german-b2b-outreach-email", run=run_skill, threshold=0.80)
)

The globals().update(...) line injects one generated test per case into the module, so pytest collects them like hand-written tests:

test_outreach_skill.py::test_german_b2b_outreach_email_case_0 PASSED
test_outreach_skill.py::test_german_b2b_outreach_email_case_1 PASSED
test_outreach_skill.py::test_german_b2b_outreach_email_case_2 PASSED

Each case maps like this:

tests.json	BehaviorCI
cases[n].input	The prompt passed to your run callable
cases[n].expected.contains	must_contain — Layer 0 lexical guardrail
The returned output text	Snapshotted — Layer 1 semantic similarity
Folder name + case index	behavior_id, e.g. german-b2b-outreach-email/case-0

The usual loop applies unchanged:

$ pytest --behaviorci-record   # record baselines per case
$ pytest --behaviorci          # fail CI on skill regressions
$ pytest --behaviorci-update   # accept an intentional SKILL.md change

What a failure looks like

Suppose an edit to SKILL.md makes the email pushy while all required substrings survive. The contains check passes — the snapshot doesn't:

FAILED test_outreach_skill.py::test_german_b2b_outreach_email_case_2
BehaviorCI: Similarity 0.6421 < threshold 0.8000

--- STORED OUTPUT (Primary Sample) ---
Betreff: DSGVO-konforme Account-Priorisierung — kurzer Austausch?
... would a 20-minute call next week work for you? ...

--- CURRENT OUTPUT (Primary Sample) ---
Betreff: Letzte Chance — Angebot läuft ab!
... buy now, slots are almost gone ...

Options

from_skill_tests(
    skill_dir,            # folder containing tests.json (str or Path)
    run,                  # Callable[[str], str] — your skill runner
    threshold=0.85,       # similarity threshold applied to every case
    id_prefix=None,       # override the folder name in behavior ids
)

• Behavior ids are <skill-name>/case-<n>, so several skills coexist in one suite without colliding. Use id_prefix if two skill folders share a name.
• A case without an expected.contains list still works — it's snapshot-only.
• A malformed tests.json (missing file, invalid JSON, empty cases, a case without a string input) raises a ConfigurationError at collection time with the exact case index.

Keeping the runner honest

The generated tests are only as meaningful as the run callable. Two rules of thumb:

• Pin what you can. Fix the model version and temperature in the runner so the snapshot tracks skill changes, not sampling noise. For skills that stay noisy anyway, lower the threshold — or write the noisiest case by hand with samples=N.
• Return only the artifact. If your agent wraps the output in logs or markdown fences, strip them in the runner so the baseline is the email (or JSON, or SQL) itself.

A complete runnable example — skill folder, canned runner, record/check flow — lives in the repository under tests/examples/.