Score and improve skills

Before the whole team relies on review-checklist, Dana wants a second opinion on it. SkillCatalog can ask an AI tool to rate a skill against a rubric and then to suggest a better version, which Dana reviews, applies, and publishes to the catalog.

What scoring sends and where

Dana's skill text leaves her machine only when she scores or improves the skill. SkillCatalog does not call an AI service itself. It runs the Claude Code or Codex command-line program that Dana already uses, signed in with her own account, and that program sends the text to its AI service. SkillCatalog calls this program an AI provider. Each run counts toward Dana's usage with that service.

The AI provider receives the skill's SKILL.md and SkillCatalog's rubric, the list of qualities that a score rates. An improvement also sends the latest scores and the skill's check results. SkillCatalog puts none of the skill's other files in the prompt. It runs Claude Code with its tools turned off and Codex in a read-only sandbox, both in an empty temporary folder, so neither program can change your files. The sandbox stops Codex from writing, not from reading, so Codex can still read files that your account can reach.

Scores are saved in SCORE.json, next to SKILL.md in the skill's folder, so that everyone who has the catalog sees them once they sync. Delivery never copies SCORE.json to your tools.

Set up AI providers

Dana already works in Claude Code, so she turns it on as her AI provider. If you use neither program, install one first. The provider ids are claude-code-local and codex-local:

skc scoring providers enable claude-code-local

Enabled scoring provider 'claude-code-local'.

Then she checks that SkillCatalog can use it:

skc scoring providers list

provider_id	enabled	usable	auth_mode	source	readiness	model	effort	display_name
claude-code-local	true	false	local_cli	local_cli	installed_not_authenticated	<unset>	<unset>	Claude Code (local)
codex-local	false	false	local_cli	local_cli	not_checked	<unset>	<unset>	Codex CLI (local)

SkillCatalog checks only the providers you turned on. Dana has not signed in to Claude Code on this machine yet, so its readiness is installed_not_authenticated and usable is false. She signs in with claude auth login (for Codex, codex login) and lists the providers again. A provider is ready when usable is true.

Each provider uses its own default model and effort. To choose others, save them with skc scoring providers set-model and skc scoring providers set-effort.

In the desktop app, open Settings, turn on the provider under AI providers, and choose Check readiness.

Score a skill

With Claude Code ready, Dana scores the skill. She passes --catalog-id, because without it the command uses the catalog last chosen in the desktop app. The command prints the provider's progress, then the skill's runs as JSON on one line, so she pipes it through jq to read it:

skc skill score review-checklist --catalog-id acme-skills | jq .

[claude-code-local] Starting local CLI scoring process.
...
{
  "history": {
    "skill_slug": "review-checklist",
    "runs": [
      {
...
        "providers": [
          {
            "provider_id": "claude-code-local",
...
            "envelope": {
              "schema_version": 1,
              "overall_pass": false,
              "score": 72,
...
              "findings": [
                {
                  "dimension": "examples-and-templates",
                  "severity": "warn",
...
                  "rule": "examples-and-templates.missing-example",
                  "suggestion": "Add one short example."
                }
              ],
...
  "new_run_id": "01M37X9XASJAXD18KTBTBE4BGJ"
}

The new run comes last. For each provider, it holds a score out of 100 and overall_pass, which says whether the skill passes the rubric. Each of its findings names the rule the skill missed and a suggestion for meeting it. Here, Claude Code asks for one short example.

SkillCatalog adds the run to SCORE.json in Dana's clone of the catalog. Like any other change there, it stays on her machine until she syncs, so she syncs to share the score with her teammates:

skc sync --message "Score the review checklist"

On a review-branch catalog, scoring and applying an improvement push proposals instead, and leave your clone unchanged. skc skill improve then has no score to start from until the scoring proposal is merged and synced.

In the desktop app, open the skill's Scoring tab, then choose Score this skill and Start scoring. Once the skill has a score, View rubric on the same tab shows the rubric and the prompt that SkillCatalog sends.

Read score history

Scores change as the skill changes, and Dana wants to see how. SCORE.json keeps up to the last 100 runs, and skc skill score-history prints them as JSON, oldest first:

skc skill score-history review-checklist --catalog-id acme-skills

A run is fresh while SKILL.md stays as it was when the run was made. After Dana edits the skill, the earlier runs show "stale": true and "stale_reason": "content has changed since this run", which tells her the skill needs a new score.

In the desktop app, the skill's Scoring tab lists its runs and gives the reason next to each stale one.

Improve a skill

To act on the findings, Dana asks for an improvement. Each AI provider rewrites SKILL.md from the latest scores, and SkillCatalog scores every rewrite and keeps the best one. An improvement starts from a fresh run, so score the skill again after each edit.

Dana wants to read the rewrite before it replaces her skill, so she saves it in an improvement proposal file: a JSON file that holds the new SKILL.md and its scores. It has nothing to do with the proposals of a review-branch catalog. The path must be new, and outside the catalog's clone:

skc skill improve review-checklist --catalog-id acme-skills --proposal-out ~/review-checklist-improvement.json

The command prints the provider, the old and new scores, and a diff of SKILL.md. Without --proposal-out, it only prints and saves nothing.

To read the whole rewrite, or to send it to a teammate for review, print the new SKILL.md from the file:

jq -r '.proposal.files[0].content' ~/review-checklist-improvement.json

Dana reads the rewrite as she would review code, because a score cannot tell whether Acme's checklist asks for the right checks. The rewrite is right, so she applies the file:

skc skill improve --apply-proposal ~/review-checklist-improvement.json

This writes the new SKILL.md and adds its scores to SCORE.json, without asking the provider again. When you do not need to review the rewrite first, skc skill improve review-checklist --catalog-id acme-skills --apply skips the file and applies the best rewrite at once.

In the desktop app, choose Improve skill on the Scoring tab, then Draft improvement. Review the rewrite, then choose Apply updated skill.

Publish an improvement

Applying the rewrite changes only Dana's clone. To share it, she publishes it with her own commit message:

skc skill publish-accepted --proposal ~/review-checklist-improvement.json --message "Improve review checklist"

Publishing commits only the skill's SKILL.md and SCORE.json, pushes the commit, and delivers the new version to Dana's tools. When other changes wait in her clone, it stops after the push, before delivery, and skc sync finishes the job.

After --apply, which saves no file, publish with skc sync instead. In the desktop app, choose Publish accepted changes on the Scoring tab.