Quality scoring

Publishing a skill and scanning it for security problems tells you it’s safe to install. It doesn’t tell you whether the skill actually helps. Quality scoring is skael’s answer to that: a real evaluation, run against a panel of models, that measures a skill’s effect on task outcomes.

What a score is

A panel of models attempts a set of real tasks twice — once with the skill available, once without. The gap between those two runs is the skill’s contribution: how much better (or worse) an agent does with it.

A separate model, acting as judge, compares the two transcripts and scores the outcome. Alongside that, a contract checker watches whether the skill did anything it explicitly promised not to — for example, a skill whose spec says “never touch the network” gets flagged if a transcript shows it doing exactly that.

All of that rolls up into a single headline score from 0 to 100.

What you need to run one

Two things, run separately:

1. The server. It queues evaluation jobs but does not run them — no Docker socket and no LLM key live there.
2. A skael-worker process, with a Docker daemon available. The worker claims jobs from the server, runs the evaluation in a sandboxed container, and posts the score back.

Without a worker running, jobs just sit in the queue — nothing gets scored.

One credential, two jobs

The judge and the claude-code panel agent both need Anthropic credentials, and in the simplest setup one variable covers both.

• The judge. A separate model compares the two transcripts and scores the result. It authenticates with ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN, which the worker checks at startup and exits naming if neither is set. It talks to the direct Anthropic API by default, and to any Anthropic-compatible gateway you point it at (see Choosing a model and a gateway below). It never falls back to a subscription CLI on PATH.
• The panel agent. The claude-code adapter is the only one wired up (codex, cursor, and opencode are registered but their parsers aren’t implemented yet, so they can’t run and have no credentials to configure). It reads credentials as environment variables in the worker’s own process and forwards whichever are set into the sandbox:
  • ANTHROPIC_API_KEY — the same variable the judge uses. API-billed, no login step. This is what the Claude Code CLI uses by default in non-interactive mode, and it’s the recommended setup for a server or VPS.
  • CLAUDE_CODE_OAUTH_TOKEN — subscription-billed (Pro/Max) instead of pay-per-call. Generate it once on any machine where you can log in interactively: claude setup-token. Set the resulting token as this variable on the worker.

Set one of these and both jobs are covered. Neither is checked for the panel at startup, since the sandbox may already have credentials baked into its image — but if neither is set and no auth directory is available either, the worker logs a warning naming the adapter and the variables that would fix it, instead of the job silently coming back with an incomplete panel (see “Reading a score” below).

For local development on a machine that already has an interactive Claude Code login, the adapter also falls back to mounting ~/.claude and ~/.config/claude from the host into the sandbox, read-only. That’s a convenience, not something to rely on for a server: on macOS the actual credential lives in the Keychain, not in ~/.claude, so copying that directory to a Linux worker was never going to authenticate anything. Set an environment variable instead.

Worker environment variables

Required — the worker exits at startup naming whichever is missing:

Variable	Description
SKAEL_ENDPOINT	Base URL of the skael server the worker claims jobs from
SKAEL_API_KEY	API key the worker authenticates with
ANTHROPIC_API_KEY	Direct Anthropic API key for the judge model — never a subscription CLI on PATH

Optional, with defaults:

Variable	Default	Description
CLAUDE_CODE_OAUTH_TOKEN	—	Subscription auth for the claude-code panel agent, as an alternative to ANTHROPIC_API_KEY. Generate with claude setup-token. Beside ANTHROPIC_BASE_URL it also splits the judge from the panel — see Keeping the panel on a subscription
WORKER_ID	{hostname}-{pid}	Identifies this worker in job leases
WORKER_LEASE	5m	How long a claimed job’s lease lasts before it’s considered abandoned
WORKER_POLL	15s	Interval between claim attempts when the queue is empty
WORKER_WORK_ROOT	OS temp dir	Directory to materialise eval workspaces under
WORKER_CONCURRENCY	1	Concurrent sandbox sessions. Must be a positive integer
WORKER_GRADE_CONCURRENCY	whetstone’s default (8)	Concurrent judge calls. A container is bounded by CPU and memory, a judge call by the account’s rate limit, so the two are separate knobs

The worker also needs a Docker daemon it can reach — every evaluation runs inside a sandboxed container, one job at a time per worker process. Run more worker replicas for more throughput.

Choosing a model and a gateway

Four variables configure every model call. skael-worker and whetstone read the same four, and the judge that scores a run and the panel agents that attempt the tasks use one gateway. Set none of these and everything talks to Anthropic’s own API.

Variable	Default	Meaning
ANTHROPIC_API_KEY	—	Credential, sent as x-api-key.
ANTHROPIC_AUTH_TOKEN	—	Credential, sent as Authorization: Bearer — what OpenRouter issues. Wins when both are set.
ANTHROPIC_BASE_URL	https://api.anthropic.com	Gateway base URL. It posts to {base}/v1/messages, so the base carries no /v1.
LLM_MODEL	shipped defaults	Comma-separated model ids, most capable first.

The auth header is inferred from the credential you set, so there is nothing to keep in sync. Run whetstone doctor to see what resolved.

A complete OpenRouter setup, covering both the judge and the panel:

ANTHROPIC_BASE_URL=https://openrouter.ai/api
ANTHROPIC_AUTH_TOKEN=<your OpenRouter key>
LLM_MODEL=anthropic/claude-sonnet-5,anthropic/claude-haiku-4.5

LLM_MODEL is not optional there. OpenRouter model identifiers are namespaced (anthropic/claude-sonnet-5), unlike Anthropic’s bare names (claude-opus-5), and the panel would otherwise keep asking your gateway for Claude Code’s bare alias sonnet. A gateway that namespaces its identifiers answers those with a 404; every panel member then fails its health probe and the run refuses with the model names in the error. Check the ids against your gateway’s own catalogue before you set them: a retired id fails the same way a mistyped one does.

Keeping the panel on a subscription

Set CLAUDE_CODE_OAUTH_TOKEN alongside ANTHROPIC_BASE_URL and the two separate. The judge keeps the gateway, because a published score must come from a metered, reproducible backend. The panel authenticates with the subscription token instead, and the worker withholds the gateway variables from the sandbox so it cannot follow the judge onto the gateway.

ANTHROPIC_BASE_URL=https://openrouter.ai/api
ANTHROPIC_AUTH_TOKEN=<your OpenRouter key>
LLM_MODEL=anthropic/claude-sonnet-5
CLAUDE_CODE_OAUTH_TOKEN=<output of: claude setup-token>

LLM_MODEL then names the judge alone. The panel asks for the shipped alias, which is what a subscription serves.

This is a local and small-team setup rather than a shared-instance one. The panel it produces is recorded in model_panel like any other, so turning it on splits a skill’s score trend at the changeover, and a subscription-backed panel is neither metered nor pinned to a model version. whetstone doctor and the worker’s startup log both name the split when it is active.

The list is ordered. The first entry judges every run and is the panel’s primary member. Later entries are the panel’s floor members, which only the deep tier runs — one list rather than a slot per tier, because a half-configured pair produced a panel with one working member and one that 404s, which is not an error but a complete run: it scores, reports panel_complete: false, and so can never release a version held for review, after paying for a full tier to get there.

Naming a model without a gateway moves the judge alone. The panel keeps running the shipped alias against Anthropic directly, which is what you want when only the judge is BYOK — and it keeps your existing scores comparable, since a changed panel splits the trend line.

You are not limited to Claude models. OpenRouter’s Anthropic-compatible endpoint accepts the same request shape whatever model you route to, so google/gemini-2.5-flash-lite and the rest of its catalogue work for the judge too. Reasoning models are the one thing to watch: if a model spends its whole output budget on thinking tokens, the reply carries no text block and the run fails rather than scoring something empty. Give those a larger budget or pick a non-reasoning model for the judge.

Changing the judge model changes what the score means. Two scores judged by different models are not comparable and are not charted on the same trend line — the platform records which model judged each run and splits the trend when it differs, with the reason shown, the same way it already splits on a changed suite or panel (see “Comparing versions over time” below).

The judge is also a calibrated instrument: its agreement with human labels (κ) was measured for a specific judge model. Swap the judge and that calibration no longer describes the judge actually in use.

A skill needs a suite first

Before a skill can be scored, it needs a registered evaluation suite — the set of tasks the panel will attempt. Generate one with whetstone, then register it:

whetstone suite gen <skill>
whetstone suite push <skill>

A skill with no suite gets no score, permanently — there’s nothing to run. This is deliberate: scoring against a suite the skill’s author never signed off on would measure the wrong thing.

How long it takes

Roughly 45 to 90 minutes per evaluation. It’s running real tasks against real models, twice, plus a judge pass — not a quick lint.

Reading a score

A score isn’t a single flat state. Each of these means something different, and the UI shows them differently on purpose:

• Unscored. The skill has never been evaluated. This is not the same as scoring zero — a zero says “measured, and it did badly”; unscored says “nobody has measured it yet.” Shown as a dash, not a number.
• Attested vs. verified. A verified score came from the queue — the worker ran it and posted the report back through the platform. An attested score is a claim without that chain of custody. Only a verified score can release a version the publish gate is holding for review.
• Incomplete panel. One or more models in the panel failed their health check partway through, so the evaluation couldn’t be finished properly. This is flagged separately from a low score — a panel that couldn’t finish tells you nothing about whether the skill is good.
• Stale. The score is for an older version than the one currently being served. The skill has moved on since it was last measured.

Comparing versions over time

GET /api/skills/{name}/quality/series (and the trend chart on the skill detail page) show how a skill’s score has changed across versions.

The one thing to understand here: two scores are only comparable if they came from the same tasks and the same models. Change the suite, or change which models are in the panel, and the number can move even though the skill itself didn’t change. Charting that as one continuous line would be misleading — a jump could look like the skill got better or worse when really the yardstick changed.

So the trend line only plots scores that are genuinely comparable to each other. Everything else is listed separately, with the reason it’s not part of the trend (different suite, different panel). A trend line that quietly mixed incomparable scores together would be worse than no trend line at all.

In the example above, four versions were scored the same way, so they form one line. A fifth score exists but was run against a different set of tasks, so it sits under the chart with the reason rather than being plotted.

How this feeds back into publishing

A version held for review by the publish gate clears automatically once it has a verified score at or above QUALITY_FLOOR (an operator-configured minimum, default 0 — any verified score with a complete panel and no contract violations clears it). Short of that, it takes an instance admin — an account whose role is owner or admin — running skael review <name> <version> --approve --reason-kind scan --reason "...".

A score clears one hold reason and only one

A version can be held for two independent reasons: scan (a blocking security finding) and ownership (published to a name by someone who is not a skill owner of it). They are a set, not a state, and each has to be cleared on its own terms.

A quality score clears scan. It can never clear ownership. No score, however high, releases a version the skill owners have not agreed to. If a score could clear an ownership hold, the whole review path would be decorative — anyone could publish into someone else’s namespace and let a passing eval wave it through.

The reverse holds too: a skill owner approving the ownership reason does not clear a scan finding. Only an instance admin does that. If a namespace owner could, the security gate would only be as strong as the least careful self-managed namespace on the instance.

See Scanning for the rest of what the gate does, and skael review / the review queue API for acting on held versions.