Skip to content
Evaluations

Scorecards, review, and a canary that rolls back on its own

Every agent has a scorecard. Every change to an agent goes through review, then a canary. If the canary fails, the change rolls back automatically.

Become a design partnerSee how it works
Changing an agent

Review first. Then a canary.

A change to a playbook, a tool or a rule never goes straight to every run. It is reviewed, then tried on a canary. A failing canary rolls back on its own.

  1. 01
    Propose

    A person proposes a change to an agent.

  2. 02
    Review

    A person reviews the change before it applies.

  3. 03
    Canary

    The change runs on a canary first, next to the version you trust.

  4. 04
    Promote or roll back

    If the canary holds, the change is promoted. If it fails, it rolls back automatically.

change / support agent playbook (example)
New step: offer a callbackRolled back
Proposed by
Support lead
Reviewed by
Head of support
Canary
Failed
Result
Rolled back automatically. The previous playbook keeps running.
Judging agents

A scorecard for every agent

You judge an agent the way you judge any role: against the measures on its scorecard, with the record of what it did.

01

Scorecards per agent

Ready-made agents come with a scorecard. Agents you build get one too.
02

Replay any run

When a score drops, replay the runs behind it, step by step.
Replay and the log
03

Every change on record

Who proposed a change, who reviewed it, and what the canary did, all in the log.

Frequently Asked Questions

Put an agent on real work, with people in charge

Design partners run one real workflow on OrchKernel and work directly with the engineers building it.

Become a design partnerSee how it works
OrchKernel is source-available and self-hosted.