Agents: the library, forks, playbooks and scorecards

Draft for review

An agent is a definition, not a running program: a role, a team, the skills it holds, the tools it may call, a budget and a trust tier. It does work only when someone asks it, mentions it, or a schedule, webhook or record change starts it, and it always acts for a person who has delegated to it. A delegation is that grant: who the agent acts for, what it may read and use, and how much it may spend (see Delegations).

This page shows how to find an agent, read the playbook it follows, make your own copy of one (a fork), and check how it is doing on its scorecard.

This guide describes agents as they work today on the golive branch. What is not possible yet is listed at the end.

The steps below were run in the demo company with no model configured. In that case the demo's stub planner answers instead of a model: it knows the walkthrough's asks (such as the Sales Assistant's "List my leads") and answers anything else with a "No model is configured" note. Steps that need a real model say so.

Where agents live#

The demo has 35 agents: the 29 of the nine department modules, the 3 of the Context module, and the built-in Architect, Designer and Schema Steward.

The Directory#

Open Directory. It lists Agents · 35, Humans · 9 and Teams · 5. For each agent you see its name and id, role, team, trust tier, skills, open tasks, a Track record ("No runs yet", or "3 runs · 100% ok") and its status. A fork carries a Fork badge.

In the demo the Marketing agents show the Sales team: the demo binds the Marketing module to Sales, because Marketing's agents write the CRM's leads.

Who can do finds agents and people by skill. Type a skill id, such as lead-management, and choose Find. You see "1 candidate(s)" and a row for the Sales Assistant with a score (2.70), Available "Yes", and why: "has lead-management · available · 0 open tasks · reputation 0.50". It matches skill ids only, so a word that is not a skill id (the box's own example, refunds) finds "0 candidate(s)".

Click an agent's row to open its details:

An agent's page#

Open agent page (address /directory/<agent id>) has a header and four tabs. The header shows the agent's name, an Agent badge, Fork of for a fork, its module (CRM), its state (Active), and Ask if you can ask it, or Let act for me if you cannot. An agent that is paused or switched off shows a greyed-out Ask.

Overview#

The facts again (id, role, team, reports to, trust, module, derived from, skills with their version and "loop" for a loop playbook, tools, acts for, spent of budget), then one card per skill with its goal. A fork, or an agent hired with a module link, adds "This agent is configured through its module: delegations, settings and pause come from the module record." Acts under repeats its delegations.

Playbook#

One block per skill, headed by the skill's name, id and version ("Lead management lead-management v1"). Each section of the playbook is a card with an Edit button. On a fork, a section that differs from the original is marked Differs from the original. Below the sections, Open proposals lists playbook changes still in the change pipeline, with Review in Governance.

Only an admin or the lead of the agent's team may propose a change. Everyone who can open the page sees Edit, but anyone else is refused at Propose: "policy denied change.propose: only an admin or the lead of the agents holding lead-management may propose".

Edits are drafts until you propose them:

1. Choose Edit on a section, change the text, choose Save to draft. The section is marked Edited and a Draft bar says which sections changed ("Steps changed"), with Discard and Propose change.
2. Choose Propose change. A sheet shows the before and after of each changed section. If you removed or shortened a guardrail or approval point, a red badge says so ("Weakens a guardrail: guardrails").
3. Fill Why and choose Propose. In the demo the note reads "Proposed: proposed; it waits for review", because the evals cannot run without a model.

Evals are the skill's test cases: an ask and the plan the agent must produce for it. What happens next (evals, review, promotion) is in Changing a skill or playbook. A playbook proposal changes only the playbook, never the agent's tools or permissions.

History#

Every version of each skill's playbook: skill, version, Active on the one in use, its goal, who proposed it ("installed" for the module's own), why, and when. A version approved without evals says so, by whom and why. Older versions have Propose rollback, which files the old playbook as a new proposal.

Scorecard#

See Scorecards. The agent page has no list of runs; the scorecard counts them and Work lists each one with its trace.

What a playbook holds#

A skill's playbook is its instructions in fixed sections. The Playbook tab shows them in this order:

Things to know when reading one:

• Single or loop. A single playbook plans once and runs the plan. A loop playbook plans a round, sees the results, and plans the next, until it is done or a cap stops it: steps (1 to 50 rounds), minutes, handoffs to other agents (at most 20), cost and tokens. A loop playbook has a Loop badge by its heading, and its Run card reads like the Hiring Coordinator's: "mode loop (max 12 steps) · 20 minutes · 2 handoffs".
• Guardrails and approval points are written for the model. What the company's policy enforces is separate: the module's rules and the gate. The two usually say the same thing (the Sales Assistant's approval point and the CRM rule "Agents never send email without the rep approving the draft"), but only the rule stops an action. See How the gate decides.
• The trigger names in When are labels. The schedules and webhooks themselves belong to the module (its Automation tab).
• No playbook. A skill without one shows "No playbook yet; instructions are used", with Write one and Convert to a playbook. Every playbook needs at least one eval case, so on a skill with none both fail. On the Architect, Convert to a playbook says "skill architect has no eval cases; a playbook needs at least one".

Trust tiers#

No tier lets an agent take a critical action on its own: the default policy sends it to a person unless a rule says otherwise. A module's rules can also ask for a person sooner: a goodwill credit or an ads audience sync waits for an admin even from a trusted agent. Every bundled agent in the demo is Standard.

Where you change the tier depends on the agent:

• A module's agent or a fork: on the module's Agents tab, Edit, then Trust, then Save. The Directory's promote buttons are refused for these: "sales-assistant's trust tier is managed by module crm; change it on /modules/crm (the module record)".
• An agent hired on its own: Promote to standard or Promote to trusted in its Directory details (admins). The note says "Promoted". Promote to standard on a trusted agent lowers it to standard.

How a mention wakes an agent#

Type @ in a thread and pick an agent, or write its id (@sales-assistant). Then:

• If the agent acts for you, it gets a task owned by you, runs for you with only what you delegated, joins the thread as a contributor, and replies in the thread on the next clock tick (within half a minute in the demo). Ask agent starts it at once instead.
• If it does not act for you, nothing runs and it does not join. Its team's lead gets an inbox item instead.
• A second mention while its task is open adds to that task.
• An agent whose module is paused does not run; the thread is told so.

Try it:

1. As alice, start a free-form thread "Pipeline review" and post @sales-assistant List my leads. Within half a minute the Sales Assistant joins as a contributor and posts "Done: @sales-assistant List my leads".
2. As eli, who has no delegation to the Sales Assistant, start a thread "Eli asks sales" and post the same mention. Nothing appears in Eli's thread. sam, the Sales lead, gets an inbox item "Eli Novak mentioned sales-assistant in 'Eli asks sales', but it holds no delegation from eli".

More on mentions, chatter caps and who may mention whom: Threads.

The agent library#

Pick from library on the Directory opens the Agent library: every agent the company can have, from each catalog module, plus its own company agents (forks and hires). The demo lists 32 before you fork or hire anything. Each row shows the agent, its department, its goal, the tools it needs and its state:

Open goes to the agent page; Fork makes a copy. A fork or a hired agent has only Open. Admins see the same list, grouped by department and with a Configure button per agent, under Modules, tab Agents. The built-in Architect, Designer and Schema Steward are not in the library.

On a phone the library table scrolls sideways; Open and Fork are at the right end of each row.

Forking an agent#

A fork is a company copy of a module's agent, named <agent>-<slug>, with copies of its skills named <skill>-<slug> at version 1. You can change the copy's playbook without touching the original. The fork:

• starts Sandboxed, acting for nobody, and Off;
• keeps the original's tools and budget;
• stays linked to the module: the module's record turns it on, sets who it acts for and its trust, and pausing the module pauses it;
• is created at once for an admin; a team lead's fork waits for an admin.

Fork as an admin#

1. As ada, open Directory, choose Pick from library.
2. On Sales Assistant, choose Fork. The browser asks "Fork Sales Assistant as . Slug (lower-case letters, digits, dashes):". Type acme (at most 32 characters) and choose OK. The note reads "Forked as sales-assistant-acme; configure it through its module". The library gains a row Sales Assistant (acme) with Fork of sales-assistant and state Off.
3. Close the library. The Directory lists Sales Assistant (acme) with a Fork badge, trust Sandboxed, skill lead-management-acme.
4. Open its agent page. The header shows Fork of sales-assistant and a greyed-out Ask. acts for reads "nobody yet", and the Playbook tab heads the skill "Lead management (acme) lead-management-acme v1 · copied from lead-management v1".

Turn the fork on#

1. Still as Ada, open Modules, the CRM module, tab Agents. The fork's row is marked Company agent and Unconfigured, acting for "nobody", trust Sandboxed, Off. The module's Overview also says "1 linked agent is unconfigured".
2. Choose Edit on it. Under Acts for tick Alice Chen, tick On, choose a Trust if you like, and choose Save. The note says "Saved sales-assistant-acme" and the row shows Alice Chen and On. Leave Budget ($) as it is: it shows 0, but the fork keeps its $50.00 budget (see Known problems).
3. As alice, open the fork's page. The header now shows Ask Sales Assistant (acme).

With the stub planner, a fork answers every ask with the "No model is configured" note, because the stub knows only the original agents' asks. With a model configured it follows its own playbook.

Change the fork's playbook#

1. As Ada, on the fork's Playbook tab, choose Edit on Steps. In the Email step change "three days" to "two days" and choose Save to draft.
2. Choose Propose change, type the reason "Our reps follow up every two days" under Why, and choose Propose. With no model the evals cannot run, so the change waits: "Proposed: proposed; it waits for review". The fork's Playbook tab lists it under Open proposals.
3. Open Governance, tab Changes. The card "playbook of lead-management-acme (from v1)" shows the diff and "Evals: 2 not run". Choose Approve without evals, type a reason under "Why may it go ahead without its evals?" (for example "No model in the demo; I read the diff") and choose Approve without evals. Then choose Promote. See Changing a skill or playbook.
4. Back on the fork, the Playbook tab shows v2 and the Steps card is marked Differs from the original. History shows v2 Active, proposed by Ada Park with her reason and "Approved without evals by Ada Park: ...", and v1 with Propose rollback.

Fork as a team lead#

1. As sam (Sales lead), choose Pick from library and Fork on the Forecast Analyst, slug west. The note reads "Fork proposed: an admin approves it from Governance".
2. As ada, the Inbox has an approval Change an actor: "Sam Ortiz asks to change an actor.", with the reason "critical actions require an admin". Open payload to see that it is a fork of forecast-analyst as forecast-analyst-west. Choose Approve. The note says "Approved", and forecast-analyst-west is created Off and Sandboxed, like an admin's fork.

When a fork is refused#

The library shows Fork on every module agent to every person, so some forks are refused only after you type a slug. The note shows the reason:

A fork, a hired agent and the built-in agents have no Fork button. Asked through the API, a fork of one is refused with "not_forkable: sales-assistant-acme is a company agent; only an installed module's agents fork".

Hiring an agent#

Hire an agent on the Directory (admins only) makes a company agent that is not a copy:

Try it:

1. As ada, choose Hire an agent. Type ops helper as the Id, pick Support as the Team, Ticket triage (ticket-triage) under Skills and Maya Patel under Acts for, and choose Hire.
2. The note says "Hired ops-helper (sandboxed). Promote trust from the directory when it has earned it." Its details show it acting for Maya Patel, "hired to act for this person".
3. As maya, the agent is among the agents she can ask at once.

People who are not admins do not see Hire an agent. Through the API, a hire by anyone else becomes an approval for an admin.

The bundled agents#

Nine department modules carry 29 agents. Four modules extend another and can be enabled only once it is: Sales and Marketing need CRM, Support operations needs Support, Product operations needs Product. In the last column, (rule) marks what the module's policy enforces; the rest is what the playbook tells the agent to do. These descriptions come from the modules' definitions; most were not run in the demo.

CRM#

Sales (needs CRM)#

Marketing (needs CRM)#

No Marketing agent may delete leads, contacts or audiences (rule).

Support#

Support operations (needs Support)#

Product#

Product operations (needs Product)#

Tech#

Founder#

The Context module adds three more agents, Context capture, Curator and Distiller, which capture and file what the company writes down; see Context capture.

The built-in agents#

Three agents come with the kernel itself, belong to no team and are not in the library. None installs or changes anything itself: each proposes, and a person approves.

Their skills are built into the kernel, so their Playbook tab shows "No playbook yet; instructions are used". All three need a real model to do their work.

Scorecards#

The Scorecard tab answers "is this agent doing its job?". It has two parts:

• Runs: for the last 7 and 30 days, how many runs finished, how many succeeded (ended done), the success rate, the cost and the tokens.
• Measures: each measure of the agent's playbooks, as a tile with its value and description.

Every measure is counted for you, over only the records you may read. So two people can see different numbers, and a measure over records you may not read shows hidden ("You may not read the records behind it.").

Check an agent after a few runs#

1. As alice, open Directory, click Sales Assistant, choose Ask Sales Assistant. Type "List my leads" and choose Ask agent. The answer is a table of her four leads: Contoso Health, Tailspin Toys, Litware and Adventure Works.
2. In the same conversation ask "Email a follow up to my contacted lead". The run parks on an approval card Send email: "Sales Assistant asks to send email." to dana@litware.example. Choose Approve. The email is sent and Litware's next follow-up moves three days on.
3. Open the Sales Assistant's Scorecard. You should see:
   • Last 7 days: 2 runs, 2 succeeded, 100%, $0.01, about 2,000 tokens;
   • follow ups due 0 (it was 1 before the email moved Litware's follow-up date on);
   • open pipeline $21,000.00: Alice's own open leads.
4. Sign in as sam and open the same tab: open pipeline is $66,000.00, the whole Sales team's. As ada both measures are hidden, because the admin is not in Sales and leads are Sales' records. As eli the tab says only admins, the team and the people it acts for see it.

The Directory's Track record column ("3 runs · 100% ok" once the mention above has run too) and the Last 7 days column on a module's Agents tab ("3 runs · 100% done · $0.01") show the same run counts in short.

Not possible yet#

• Setting an agent back to Sandboxed from the Directory. Module agents and forks change tier only on the module's Agents tab.
• Forking a fork, a hired agent, or the Architect, Designer or Schema Steward.
• Turning a fork on from the Directory or the library; it is done on the module's Agents tab.
• Hiring an agent from the Directory as anyone but an admin.
• Seeing an agent's runs on its page; use Work.
• A scorecard window other than 7 and 30 days, or measures over time.
• A playbook for the built-in agents, or for any skill without eval cases.

Known problems on this page#

• Promote buttons that always fail. The Directory offers Promote to standard and Promote to trusted on every module agent and fork, and "Promote to standard" even on an agent that is already standard. For module agents and forks both are refused ("trust tier is managed by module crm").
• Fork buttons that will be refused. The library shows Fork on all 32 module agents to everyone, including agents of other teams (refused for a team lead and for anyone who is not an admin or lead) and Context capture, Curator and Distiller (refused: no eval cases).
• Edit buttons that will be refused. The Playbook tab shows Edit and Propose change to everyone; only an admin or the team's lead may propose.
• The fork approval does not say it is a fork. An admin is asked "Sam Ortiz asks to change an actor" and must open the payload to see that it is a fork of Forecast Analyst.
• An off fork looks active and paused. Its Directory status reads Active. On its page Ask is greyed out with the tooltip "Paused by an admin", and an ask through the API is refused with "kill switch engaged: sales is paused", though nobody paused it and the Sales module is enabled; the fork is waiting to be configured. An admin still sees Ask in its Directory details.
• Write one and Convert on built-in agents. The Architect, Designer, Schema Steward and the Context agents show Write one and Convert to a playbook; both fail with "has no eval cases".
• A stub answer counts as a success. In the demo with no model, a fork's "No model is configured" reply counts as a succeeded run on its scorecard.
• A fork's budget shows $0.00 on the module's Agents tab, and its Edit sheet pre-fills Budget 0, although the agent keeps the original's $50.00 budget.
• Lead values in raw cents. The Sales Assistant's lead table in the conversation shows value as 1800000 rather than $18,000.00.

For developers#

API#

All under /api, with Authorization: Bearer <token>.

Errors: 403 denied (fork or proposal by someone not allowed), 404 not_found, 409 fork_exists, not_forkable, module_not_installed, managed_by_module, proposal_open, playbook_only, 422 invalid (bad slug, playbook out of range) and evals_required.

CLI#

The CLI reads and writes the state file directly; with a server running, use it only for reads.

ok agent library --state <state.db> --as ada             # every agent and its state
ok agent fork sales-assistant --slug acme --as ada       # fork (admin: at once)
ok playbook show lead-qualifier --state <state.db>       # an agent's or a skill's playbook, as JSON
ok playbook propose lead-qualifier playbook.yaml --reason "Score the campaign source"
ok actor hire agent.yaml --acts-for maya                 # hire from a YAML definition
ok actor trust ops-helper standard                       # set a tier (not for module agents)

Where it lives#

Writing your own pack, skills and evals: Writing packs and skills. Agents built outside OrchKernel: Outside agents.