Impact: is the agents' work helping?

Ratings tell you whether people liked an answer. They do not tell you whether the leads an agent followed up were qualified sooner, or whether tickets were resolved faster. Company → Impact answers that from your records' own history: for each outcome you turn on, how many leads, deals or tickets started, how many reached the outcome, how fast the first response came, split by who worked on them, against the weeks before agents started.

Everything on the page is counted by OrchKernel from history. No model decides whether an outcome happened, no agent can read the figures or change how they are counted, and no figure is about one employee.

This page covers what every run records, the outcomes the apps bring, turning one on, reading the cards, who sees what, adding your own outcome (on records kept here or in a connected system), the Monday report, the settings, and the API and command line. The steps use the demo company (ok demo serve): Ada is the admin, Sam leads Sales, Alice is a rep. What is not possible yet is listed at the end.

Words used on this page#

• Outcome: something that happens to a record after work on it: a lead qualified within 14 days, a ticket resolved within 3 days.
• Subject: one record an outcome counts: one lead, one ticket. It starts when it is created.
• Window: the days within which the outcome counts (14 for a lead qualified within 14 days). A subject has closed once its window, and the window in which a success can still be undone, has passed.
• Label: who worked on a subject: agent-led (an agent made the first counting touch), agent-assisted (a person touched it first, then an agent helped) or people only (no agent made a counting touch). Every subject gets one, whether or not it succeeds.
• Agent-set: a subject whose outcome an agent set on its own (it moved the stage to Qualified itself). It is not a success until a person or another source confirms it.
• Baseline: the same figures for the 8 weeks before agents started on that kind of record.
• Records touched: what a run read, created, updated, contacted or answered, and what it drafted or was declined, by record id.

What every run records#

From the day this was built, every run records which records it touched and how: read (the first 50 records of a query, or a connected system's records it read), created, updated, contacted and answered (a send, an activity, a reply), drafted (a write or send it asked approval for) and declined (an approval that was refused). These are ids, roles and times only, never the records' values.

Open a run and its Details shows Records touched: the records it changed or contacted, each with what it did, and how many it read. Older runs have none: what a run touched cannot be worked out afterwards.

Outcomes can be counted again from history at any time; what a run touched can only be recorded when it happens. That is why it is recorded for every run, whether or not any outcome is turned on.

The outcomes the apps bring#

The lead and ticket outcomes also measure the first response: from the start to the first contact by an agent or a person.

They are off until an admin turns them on.

Turning an outcome on#

1. Sign in as Ada and open Company → Impact, then Definitions.
2. Find Lead qualified within 14 days and choose Turn on.
3. The sheet says what the outcome counts and what the baseline needs: for records kept here, it is read from their history with no calls to another system; for a connected system, the number of calls the baseline will make, once, within its own budget.
4. Choose Turn on (or Turn on and start the baseline). New leads are counted from now, and the 8 weeks before agents started are counted for the baseline: at once for records kept here, and for a connected system on the clock's next tick, until the estimated calls are used.

For a connected system without a history tool the sheet says instead that there is no baseline until 8 weeks have passed: its open records are read once a day, which cannot reconstruct the past.

Turn off stops counting; what was counted so far stays readable. Change saves a new version of the outcome: it starts its own figures, and the earlier version's stay as they were counted (the outcome's page lists them under Other versions; open one to read its figures).

Reading the cards#

Overview shows one card per outcome that is on. Each card has:

• Started, last 28 days: subjects started in the last 28 days, and how many are still open.
• Succeeded: of the subjects whose window has closed, how many reached the outcome, and the rate. Under 20 subjects it says Too few to tell; before any window has closed it says No rate yet. A 14-day outcome shows no rate in its first two weeks, but its open count and first response from day one.
• First response, median: counted as soon as each first contact happens. Records nobody has contacted yet are left out of the median; How this is counted says how many.
• Before agents: the baseline, or No baseline when there is no history from before (a connected system without a history tool, for its first 8 weeks).
• Since agents started: the same rate over every subject since then, beside the last 28 days.
• Who worked on it: the rate, closed and started per label (Too few to tell for a label with fewer than 20 closed).
• The agent-set line, when agents set an outcome nobody has confirmed yet: those are never counted as success.
• Each week: the subjects started each week, split into succeeded, still open and did not succeed.
• How this is counted: the outcome in words, its window, the labels, what is left out and the caveats.

The baseline is a before and after, not proof: other things changed in those weeks too. Subjects in the baseline weeks that an agent touched are left out of it.

The figures are the last 28 days of subjects whose window has closed. A lead or a member may see two cards for one outcome: the company's, and their team's (named under the title).

Open a card's title for the outcome's page: the outcome in words under the title, the same figures in detail, the other versions, and for admins By team: every team of 5 or more people, and Unassigned for subjects whose owner matches nobody, when those subjects belong to 5 or more owners (otherwise they are one or two people's, and count in the company's figures only).

Who sees what#

A team with fewer than 5 people is never broken down further: its card has its totals only, with no split by who worked on it and no agent-set line. Nothing is shown per person. An outcome on records some people may not read (HR's, say) shows its figures only to admins and those who may read them. An agent working for one person counts as agent work in the company's figures, never in a breakdown that could single that person out. Value (an amount per success) is shown to admins and to the people an admin names in the settings; everyone else sees counts and rates. Value is never added up into a return on investment.

Adding your own outcome#

On Definitions, choose Add an outcome. Give it a name, then say what it counts:

• Records kept here: choose the collection (leads, tickets, deals).
• A connector's records: choose the connector and the kind of record (lead), then the tool that lists them by created date and the names of its arguments (created from, created to, page, page size). Every record created in a period is listed through it, including ones no agent touched, with the company's key and without saying who is asking. A connector that uses each person's own key cannot back an outcome; the editor says so.

Then say what counts as success (the field, the values it becomes, within how many days), whether a later change undoes it, whether to measure the first response, and optionally a value per success with its currency.

For a connector's records, also choose, when the system has them:

• The tool that reads one record's history: OrchKernel reads stage changes from it. Choose Read a sample: it calls the tool once with the company key and proposes which part of an entry is the field, the old and new value, when and who. Check each part and Confirm; nothing is read from the tool for outcomes until you do. A change in the system's history that matches an agent's own update (the same record, field and value within 10 minutes) is the agent's.
• The tool that lists its activities: a person's call or note there counts as their contact. It is confirmed the same way.
• The owner field and the tool that lists its people: owners are matched to people here by email, through the people tool's id (or the key the definition names); an owner nobody matches counts under Unassigned.

An agent's change counts as a success only once a person approves that very write, or a person or another system confirms it later. An approval of something else on the same record confirms nothing.

To have a connector's tool count as a contact (a call or note logged with add_lead_activity, a reply with add_ticket_comment), an admin marks it on the connector's Which tools? step: Counts for Impact as Contacting the record or Answering the record. A tool that changes no record of its own asks which kind of record it names and the argument holding its id. Changing a connector an outcome reads is an event that names the outcome.

Without a history tool, open records are read once a day within the daily budget, and there is no baseline until 8 weeks have passed. The page says so.

The Monday report#

Every Monday from 08:00 in the company's time zone, admins and team leads get the figures they may see on Impact in their inbox: per outcome the numbers on the cards, what changed since last week's report, and the caveats. It opens Company → Impact. When email is set up they also get an email saying the report is ready and which outcomes it covers, with a link: the figures, the teams and any value stay in OrchKernel, since a mailbox may be read by an inbox assistant. OrchKernel counts it; no model writes it. Nothing is sent while no outcome is on.

Settings#

Company → Impact → Settings, for admins:

• Reads a day: the most reads of a connected system's records a day, for the whole company (2,000 unless changed). When it is reached, the rest wait for the next day and the page says so.
• Monday report: on or off, and who gets it: admins and team leads, or the people you choose.
• May see money figures: the people besides admins who see value.

API and command line#

Every route answers people only: an agent token is refused with 403 agent_token_scope, and an agent named any other way with 403 people_only.

On the command line, ok impact prints the cards as you may see them and ok impact show <id> one outcome in detail (ok impact show <id>@1 its version 1); both only read.

Not possible yet#

• Comparing with a group agents did not work on, comparing an agent's versions or memory, and time saved come in part 2.
• What runs touched before this was built is not known, and never will be.
• An outcome counted by a model's judgement is not offered, on purpose.