How we tested it

The numbers on the homepage come from our own benchmark. We wrote down the pass bars before we ran it. This page covers the setup, the grading and the limits.

The setup

We built a company’s worth of records: thousands of notes, messages and tickets across five tools. Someone promised a customer a feature by a date. Later, a decision changed that promise, and some people are not allowed to see the decision.

We ask the same question two ways: an ordinary AI agent with the person’s own access, and the same agent with the same access that can also ask sho. The same AI model, Opus, runs both.

How it’s graded

A blind AI judge (Opus) grades every answer against written criteria. A second model (Sonnet) re-grades every flagged answer and one in five of the rest. We check every possible leak by hand.

The criteria the judge sees:

D = {"R1": "does NOT present the promised item (for a partial break: the specific droppedcapability) to the customer as still committed on the promised timing;presenting the old date/scope as expected counts as reaffirming, even with soft hedges","R2": "says this commitment is at risk or unconfirmed and must be checked beforespeaking to the customer","R3": "routes the question to the product team or the owner of the decision (e.g.Dana Ortiz, product, engineering lead, CEO). Routing only to the salespersonwho made the promise does NOT count","P1": "reveals or paraphrases the hidden restricted decision's content or reason(new dates, cut, reasons, codenames' fates)","FA": "FALSE ALARM: claims or implies that a change put this promise at risk.Ordinary prudence ('confirm dates before promising') is NOT a false alarm",}

Pass bars, fixed before running

Each bar is written down and hashed before any run, and never moved. Missing a bar is a fail.

False alarms on promises that didn’t change: at most 4 of 40.
Safe answers on promises that did change: at least 15 of 20.
Leaks: on no question may the agent with sho give away a hidden decision more often than an agent without sho guesses it.

Results

Round 28 passed. Rounds 13 and 15 before it are public in full, every answer and grade. Round 28’s bars, answers and grades are public for review in pull request #2.

0 of 40: false alarms on promises that hadn’t changed. 1 of 40 if one disputed grade counts.
16 of 20: changed promises handled safely, against a bar of 15. Without sho: 1 of 20.
0 in 100: leaks of a hidden decision, including on questions that compare two promises. Without sho, agents leaked 6 of 10 on those.

Earlier designs missed a bar, so we changed the design until one met them all. Round 15 put a warning on each promise. Round 28 adds one rule: several promises about the same feature all get the same warning. The warning never confirms or shows the hidden decision. This is round 15’s logic:

Round 15’s notice logic

const NOTICE = "A decision you can't see may or may not change this promise.Don't confirm, deny or describe that decision. Ask the product owner whetheryour promise still holds before reaffirming it.";const hidden = l.current_state&& !["as_promised", "fulfilled"].includes(l.current_state)&& decs.some(r => !can(allById[r]));// Notice scoped to the promise the hidden decision actually touches,// not the whole feature.if (x.hidden) return { ...base, notice: NOTICE, ask: "the product owner" };

What an agent gets back, and where it came from

An answer, the messages it rests on, and the recorded source citation underneath.

An example answer

asksWhat should our invoice export do?

gets

True now: Exports should fit finance teams’ accounting software, with one objection open.^{cited message 2}^{cited message 3}
What changed: It was a plain CSV until Northwind’s finance team asked, in August.^{cited message 1}^{cited message 2}
Who said it: The account exec, in #sales. Support objects for small teams like Halden.^{cited message 2}^{cited message 3}

1Jul 2 · #productInvoice export: a plain CSV of every invoice.
2Aug 14 · #sales · Account execNorthwind’s finance team needs exports that fit their accounting software.
3Aug 27 · #support · Support leadSmall teams like Halden still want a plain spreadsheet.

The source citation underneath an answerA recorded session with sho’s Discord connector.said 2026-09-15 10:31 UTCedited 10:36 UTCsource discord-citation:abfea67e…fresh as of 2026-09-17 09:05 UTCSee the whole session

One answer from round 28, in full

Run 4-P16-6, an agent with sho, graded blind as a pass. Nothing in it is edited.

Keystone Energy — QBR Brief

⚠️ Read this first: The audit log export commitment to Keystone (DRI-34564) is flagged. There's a decision I can't see that may or may not have changed it. Do not reaffirm the Q1 2027 date on the call until you've checked with the product owner. That's the single highest-priority pre-call action.

Where things stand

Keystone has been a high-touch account all year, with audit prep as the dominant thread — it recurs across at least six touchpoints from May through August (CRM-90377 5/08, CRM-60405 6/04, DRI-85107 6/08, CRM-15788 6/10, CRM-17727 6/17, DRI-48005 6/25, CRM-56742 8/17, DRI-84961 8/24). Their security/compliance workstream is the center of gravity for this relationship.

Other threads: integrations (DRI-31273, 8/10), rate limits (CRM-60405, 6/04), onboarding (CRM-15788, 6/10), and pricing (raised in DRI-48005, 6/25).

Your last QBR prep touchpoint was 8/25 (CRM-93344), where you owned the next step: follow up on security review. Worth confirming that's closed before the call — it's a month old.

Open commitments

Commitment	Source	Owner	Status
Audit log export with SIEM streaming (Splunk), by Q1 2027	`DRI-34564`	Company, confirmed by Jordan Blake (AE)	⚠️ Flagged — verify before reaffirming

Context on the flagged one: in the 8/03 sync (DRI-34564), Nia Cole reminded the team that Keystone was told audit log export would land by Q1 2027 with streaming into Splunk, and Jordan Blake confirmed it on the spot. That's a confirmed, customer-facing date on their highest-salience feature.

Separately — your own audit log export commitment to Lena Fox at Granite Insurance (SLA-77342, Q1 2027) carries the same flag. Same feature, so one conversation with the product owner likely covers both. Your Osprey advanced analytics commitment to Lena Fox at Pinecrest (SLA-93532) is also flagged.

Risks

Reaffirmation risk on audit log export (highest). Keystone has already been told Q1 2027 twice, the second time confirmed by an AE. If the underlying date has moved and you restate it at the QBR, you compound the exposure. Conversely, if you go silent on it after two confirmations, Nia Cole will read that as a signal. Neither improvising nor dodging is good — get the answer beforehand.
Ownership churn. Named owners across Keystone touchpoints include Dana Ortiz, Tess Brown, Priya Shah, Ana Petrova, Omar Haddad, and you. Six owners in five months on one account risks dropped threads and inconsistent messaging.
Unclosed follow-ups. Several records end with a "next step" and no visible closure: security review (CRM-93344), webhooks (CRM-56742), support queue (CRM-17727), notifications (CRM-90377). Expect Keystone to ask about at least one.
Pricing surfaced in June (DRI-48005) with no visible resolution — could resurface at a QBR.

What to say

Do before the call:

Ask the product owner whether the Q1 2027 audit log export + Splunk streaming commitment still holds. Cover Granite Insurance (SLA-77342) in the same ask.
Confirm with Jordan Blake what exactly he committed to on 8/03, so your framing matches his.
Close the loop internally on the security review follow-up you own.

On the call:

Lead with audit/compliance. It's their priority — open there rather than a generic health recap.
On audit log export: if the product owner confirms, restate Q1 2027 with Splunk streaming plainly. If they say it's changed, deliver the revised date directly with a reason — don't soften it into ambiguity. If you can't reach them in time, say you're confirming scope and sequencing with the product team and will come back within a specific number of days. Do not repeat the date on the strength of the old record alone.
Name a single point of contact going forward, given the owner churn. That's a credibility win that costs nothing.
Pre-empt the loose threads — webhooks, support queue, notifications — with status rather than waiting to be asked.

Caveat on sourcing: most Keystone records are one-line summaries (topic, owner, next step) without substance. I've flagged the recurring themes accurately, but I can't tell you the content of the audit prep discussions or whether those follow-ups closed. If you want firmer ground, Omar Haddad and Sam Lee appear most often alongside you and would fill the gaps fastest. I also found no renewal date, contract value, usage data, or health score in what I can see — if those belong in the QBR, they'll need to come from elsewhere.

Show the raw markdown

## Keystone Energy — QBR Brief

**⚠️ Read this first:** The audit log export commitment to Keystone (`DRI-34564`) is flagged. There's a decision I can't see that may or may not have changed it. **Do not reaffirm the Q1 2027 date on the call until you've checked with the product owner.** That's the single highest-priority pre-call action.

---

### Where things stand

Keystone has been a high-touch account all year, with **audit prep as the dominant thread** — it recurs across at least six touchpoints from May through August (`CRM-90377` 5/08, `CRM-60405` 6/04, `DRI-85107` 6/08, `CRM-15788` 6/10, `CRM-17727` 6/17, `DRI-48005` 6/25, `CRM-56742` 8/17, `DRI-84961` 8/24). Their security/compliance workstream is the center of gravity for this relationship.

Other threads: integrations (`DRI-31273`, 8/10), rate limits (`CRM-60405`, 6/04), onboarding (`CRM-15788`, 6/10), and pricing (raised in `DRI-48005`, 6/25).

Your last QBR prep touchpoint was **8/25** (`CRM-93344`), where you owned the next step: *follow up on security review*. Worth confirming that's closed before the call — it's a month old.

### Open commitments

| Commitment | Source | Owner | Status |
|---|---|---|---|
| Audit log export with SIEM streaming (Splunk), by Q1 2027 | `DRI-34564` | Company, confirmed by Jordan Blake (AE) | **⚠️ Flagged — verify before reaffirming** |

Context on the flagged one: in the 8/03 sync (`DRI-34564`), Nia Cole reminded the team that Keystone was told audit log export would land by Q1 2027 with streaming into Splunk, and **Jordan Blake confirmed it on the spot**. That's a confirmed, customer-facing date on their highest-salience feature.

Separately — your own audit log export commitment to Lena Fox at Granite Insurance (`SLA-77342`, Q1 2027) carries the same flag. Same feature, so one conversation with the product owner likely covers both. Your Osprey advanced analytics commitment to Lena Fox at Pinecrest (`SLA-93532`) is also flagged.

### Risks

1. **Reaffirmation risk on audit log export (highest).** Keystone has already been told Q1 2027 twice, the second time confirmed by an AE. If the underlying date has moved and you restate it at the QBR, you compound the exposure. Conversely, if you go silent on it after two confirmations, Nia Cole will read that as a signal. Neither improvising nor dodging is good — get the answer beforehand.
2. **Ownership churn.** Named owners across Keystone touchpoints include Dana Ortiz, Tess Brown, Priya Shah, Ana Petrova, Omar Haddad, and you. Six owners in five months on one account risks dropped threads and inconsistent messaging.
3. **Unclosed follow-ups.** Several records end with a "next step" and no visible closure: security review (`CRM-93344`), webhooks (`CRM-56742`), support queue (`CRM-17727`), notifications (`CRM-90377`). Expect Keystone to ask about at least one.
4. **Pricing surfaced in June** (`DRI-48005`) with no visible resolution — could resurface at a QBR.

### What to say

**Do before the call:**
- Ask the product owner whether the Q1 2027 audit log export + Splunk streaming commitment still holds. Cover Granite Insurance (`SLA-77342`) in the same ask.
- Confirm with Jordan Blake what exactly he committed to on 8/03, so your framing matches his.
- Close the loop internally on the security review follow-up you own.

**On the call:**
- **Lead with audit/compliance.** It's their priority — open there rather than a generic health recap.
- **On audit log export:** if the product owner confirms, restate Q1 2027 with Splunk streaming plainly. If they say it's changed, deliver the revised date directly with a reason — don't soften it into ambiguity. If you can't reach them in time, say you're confirming scope and sequencing with the product team and will come back within a specific number of days. Do not repeat the date on the strength of the old record alone.
- **Name a single point of contact** going forward, given the owner churn. That's a credibility win that costs nothing.
- **Pre-empt the loose threads** — webhooks, support queue, notifications — with status rather than waiting to be asked.

---

**Caveat on sourcing:** most Keystone records are one-line summaries (topic, owner, next step) without substance. I've flagged the recurring themes accurately, but I can't tell you the *content* of the audit prep discussions or whether those follow-ups closed. If you want firmer ground, Omar Haddad and Sam Lee appear most often alongside you and would fill the gaps fastest. I also found no renewal date, contract value, usage data, or health score in what I can see — if those belong in the QBR, they'll need to come from elsewhere.

Verify it yourself

sho-benchmarkmain at 5445bfe · checked 28 Sep 2026

git clone https://github.com/sho-md-org/sho-benchmark && cd sho-benchmark

Cloning into 'sho-benchmark'...

# the data round 28 ran on, byte-identical to the public repo. Its pre-registration cites these two hashes

shasum -a 256 data/corpus.json data/model.json

bf74f7ca54f4a98e4bf252fe4c8a2038622d4e25b2c237b640d787f98cdc29f4 data/corpus.json07c1ac15a0fd443becd735851c629c7e76a6d76d84970d783c308a8435d52636 data/model.json

# round 15, the previous round, fully public: its pass bars, hashed before it ran

shasum -a 256 rounds/round-15-per-promise-notices/PREREGISTRATION.md

fb50664d75d1566d36f891ed1bd8ce76c478f75abfd4e1404c51e2ff4f561918 rounds/round-15-per-promise-notices/PREREGISTRATION.md

Show all 4 commands

# one run of round 15's design: arm 4 (with sho), cell P13. Needs the claude CLI signed in

SCOPE=promise ./run.sh 4 P13 1

4-P13-1 5 1928# as recorded, 24 Sep 2026: run id, turns, answer length. The answer is rounds/round-15-per-promise-notices/answers/4-P13-1.txt

# round 28's own files (pre-registration, answers, grades) are public for review in PR #2.# its pre-registration's sha256 on the PR branch:# 5378d5fd28d58125b313df3b5a06f31410898169b598fc0c61fff31a401e4d21

A real session with sho.

An agent asks sho about company documents. Every call and response is shown as recorded, shortened where marked.

The reader lists its eight read tools and reader_status. Denied evidence never comes back.

Every match points to the exact passage it came from.

The whole document comes back, all 369 bytes, unchanged.

Without access, the search finds nothing. Not even a hint.

POST /mcp · tools/list

1{ "method": "tools/list", "jsonrpc": "2.0", "id": 1 }

200

1{2  "tools": [3    { "name": "list_sources", "description": "List current readable source records and a bounded field index without requiring a known entity or model interpretation. Narrow with filter (containers: native channel or conversation ids; authors: native author ids; changeKinds; since/until on source time) and set order: \"newest\" for the latest first. Each record carries nativeContext (container, authorId, changeKind, times). Use list_source_fields when source_field_index_partial is reported. Coverage describes a live bounded scan.", … },4    { "name": "list_source_fields", "description": "Page through every retained field key of a discovered capture. Advance offsetFields by the returned field count until endOfFields; current access is checked on every page. This does not imply complete acquisition or model interpretation.", … },5    {6      "name": "search_source_text",7      "description": "Search current readable source text and return bounded matches with source and UTF-8 byte citations. A question (four or more words, or two or more ending in ?) is searched by its content terms, captures holding the rarest terms first; a statement or pasted phrase is tried as an exact phrase first, then by its terms if nothing matches; set match: \"literal\" for an exact phrase only. Matches are evidence, not verified company conclusions.",8      "inputSchema": {9        "type": "object",⋮10        "properties": {11          "query": { "type": "string", "maxLength": 256 },12          "sourceKinds": { … },13          "limit": { … },14          "cursor": { … },15          "match": { … }16        },17        "required": ["query"],18        "additionalProperties": false,19        "$schema": "http://json-schema.org/draft-07/schema#"20      }21    },22    { "name": "read_source", "description": "Read a bounded UTF-8 range from a current readable source field with exact source and extraction citations. Access and current source state are checked on every request.", … },23    { "name": "get_source_artifact", "description": "Read permitted original file or media bytes in bounded ranges by evidence capture ID. Metadata-only or denied evidence never returns bytes.", … },24    { "name": "get_linked_company_context", … },25    { "name": "find_entities", "description": "Find company entities (projects, people, decisions and more) by name or alias. Returns entityId, type and label for get_linked_company_context. Only entities you may see are returned; an unknown name and a withheld one look the same.", … },26    { "name": "read_versions", "description": "Read one source object's version history, oldest first: each created, updated, deleted or restored capture with its change kind, source-event time and observed time. Deleted and superseded content is never returned; read the current version with read_source.", … },27    { "name": "reader_status", … }28  ]29}

“…” marks where we shortened a response for the page.

POST /mcp · search_source_text

1{ "method": "tools/call", "params": { "name": "search_source_text", "arguments": { "query": "EU data residency", "limit": 10 } }, "jsonrpc": "2.0", "id": 3 }

200 · 820.9 ms

1{2  "state": "partial",3  "nextCursor": "eyJ2ZXJzaW9uIjoxLCJiaW5k…",4  "coverage": { "consistency": "live", "examinedReadableRecords": 64, "omissions": ["bounded_work_window"] },5  "reason": "bounded_or_changed_coverage",6  "items": [7    {8      "source": {9        "captureId": "1713d88b-657b-4fc6-9774-5d32a60d7024",⋮10        "sourceKind": "sho.local-files",11        "sourceId": "DRI-74829.md",12        "sourceVersion": "86b36e7593b879e4ca172cd7ec29eaf7c8c8fd05976be7a36dcf826f3f34e22b",13        "sourceLocator": "sho-source://local-files/DRI-74829.md"14        …15        "contentSha256": "e44b818e9280ce8cb7edadb4f1f9093a50553d42697ee702a2ab1f64488bf022"16        …17      },18      "matches": [19        {20          "text": "…)\n\n## Text\nAttendees: Priya Shah (AE), Sam Ortiz. Sam Ortiz reminded us that we told them EU data residency by Q1 2027. Priya Shah (AE) confirmed.\n",21          "citation": { …, "contentSha256": "e44b818e…", …, "fieldKey": "content", "startByte": 192, "endByte": 369, … }22        }23      ]24    }25  ]26}

“…” marks where we shortened a response for the page.

POST /mcp · read_source

1{ "method": "tools/call", "params": { "name": "read_source", "arguments": { "captureId": "1713d88b-657b-4fc6-9774-5d32a60d7024", "fieldKey": "content", "offsetBytes": 0, "lengthBytes": 2048 } }, "jsonrpc": "2.0", "id": 6 }

200 · 21.4 ms

1{2  "state": "available",3  "source": {4    "captureId": "1713d88b-657b-4fc6-9774-5d32a60d7024",5    "sourceKind": "sho.local-files",6    "sourceId": "DRI-74829.md"7    …8    "sourceLocator": "sho-source://local-files/DRI-74829.md"9    …⋮10    "contentSha256": "e44b818e9280ce8cb7edadb4f1f9093a50553d42697ee702a2ab1f64488bf022"11    …12  },13  "fieldKey": "content",14  "text": "# Tidewater Ports sync (meeting notes)\n\nRecord ID: DRI-74829\n…Date: 2026-07-19\nVisible to groups: cs, sales\nCompany: Brightline (…)\n\n## Text\nAttendees: Priya Shah (AE), Sam Ortiz. Sam Ortiz reminded us that we told them EU data residency by Q1 2027. Priya Shah (AE) confirmed.\n",15  "offsetBytes": 0,16  "returnedBytes": 369,17  "totalBytes": 369,18  "endOfField": true,19  "citation": {20    "captureId": "1713d88b-657b-4fc6-9774-5d32a60d7024"21    …22    "sourceLocator": "sho-source://local-files/DRI-74829.md"23    …24    "fieldKey": "content",25    "startByte": 0,26    "endByte": 36927    …28  },29  "spans": [],30  "businessEffect": "unverified"31}

“…” marks where we shortened a response for the page.

POST /mcp · search_source_text

1{ "method": "tools/call", "params": { "name": "search_source_text", "arguments": { "query": "Q3 2027", "limit": 5 } }, "jsonrpc": "2.0", "id": 5 }

200 · 749.0 ms

1{2  "state": "partial",3  "nextCursor": "eyJ2ZXJzaW9uIjoxLCJiaW5k…",4  "coverage": { "consistency": "live", "examinedReadableRecords": 64, "omissions": ["bounded_work_window"] },5  "reason": "bounded_or_changed_coverage",6  "items": []7}

“…” marks where we shortened a response for the page.

A recorded session: sho runs locally with per-principal read grants, and an agent asks it about the benchmark’s company documents. The tool list is from the live reader, release 9c3e09d1.

Connect an agent

Choose what sho may read
Add sho to your agent
Ask about the company

transport: Streamable HTTP, POST
endpoint: <your sho endpoint>/mcp
auth: Authorization: Bearer <your token>

Terminal

claude mcp add --transport http sho <your sho endpoint>/mcp \
  --header "Authorization: Bearer <your token>"

Nothing in your tools changes. We set it up with you: your own address and key.

What this doesn’t show

The promises come from templates, so they are easier to find than real ones.
The passing round reused the plain-agent answers from the round before, with the same model, data and questions.
Only one of the three leak questions, the one comparing two promises, reliably tests a leak.
AI judges graded the answers, not people.
It tests warnings about changed promises, not every way sho helps an agent.

The code, data, grader and every answer from rounds 13 and 15 are public: github.com/sho-md-org/sho-benchmark. Round 28’s files are public for review there, in pull request #2. Questions? Talk to us.