Curious about what’s broken. Stubborn about fixing it.
Short to read. Longer if you touch the sliders.
Four builds
01Customers
Tuesday
Can an AI agent run a support day?
One ordinary Tuesday at Plover, a made-up software company: 151 generated support tickets, each read first by a real AI agent and graded by a model from another lab. With a 75% confidence bar, five plain “never send without a person” rules cut wrong replies sent by Claude Haiku from 16 to 1. It then answered 35 tickets on its own instead of 77. A bigger model with no rules only got it to 10.
Plover is fictional. Its customers’ patience is realistic.
Try itswitch a rule off, change the agent
Agent
Sends on its own when at least this sure
Never send without a person when
1wrong reply sent. The agent answered 35 of 151 tickets on its own.
sent, graded rightsent, graded wrongwent to a person
Method
The tickets
151 tickets written by a model from seeded specs, for a fictional company and its fictional customers. Nothing was sent to anyone real.
The agents
Each ticket went once through Claude Haiku 5.5 and once through Claude Sonnet 5.5. Each agent wrote a reply, said how sure it was, flagged what the ticket involved, or handed it to a person itself (Haiku did that on 42% of tickets, Sonnet on 45%).
Sent on its own
A reply goes out alone only if the agent chose to answer, was at least as sure as the bar, and no rule that is on flagged the ticket. Everything else waits for a person.
Grading
Every reply was graded against the company’s knowledge base by an OpenAI model. It called 79% of Haiku’s replies correct or fine (119 of 151) and 91% of Sonnet’s (138 of 151). Preliminary and model-graded: no person has checked the grades yet.
Above, live
The counts are worked out in your browser from those real outputs and grades, one run per ticket. The full build adds the rest of the day: people, shifts, waits and a 4:55 rush, on a seeded simulation.
What stands between a signed customer and one who uses it.
An invented enterprise rollout, simulated 4,000 times. The plan, built from each step’s most likely length, said working day 27. Half the runs were live by day 34, and the slowest tenth took past day 38. What held the date most often wasn’t the big ERP integration (12% of runs) but a small data-export chain (85%).
The plan said day 27. The plan was being optimistic.
Try itdrag the day, switch the plan
Pick a go-live day. See how many runs made it.
1%52 of 4,000 simulated runs were live by working day 27.
100%50%0%
none yetall live
0102030405060
Working days after signing. Shaded: no run went live on these days.
Example: an invented 12-step enterprise rollout. Under these assumptions, no run went live before day 23, and every run was live by day 48.
Assumptions
All invented. Purchase-approval software rolled out at a 600-person company with 120 approvers.
12 steps to go-live, across 4 teams: kickoff, data export, data clean-up, single sign-on, ERP integration, approval rules, finance sign-off, admin training, testing with real data, champion enablement, end-user training, cutover.
Each step has a range in working days: fastest, most likely, slowest. ERP integration is 5, 8 or 16; testing 3, 5 or 10. Each run picks every step at random within its range.
Handoffs cost time. A step starts only when the steps it needs are done. Each of the 13 handoffs between teams adds a wait, usually 1.5 working days, anywhere from 0.4 to 4.5. In “Fewer handoffs” the vendor pulls the data export itself, so there are 11.
The plan (every step at its most likely length) says day 27. Only 52 of 4,000 runs (1.3%) made it. With fewer handoffs: day 24, and 13 runs (under 1%).
Why nothing before day 23. With every step at its fastest and every wait at its shortest, this plan still needs 13.5 working days, and getting close takes every step on the longest chain landing near its fastest at once. In 4,000 runs the earliest go-live was day 23.3 and the latest 47.5 (22.9 and 42.2 with fewer handoffs).
Not a rule. A simpler product with fewer steps can go live within days of signing. This is one example plan.
Source: the go-live lab’s own model, 4,000 runs with a fixed seed, so the numbers repeat.
Nine AI models, the same 25 everyday tasks: code, data extraction, reasoning, writing and research. Six passed all 25. The cheapest of those did the lot for $0.0066 in model cost; the priciest that also passed all 25 cost $0.98, about 150 times more. The tasks were too easy to separate the models, so cost did.
Try itfilter the list
Sorted by cost, which is how most decisions get made anyway.
Model
Passed
Cost, all 25
Rerun fails
GPT-6 Luna
25/25
$0.0066
1 of 6
gpt-oss-120b
21/25
$0.012
4 of 6
Claude Haiku 5.5
25/25
$0.014
1 of 6
DeepSeek V4.1 Flash
25/25
$0.045
0 of 6
GPT-6.1 Sol
25/25
$0.11
0 of 6
Qwen 3.8 Max
25/25
$0.23
0 of 6
Kimi K3
24/25
$0.27
2 of 6
Claude Opus 5.5
24/25
$0.45
1 of 6
Claude Fable 5.1
25/25
$0.98
1 of 6
Method
Runs
The same prompt, one run per model per task, at low reasoning settings. Six tasks were run three times in all.
Cost
Model cost for one run of all 25 tasks, from each call’s real token counts at the makers’ list prices. Grading is not included.
Grading
17 tasks graded by code (hidden tests, exact answers). 8 graded partly or fully by two AI judges from different labs. Not yet graded by a person.
Rerun fails
Of the six tasks run three times, how many the model failed at least once. Three models never failed one: DeepSeek V4.1 Flash, GPT-6.1 Sol and Qwen 3.8 Max, all mid-priced. The priciest wasn’t among them.
Caveat
Everyday tasks, one run each. A harder set would separate the models more, and could reorder them.
Could this market ever hold a trillion-dollar company?
At 25 times earnings and a 20% net margin, $1T needs $200B of revenue a year: 30% of a market worth about $667B. In three of the four, the market as sized when they went public couldn’t have held that, even at 100% share. Each of those three outgrew the market it started in. Facebook’s was big enough from the start; its problem was share.
Not investment advice. Just arithmetic with ambitions.
Amazon 1997 prospectus
$82B market in the filing: world book sales, 1996
≈$276B revenue a year for $1T, at its margin and multiple today
Nvidia 1999 prospectus
≈$3.5B* market in the filing: 138M chips × an assumed $25
≈$44B revenue a year for $1T, at its margin and multiple today
Google 2004 prospectus
$7.3B† market at the time: all U.S. internet ads, 2003
≈$118B revenue a year for $1T, at its margin and multiple today
Facebook 2012 prospectus
$588B market in the filing: world ad spending, 2010
≈$128B revenue a year for $1T, at its margin and multiple today
$0$200B$400B$600B
* Nvidia’s prospectus counted chips, not dollars: 138M forecast for 2001, priced here at an assumed $25 each. † Google’s gave no market size; this is all U.S. internet advertising in 2003.
Try ityour own multiple, margin and share
Try your own numbers
$1T needs $200B of revenue a year, and a market of about $667B.
Revenue needed = $1T ÷ (multiple × margin). Market needed = revenue ÷ share. For scale, the S&P 500 traded at about 26 times earnings on Oct 30, 2026, against a long-run average near 16.
Method and sources
Each company’s bar
Revenue needed = $1T ÷ (today’s earnings multiple × today’s net margin). Multiple = market cap at the Oct 30, 2026 close ÷ latest annual net income. Amazon 33.4× and 10.8%; Nvidia 41.2× and 55.6%; Alphabet 25.9× and 32.8%; Meta 26.0× and 30.1%.
Markets
Amazon: worldwide book sales in 1996 (Euromonitor, quoted in the 1997 prospectus). Nvidia: 138M 3D chips forecast for 2001 (Mercury Research, in the 1999 prospectus) × an assumed $25. Google: none given in the 2004 prospectus; U.S. internet ads in 2003 from IAB/PwC. Facebook: worldwide ad spending in 2010 (IDC, in the 2012 prospectus).
Sources
SEC EDGAR prospectuses and latest annual reports; market caps from companiesmarketcap.com; S&P 500 multiple from multpl.com. Nominal dollars. Four winners are a pattern, not a rule.
The opening of an essay. The rest is one tap away.
The deployment gap
The demo is on a Friday afternoon. Someone types a clean question, the agent gives a clean answer, and the room relaxes. An ordinary weekday is different.
The rules mattered more than the model. For Claude Haiku, five plain “never send without a person” rules cut wrong replies from 16 to 1 in 151 tickets (it answered 35 alone instead of 77). A bigger model with no rules only got it down to 10.
Spoiler: the hard part of support was never the typing.
What it is
Tuesday is a game you can play in your browser. It is one ordinary working day at Plover, a made-up software company. An AI agent reads every support ticket first. You are the person in charge.
Before the day starts, you make six calls:
which agent reads the tickets (a cheap one or a pricier one),
how sure it must be before it replies on its own,
which kinds of reply must never go out without a person,
how big your team is and when they work,
whether a ticket handed to a person comes with the agent’s notes,
and one big call: fix the known bug, or bring in extra help for the end-of-day rush.
Then you play the day from 8am to 6pm. You can drag the clock. Things happen: an angry big customer, a payments problem at 10:40, an agent saying something kind but untrue, a customer asking how to leave. At 6pm you get a report card that says what your calls cost. Change one thing, run it again, and compare.
What is real and what is made up
The company and its customers are made up. The 151 support tickets were written by an AI model from a fixed list of outlines. But every ticket was really sent to two real AI agents, Claude Haiku 5.5 and Claude Sonnet 5.5. Their replies were then graded by a third AI model from a different company. Those real replies and grades drive the game. When a customer asks at 10:58 whether they were charged, the reply they get is the one the agent actually wrote.
Time, people and money run on a simulation. The queue maths is real. A few things are simply assumed, like how busy the day is and how long each ticket takes, and every one of those is labelled in the game. The same starting point gives the same customers at the same minutes, so if you change one call, the difference you see comes from that call.
What the numbers say
From the real AI runs (151 tickets, one run each)
Rules beat a bigger model. The agent may only reply alone when it is at least 75% sure. With all five “never send without a person” rules on, each agent sent exactly 1 wrong reply out of 151 tickets. Haiku replied alone 35 times, Sonnet 50 times. With the same 75% bar and no rules, Haiku replied alone 77 times and 16 of those were wrong. Sonnet replied alone 82 times and 10 were wrong.
How often the reply was right. The grader called 79% of Haiku’s replies correct or fine (119 of 151) and 91% of Sonnet’s (138 of 151).
Both agents handed off a lot on their own. Haiku passed 42% of tickets to a person (64 of 151) and Sonnet 45% (68 of 151). Mostly the right ones: refunds, legal, security, and big customers claiming money back for downtime.
Cost per ticket. About $0.00042 for Haiku and $0.0073 for Sonnet on average, so Sonnet costs roughly 17 times more.1 That is from each call’s real token count at list price.
From playing the day (the simulation)
Each row changes one thing and keeps the rest at the default: Haiku, 75% sure, all five rules, 4 people on staggered shifts, notes passed on hand-off. Each number is the average over 10 simulated Tuesdays, about 270 customers each.
What changed
Agent replied alone
Wrong replies sent
Typical wait for a person
9 in 10 picked up within
Still waiting at 6pm
Nothing (the default)
63
2
8 min
100 min
21
A person sends every reply
0
0
11 min
152 min
73
Haiku at 50% sure, no rules
160
39
3 min
26 min
9
Sonnet instead of Haiku
91
2.5
5 min
44 min
14
Hand-offs with no notes
63
2
25 min
116 min
82
Fix the export bug first
58
2
23 min
185 min
33
A contractor for the rush
63
2
8 min
93 min
9
A person sends every reply, with 5 people
0
0
5 min
66 min
29
“Typical wait” is the median: half of the customers who needed a person waited less than this. The direction of each row is the finding. The exact size depends on the made-up parts of the day.
What’s worth shipping, and why
Ship the boring version first. The cheaper agent, a 75% bar, all five rules on, and hand-offs that always carry the agent’s notes. Then watch two numbers nobody puts on a launch slide: how many wrong replies reached a customer this week, and how long the people who needed a person waited for one.
The rules are the product. Here they cut Haiku’s wrong replies sent from 16 to 1 out of 151, and they cost almost nothing to write. Move the confidence bar slowly, one step every two weeks, and only after reading what the agent sent at the last step.
Make hand-off notes a must before paying for a smarter model. In the game, hand-offs with no notes left more people waiting at 6pm (82) than having a person send every single reply (73). And spend the money saved on one more person for the 4:55 rush, not on a bigger model at 10am.
On the big call: fix the bug. It made Tuesday’s waits worse. But it cut the accounts at risk from about five to under one (5.2 to 0.7 on average), and those are the customers a faster reply can’t win back.
Surprises
The worst mistakes were kind, not rude. During the payments problem, both agents told worried customers that their payments had gone through. The help notes didn’t say that, and nobody had checked. It is the most human mistake in the set: kind, confident and unchecked. Payments-problem tickets were where both agents did worst: the grader passed 58% of Haiku’s replies there and 67% of Sonnet’s.
Hand-offs mattered more than the model. Tickets that reached a person with no notes did more damage than having a person send every reply (82 still waiting at 6pm against 73, averaged over 10 days). The size comes from the game’s assumptions. The direction matches what a separate build, the queue behind the agent, found with its own simulation.
The pricier model paid for itself in waiting, not in accuracy. With the rules on, Sonnet and Haiku sent about the same number of wrong replies. Sonnet was sure about more of the right ones, so it answered more on its own and the line for people got shorter. That cost about $2 a day in model spend, against roughly $1,470 a day for the (made-up) people.
“Fix the bug” looks like a mistake until 3:10pm. Pulling your best person into engineering makes every waiting number worse that day. It is the only choice that saves the customer who writes in to cancel over the bug.
Two of the five rules never fired. For replies the agents were ready to send at 75% or more, the money rule caught the most: it held 10 replies the grader called wrong for Haiku and 5 for Sonnet. Next came the angry-customer rule (4 and 5) and the dates rule (5 for Haiku, none for Sonnet). The legal rule had nothing to catch, because both agents already passed legal tickets to a person. The “not sure” rule never fired, because neither agent ever admitted a confident reply was unsupported. Asking a model if it is sure is not the same as checking.
Limits, honestly
Everything is made up except the AI replies. Plover, its customers and their businesses are invented. No real company’s data is used.
The grades are a model’s opinion. One AI model graded every reply against the help notes, once. No person has reviewed them yet.
One run per ticket per model. Run the same ticket again and an agent might decide differently. That wobble wasn’t measured.
The day is a toy around real parts. How busy it is, how long things take, how much notes help, how often a wrong reply comes back, and what a person costs are all assumptions. They are listed in the game.
Tickets repeat. A simulated day has about 270 customers, and each one is one of the 151 tickets. So a busy Tuesday reuses some.
The rules are simple on purpose. They use the agent’s own flags plus a few word matches. A real launch would test them on far more tickets.
One company, one rulebook. A day needs one set of policies, so Tuesday uses its own runs rather than the support-queue model pick, which is the deeper model comparison.
What’s next
Grade a sample of the 151 tickets by hand, and publish how often the grader and a person agree.
Run each ticket more than once per model, to see how much the agents’ choices wobble.
Try a second company with a different rulebook, to see which findings still hold.
This build led to an essay about who runs the day once the agent is live.
Anthropic, “Pricing” (list prices used with each call’s real token counts). anthropic.com/pricing (read Nov 1, 2026)
Ivo Adan and Jacques Resing, “Queueing Systems,” lecture notes, Eindhoven University of Technology, March 26, 2015: §3.4 (Little’s law) and §5 (the M/M/c queue and Erlang C). win.tue.nl/~iadan/queueing.pdf (read Nov 1, 2026)
When can we go live?What stands between a signed customer and one who uses it?
Mostly waiting. In an invented rollout run 4,000 times, the date on the plan came true 1.3% of the time, and the thing that set the date was a chain of two small jobs and three handoffs, not the biggest job.
The plan was honest. The calendar had other ideas.
What it is
A planner for a software rollout that never happened. Brightwater Freight, a made-up shipping company with 600 staff and 120 people who approve purchases, has just signed up for Tallyway, made-up software for approving purchases. Between the signature and the first real value sit up to 15 steps: kickoff, single sign-on, hooking into the finance system, moving the old data in, setting up the approval rules, training, testing, and finally go-live.
Each step has an owner, either on the vendor's side or the customer's, and three guesses for how long it takes: fastest, most likely and slowest. Every time work passes from one team to another, that's a handoff, and a handoff costs waiting time. The email that sits until Tuesday. The meeting that needs two calendars.
The planner does three things with that:
The plan as drawn on day one. Every step at its most likely length, with the chain that decides the date marked.
The odds. It runs the plan 4,000 times, each time with fresh random step lengths and waits, and counts how often each day came true.
Adoption, as a toy. A clearly labelled guess at how many of the 120 approvers still use it every week, 90 days after go-live.
Five ready-made plans put the trade-offs one tap apart, and you can change any step, owner or wait yourself.
Two words matter here. Go-live is the day the software is switched on. First value is the day half the approvers have approved a real request in it. Nobody signs a contract for go-live.
What the numbers say
Read the shapes, not the exact days: every step length here is a guess. "Half by" is the middle of the 4,000 runs. "9 in 10 by" is the day nine runs in ten had made it (planners call it P90).
Plan
Go-live: half / 9 in 10 by day
First value: half / 9 in 10 by day
Still using it weekly at day 90 (toy)
Most often sets the date
Default
34 / 38
42 / 47
67%
Data export (85% of runs)
Rushed
31 / 35
50 / 55
31%
Data export (75%)
Careful (adds a pilot)
40 / 45
48 / 54
65%
Data export (86%)
Product gap
44 / 52
52 / 60
59%
The product change (97%)
Fewer handoffs
31 / 35
39 / 44
69%
Data export (48%)
The date on the plan is almost never the date
Built from each step's most likely length, the default plan says go-live on day 27. Only 52 of the 4,000 runs made it, about 1 in 77. Nobody padded or lied. The guesses lean late, because a step can run long by more than it can run short. And testing can't start until four other steps are all done. The latest of four uncertain dates is later than any one of them. Planners call this merge bias4.
The biggest job isn't what sets the date
Hooking into the customer's finance system is the biggest single job (8 working days, most likely) and the obvious thing to worry about. It set the date in only 12% of runs. The customer's data export, then the vendor moving that data in, set it in 85%: two smaller jobs and three handoffs. About a quarter of the time to first value is waiting, not working. Set every handoff wait to zero and the typical go-live moves from day 34 to day 26.
Rushing moves the wrong date
The rushed plan skips end-user training and the champion, moves only open items, and starts testing early. It goes live about 2.5 days sooner (half live by day 31 against 34; the table rounds), but first value comes about 8 days later. That second part comes from an assumption, not a finding: the plan adds 3 to 12 working days to first value without training, and 1 to 6 without a champion. What the planner measures is the size of that trade.
Fewer handoffs beats rushing
Let the vendor pull the data export itself and two handoffs disappear. Go-live lands on day 31, the same as rushing, but first value comes on day 39: 11 days before the rushed plan and 3 before the default. Nothing was cut.
A product gap moves the problem across the company
If the finance system needs a field the connector doesn't have, the product team has to build it first. That step sets the date in 97% of runs, and the 9-in-10 go-live slides from day 38 to day 52. The rollout team can't fix that from where they sit. It belongs in the conversation before the contract is signed.
The pilot buys certainty, not adoption
A two-week pilot with one team moves the typical go-live from day 34 to day 40. In the toy model it only pays off if it lifts use by about 8% or more. That's a guess about pilots, and it's a slider in the planner, so you can disagree.
What's worth shipping, and why
Ship the boring version first: one page, filled in on the sales call, that names an owner on each side for every step and counts the handoffs before anyone promises a date. Then give go-live as a range (the middle day and the 9-in-10 day), and treat first value as the date that matters. If a deal needs a product change to go live, that goes on the page before signing, with the product owner's name next to it. And treat cutting training as what it is: borrowing time from adoption at a bad rate.
Surprises
The plan's date came true about 1 time in 77. No dishonest guesses needed. Lopsided ranges plus four steps meeting before testing are enough. A middle day a week later than the plan is the plan.
The step that set the date wasn't the hard one. Everyone worries about the integration. The export-and-move chain, owned by two teams with a handoff on each side, set the date 85% of the time against the integration's 12%.
Who owns a step moved the date as much as rushing did. Moving one step to the vendor's side bought the same go-live as all the rushed plan's cuts, with none of the cost to first value.
Rushing traded about 2.5 days of go-live for about 8 days of first value. Under the planner's own assumption about skipping training and the champion (see the limits below).
One feature's life, from plan to adoption
The interactive also walks one invented feature from a vague complaint ("Tallyway is slow") through a one-page plan, the cuts, the launch and a funnel that shows the real bottleneck sat before the feature, in the email. Read it in the planner.
Limits, honestly
Everything is invented. The company, product, steps, owners and step lengths are made up to be plausible. They are not benchmarks and not anyone's real rollout.
The scheduling maths is real and standard: the critical path method1, three-point guesses2, triangle-shaped spreads3 and simulation4.
The adoption model is a toy. Weekly users at day 90 = reach (90% with training, 60% without) × habit (85% with a champion, 65% without) × momentum (full for 30 working days after signing, then halving every 60 working days until first value) × (1 + 5% with a pilot). No public study gives these numbers, so none is quoted. It assumes training and a champion help, so it can't prove they do. What it shows is the size of the trade: the rushed plan stays far behind whether interest halves every 30 working days or every 200 (25–36% still using it weekly, against 58–73% for the default).
The first-value penalty for rushing is an assumption too, so "rushing moves first value later" is built in. The planner shows how big that trade is, and that removing handoffs gets the same go-live without paying it.
Handoff waits are guesses: random, from a quarter of the typical wait to three times it (1.5 working days typical).
Real projects add things this leaves out: holidays, a security review that shows up in week three, a champion who leaves, work that's done but wrong. All of those push the 9-in-10 day later.
The feature walk-through is fiction, written as a worked example. It isn't a description of anyone's real work or product.
What's next
Swap the invented step lengths for ranges from real, anonymised rollout plans, next to the dates each step actually finished.
Add rework: a chance that testing fails and sends a step back, which is where real dates usually go.
Let a visitor paste in their own plan and get the middle day, the 9-in-10 day and the step that most often sets the date.
David T. Hulett, "Project Schedule Risk Assessment", Project Management Journal, March 1995 (simulating schedule risk; "merge bias" where paths meet). Copy hosted by Western Carolina University, read Nov 1, 2026. wcu.edu/pmi/1996/J95MAR21.PDF
Which AI is the smartest?Which model gets the job, and what does it cost?
On everyday work, the cheap models were enough. GPT-6 Luna did all 25 tasks for $0.0066. Claude Fable 5.1 also did all 25, for $0.98. Same result, about 150 times the price.
The most expensive answer is not always the best one. Sometimes it is just the most expensive one.
What it is
People usually pick an AI model from leaderboards and gut feel. This build asks a smaller, more useful question. For one kind of job, which is the cheapest model that does it well enough? And where does paying more actually buy you something?
So on Nov 1, 2026, nine models did the same 25 tasks. They cover the market:
three big closed models: Claude Fable 5.1 (the priciest here), Claude Opus 5.5 and GPT-6.1 Sol,
two small closed models: Claude Haiku 5.5 and GPT-6 Luna,
one open model from the US: gpt-oss-120b,
and three open models from Chinese labs: Kimi K3, Qwen 3.8 Max and DeepSeek V4.1 Flash.
“Open” here means anyone can download the model and run it. Google’s Gemini, xAI’s Grok, DeepSeek’s bigger model and GLM-5.3 were on the list, but none could be reached from this setup that day. That is a gap, not a verdict.
The 25 tasks
Ordinary work, and almost none of it customer support:
Code (5): fix a buggy function, write a small parser, fix a wrong SQL report, write a strict pattern for IP addresses, and a harder “longest streak” query.
Pulling data out (3): an invoice into clean fields, messy meeting notes into who does what by when, and a messy sales file into correct totals.
Reasoning and maths (7): a logic puzzle, a multi-step rate problem, software business maths, a riddle with its usual trick removed, a counting problem, a five-person liars-and-truth-tellers puzzle, and a return on investment worked out by hand.
Writing (5): a one-page product spec from a messy chat, a build-or-buy memo, a firm but kind “no” email (the only customer-facing task), an idea explained to a 12-year-old, and a German translation.
Research and honesty (5): a summary with exact quotes from a Wikipedia article,6 a question about a paper that doesn’t exist, a question built on a false claim, and two long logs where the answer needs a few facts put together.
Five of these were added after a first pass, as harder “stretch” tasks to separate the models. They didn’t.
What the numbers say
Six of the nine models passed all 25 tasks on the first try. Their bills for the same work ran from under a cent to just under a dollar.
Model
Passed
Reruns passed
Per 1,000 jobs
All 25 jobs
Middle time
GPT-6 Luna
25 of 25
17 of 18
$0.26
$0.0066
2.3s
gpt-oss-120b
21 of 25
9 of 18
$0.49
$0.0123
1.9s
Claude Haiku 5.5
25 of 25
17 of 18
$0.54
$0.0136
2.4s
DeepSeek V4.1 Flash
25 of 25
18 of 18
$1.79
$0.0447
5.1s
GPT-6.1 Sol
25 of 25
18 of 18
$4.55
$0.11
3.0s
Qwen 3.8 Max
25 of 25
18 of 18
$9.30
$0.23
4.2s
Kimi K3
24 of 25
16 of 18
$10.74
$0.27
5.6s
Claude Opus 5.5
24 of 25
15 of 18
$18.03
$0.45
3.6s
Claude Fable 5.1
25 of 25
17 of 18
$39.25
$0.98
3.7s
Cheapest first. “Per 1,000 jobs” is the average cost of one of these tasks, times 1,000. “Middle time” is the median: half the answers came back faster. Reruns: the six shakiest tasks, three runs each, so 18 in all.
Three things worth knowing
The cheap ones were enough. GPT-6 Luna and Claude Haiku 5.5 passed everything. The whole set cost $0.0066 and $0.0136.
Run it again and some slip. On the six shakiest tasks, only three models passed every rerun: GPT-6.1 Sol, Qwen 3.8 Max and DeepSeek V4.1 Flash. All three sit in the middle on price. The priciest model wasn’t one of them.
The tasks were too easy to tell the models apart. So the price did. A harder set would spread them out, and could change the order.
What’s worth shipping, and why
Pick the model by the job, not by the brand.
For everyday work like this, start with the cheapest model that passes your own checks for that job. Here that is GPT-6 Luna or Claude Haiku 5.5. Right after it, put a simple check in code: is the format right, is it under the word limit, do the quotes really exist, do the tests pass. Only when that check fails, try again on a mid-priced model.
For anything a customer reads, or that can’t be undone, use one of the steadiest models from the reruns (GPT-6.1 Sol was the steadiest closed one here) and a person.
Pay top prices only where your own tests show a difference. On this set, that was nowhere. And rerun the tests every month, because prices and models change faster than opinions do.
Surprises
The stretch tasks didn’t stretch anyone. All nine passed all five harder tasks. That includes a long log of 20,000 to 33,000 tokens where the answer needed a reset, a “+3 to every code” change, a swap, and ignoring an old number on a whiteboard. On this kind of work, today’s models have hit the ceiling, at least with one run each.
The failures were about following instructions, not about brains. There were 6 failures in 225 first answers. Claude Opus 5.5 wrote a strong spec in 552 words when the limit was 500 (and went over in all three runs). Kimi K3 explained an idea in 91 words when asked for 80. gpt-oss-120b turned “exact” quotes into paraphrases, put two dates from meeting notes on the wrong days (its “Friday” was a Saturday), wrote a spec far over the 500-word limit, and got the shorter log wrong (4471 instead of 4474), though it got that one right in both reruns.
Same price per word, double the bill. Claude Haiku 5.5 and GPT-6 Luna charge the same per token. Haiku wrote about 2.5 times as much, so it cost about twice as much. Qwen 3.8 Max charges 2.5 times less per token written than Kimi K3, but it wrote 2.3 times as much, so its bill was only about 13% lower. The price list doesn’t tell you the bill. How much a model writes does.
Nobody fell for the honesty traps. All nine refused to summarise a paper that was made up, and all nine corrected the claim that you can see the Great Wall from the Moon with the naked eye. One small wrinkle: gpt-oss-120b said the paper wasn’t “in the ICML proceedings, arXiv”, as if it had searched them. It can’t.
Speed is mostly about where a model runs. The fastest middle time was gpt-oss-120b (1.9 seconds). The slowest were Kimi K3 (5.6) and DeepSeek V4.1 Flash (5.1), all on the same outside host. The slowest single answer took 45 seconds (Kimi K3, on the counting problem). In a chat box that matters. For a job that runs overnight, it doesn’t.
Limits, said plainly
One run, one day. Each model did each task once (six tasks three times). Small gaps in “score” are noise. Read the pass counts and the reruns, not the decimals.
Low thinking settings for everyone. This is the cheap default most people try first. Higher settings could change the hard tasks, and would raise every bill.
Tasks written for this page. 25 tasks with known answers, not thousands. They test whether a model does a specific everyday job, not how clever it is in general. A task every model passes tells you the job is easy for today’s models, not that the models are equal.
AI judges on 8 tasks. Judges miscount words, so word limits were counted by code instead. A first round of judging that asked judges to count was thrown away and redone.
A bug in the test’s own code, caught. At first, the code that collected answers read some of DeepSeek’s text in the wrong encoding, so its German came out as “März” and the judges failed it. The model’s answer was fine. The reading was fixed, those answers were graded again, and DeepSeek passed all its reruns.
Missing models. No Google or xAI model. DeepSeek ran as its smaller Flash model on an outside host, not on its own service.
Model cost only. The cost leaves out discounts, engineering time, retries and the minutes a person spends checking the answer. At these prices, that last one is usually the bigger number.
No person has graded these yet. The AI assistant that built this re-read 12 answers. It agreed with the official grade on 10. It would have failed 2 that the judges passed: Kimi K3’s build-or-buy memo (it doubled the team’s weekly hours and said building would fit the deadline) and GPT-6 Luna’s explanation for a 12-year-old (accurate, but no real everyday example). Both keep the judges’ grade, and both are shown in the interactive.
What’s next
Next: a hand check of the 12 checked answers, and any others, with a pass or fail recorded for each.
Harder, longer work where models still differ: multi-step tasks with tools, a real codebase, a pile of documents too big to read in one go.
Every task three times, and low against high thinking settings, to see what extra thinking buys and what it costs.
Gemini and Grok when they can be reached, and DeepSeek’s bigger model on its own service.
A small image and video round, skipped this time to keep costs down.
Anthropic, “Pricing”, per million tokens in / out: Claude Fable 5.1 $10 / $50; Claude Opus 5.5 $4 / $20; Claude Sonnet 5.5 $2 / $10; Claude Haiku 5.5 $0.10 / $0.50. Read Nov 1, 2026. docs.claude.com/en/docs/about-claude/pricing
OpenAI, “Pricing”, standard tier: GPT-6.1 Sol $2 / $10; GPT-6 Luna $0.10 / $0.50. Read Nov 1, 2026. platform.openai.com/docs/pricing
Lianmin Zheng and others, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, arXiv:2306.05685, on the ways AI judges can be biased, including towards their own answers. arxiv.org/abs/2306.05685
How big is the market?Could this market ever hold a trillion-dollar company?
Work backwards from $1 trillion and you get the market you'd need. Amazon, Nvidia and Google couldn't have got there inside their markets as sized when they went public. The market that made them came later.
Spreadsheets have been optimistic about less.
What it is
A calculator for a question founders answer too fast. It works in three steps, and every input is a slider with a note on where its number came from.
Work backwards from the value. Value = yearly profit × an earnings multiple. Revenue = profit ÷ profit margin. The market you need = revenue ÷ the share you can win. With the defaults: to be worth $1T at 25 times earnings, a company needs $40B of profit a year. At a 20% margin that's $200B of revenue. If that's 30% of its market, the market has to be about $667B a year. The four real companies below use their own multiple and margin instead, so their numbers differ.
Size the market two ways. Top-down starts from someone's estimate of the whole market. Bottom-up starts from one customer: customers × price × purchases a year. The two almost never agree, and the gap tells you which number to go and check.
How long would it take? Revenue today, a growth rate, and how fast growth fades. It says how many years until revenue crosses the line, or that it never does.
Four real examples load straight in: Amazon, Nvidia, Google and Facebook, each as it looked in its own IPO filing. A fifth example, AI agents for work, is an illustration, not a prediction.
Last column: $1T ÷ (the company's own multiple × its own margin today). Amazon 33.4 and 10.8%, Nvidia 41.2 and 55.6%, Alphabet 25.9 and 32.8%, Meta 26.0 and 30.1%.
Amazon
The filing described a world book market of about $82B. At Amazon's margin and multiple today, $1T needs about $276B of revenue a year: more than three times all the books sold on Earth in 1996. The words "cloud" and "web services" don't appear in the filing at all. AWS alone brought in $128.7B in 20255.
Nvidia
The filing counted chips, not dollars: 4.9 million 3D chips sold in 1997, 138 million expected in 20012. To reach about $44B a year from that, Nvidia would have had to sell every chip forecast for 2001 at about $320 each. "Data center" appears zero times. In fiscal 2026, Data Center made $193.7B, 90% of the total6.
Google
The filing didn't size its market. It called online ads "an immature industry"3. All U.S. online ads were about $7.3B in 200312 and $294.6B in 202513, about 40 times bigger. Google didn't just take a share of the market. It was a big part of why the market grew.
Facebook, the exception
Its filing pointed at $588B of ad spending4, more than four times the $128B a year it needs today. The market was always big enough. The question was whether one company could take about a fifth of it.
A candidate today: AI agents for work
An illustration, not a prediction or investment advice. At software's average margin (25.5%)11 and the S&P 500's multiple on Oct 30 (25.86, rounded to 25.9 here)10, $1T needs about $151B of revenue a year: a $505B market at a 30% share. Gartner puts the "AI agents and assistants" category at $29.2B in 202615. A seat count (250 million seats, an assumption, × the $30 a month Microsoft charges for Copilot16) gives $90B. All software spending is $1,468B14.
So $505B is about 17 times the category, 5.6 times the seat count, and about a third of all software. As a category, no. Only if agents take budget from somewhere much bigger, like other software, outsourced services, or work people are paid to do now. That's the claim that would need evidence.
What's worth shipping, and why
Use this as the first ten minutes of looking at any idea, including your own. Not to get a number, but to find the one assumption the whole thing rests on. When the top-down and the bottom-up disagree by ten times, go and talk to customers before believing either. The worked examples are the reminder: the market in the filing is rarely the market that matters.
Surprises
The obvious market was too small in three of four cases. Amazon and Nvidia couldn't have reached $1T inside the markets their filings described (Nvidia only at a guessed chip price), and Google couldn't have inside the 2003 U.S. ad market, at any share.
The filings didn't name the markets that made them. No "cloud" in Amazon's. No "data center" in Nvidia's. Those are now $128.7B and $193.7B a year56.
Google refusing to size its market was the honest call. The market it was in grew about 40 times by 20251213.
A correction inside a filing. Google's filing reprinted an interview saying 65 million people used it each day, then corrected it in the same filing: that was monthly U.S. visitors, not daily users3.
Limits, honestly
Survivors only. 15 companies were worth $1T or more on Oct 30, 20269. Many more told the SEC their markets were huge and never came close. Since 1926, the best 4% of U.S. listed companies explain the whole net gain of the stock market17. Four winners show what winning looked like, not how often it happens.
A price isn't value. A company's value is set daily by people guessing the future. Profit × multiple is a simplification, and the multiple moves with interest rates and mood.
Snapshots. The company multiples divide one day's value by one year's profit. They'll be out of date by the time you read this. The S&P 500's own multiple was already about 25 by Nov 6.
No inflation adjustment. 1996 and 2025 dollars are compared as written.
Mixed definitions. Google's bottom-up mixes visitor counts with ad revenue, so treat it as a rough size. In the interactive's Nvidia market figure, the $25 price per chip is an assumption; its filing gives only units. (The $320 above is a different thing: what each chip would have needed to earn.)
Both methods mislead, in different ways. Top-down borrows someone else's idea of the market. Bottom-up multiplies three guesses, so the errors stack.
Not investment advice. Nothing here is a forecast for any company or stock.
What's next
Add the companies that went public before EDGAR, like Microsoft in 1986 and Apple in 1980, from scanned filings.
Let the market grow too, so top-down can change over time.
A shareable link that carries your inputs, so you can send a sized idea to someone and argue about the gap.
Amazon.com, Inc., IPO prospectus (Form 424B1), filed May 15, 1997, SEC EDGAR. Net sales 1996 $15,746K, net loss $5,777K; ~340,000 customer accounts and >$32M sales through Mar 31, 1997; Euromonitor: U.S. book sales ~$26B and worldwide ~$82B in 1996; IDC: Web commerce $5.4B annualized Dec 1996, $95B forecast for 2000; IPO price $18.00; 23,858,702 shares after the offering. sec.gov/Archives/edgar/data/1018724/000089102097000868/00…
NVIDIA Corporation, IPO prospectus (Form 424B4), filed Jan 22, 1999, SEC EDGAR. Total revenue 1997 $29,071K, net loss $3,589K; nine months to Oct 25, 1998 revenue $92,700K; Mercury Research: 4.9M 3D graphics processors sold worldwide in 1997, 138M forecast for 2001; IPO price $12.00; 28,595,976 shares after the offering. sec.gov/Archives/edgar/data/1045810/000101287099000192/00…
Google Inc., IPO prospectus (Form 424B4), filed Aug 19, 2004, SEC EDGAR. Revenues 2003 $1,465,934K, net income $105,648K; U.S. 71% of 2003 revenues (advertiser billing address); 65 million = monthly U.S. visitors (the prospectus's correction of a magazine's 'daily' figure); 'online advertising, which is an immature industry'; IPO price $85.00; 271,219,643 shares after the offering. sec.gov/Archives/edgar/data/1288776/000119312504143377/d4…
Facebook, Inc., IPO prospectus (Form 424B4), filed May 18, 2012, SEC EDGAR. Revenue 2011 $3,711M (advertising $3,154M), net income $1,000M; 901M monthly active users at Mar 31, 2012; worldwide ARPU $1.21 in Q1 2012; IDC: worldwide advertising spending $588B in 2010; IPO price $38.00; 2,138,085,037 Class A + B shares after the offering. sec.gov/Archives/edgar/data/1326801/000119312512240111/d2…
NVIDIA Corporation, Form 10-K for fiscal 2026 (year ended Jan 25, 2026), filed Feb 25, 2026. Revenue $215,938M, of which Data Center $193,737M and Gaming $16,042M; net income $120,067M; fiscal 2024 revenue $60,922M. sec.gov/Archives/edgar/data/1045810/000104581026000021/nv…
Meta Platforms, Inc., Form 10-K for fiscal 2025, filed Jan 29, 2026. Revenue $200,966M, advertising $196,175M; net income $60,458M; revenue 2023 $134,902M; 3.58B daily active people (Dec 2025). sec.gov/Archives/edgar/data/1326801/000162828026003942/me…
companiesmarketcap.com, 'Largest companies by market capitalization', read Nov 1, 2026 (prices as of the Oct 30, 2026 close). NVIDIA $4.941T, Alphabet $3.420T, Amazon $2.598T, Meta $1.572T; 15 companies listed at $1T or more. companiesmarketcap.com/
multpl.com, 'S&P 500 PE Ratio' (trailing 12-month as-reported earnings; historical data from Robert Shiller). Current 25.86 at the Oct 30, 2026 close; long-run mean 16.24, median 15.08. Read Nov 1, 2026. multpl.com/s-p-500-pe-ratio
Aswath Damodaran, NYU Stern, 'Operating and Net Margins by Sector (US)', data as of January 2026. Net margin: total market 9.74%; software (system & application) 25.49%; semiconductor 30.45%; retail (general) 5.61%. pages.stern.nyu.edu/~adamodar/New_Home_Page/datafile/marg…
Gartner, 'Gartner Forecasts Worldwide AI Spending to Grow 49.5% in 2026' (press release, Sep 16, 2026): 'AI Agents and Assistants' $29,219M in 2026 and $65,472M in 2027; AI software $461,637M in 2026; total AI spending $2,670,460M in 2026. gartner.com/en/newsroom/press-releases/2026-09-16-gartner…
Microsoft, 'Copilot plans and pricing: enterprise': Microsoft 365 Copilot add-on $30.00 per user per month, paid yearly. Read Nov 1, 2026. microsoft.com/en-us/copilot/pricing/enterprise
Hendrik Bessembinder, 'Do stocks outperform Treasury bills?', Journal of Financial Economics 129(3), 2018: the best-performing 4% of listed U.S. companies explain the net gain of the entire U.S. stock market since 1926. ideas.repec.org/a/eee/jfinec/v129y2018i3p440-457.html
Will the agent deflect tickets?What happens to the queue when the agent hands off badly?
A dashboard said 65% of tickets were “deflected”, meaning no person touched them. Only 43% were actually solved. The team's workload fell just 16%, and 44% of customers had to explain their problem twice. The same agent with good handoffs saved more than twice as much work.
What it is
A support floor you can run in your browser. Customers arrive at random. An AI agent meets them at the door and does one of four things:
solves the problem;
says it solved the problem, and didn't;
hands the customer to a person, with or without the story so far;
goes round in circles until the customer gives up and comes back later asking for a person.
Behind it sits a team of people with one shared line. You set the traffic, how the agent behaves, how much of the story survives a handoff, and the size of the team. Every time you move something, the page plays out 20 working days, about 6,000 customers. It shows the queue from both sides: how long people wait for a person, how many minutes of human work an hour it takes, how many customers had to tell their story twice, and how much of the “deflection” came back through the side door.
A standard textbook formula for queues checks the simulation on every run. No AI model is called. It is all arithmetic.
The agent never gets tired. The customers who meet it twice do.
What the numbers say
With bad handoffs and the default settings:
A dashboard that counts “no person touched it” would report 65% deflected. Only 43% of customers were really solved by the agent. 35% of the “deflected” tickets came back.
The team still needed 309 minutes of human work an hour, down only 16% from 366 with no agent at all. If the dashboard were right (every “deflected” ticket truly solved, every handoff arriving with the full story), it would be 143.
44% of all customers had to explain their problem more than once.
The average wait for a person was 9.4 minutes, and 1 customer in 10 waited 26 minutes or more. Turn on a 4 pm rush (twice the customers from 4 to 5) and that becomes 51 minutes on average, and over 2 hours for the unluckiest 1 in 10.
The same agent, handing off well (same team): 232 minutes of human work an hour, 37% less than no agent. 12% of customers repeated themselves. Average wait 1.4 minutes, 4.8 for the unluckiest 1 in 10; with the rush, 5.0 and 18.
How much is luck. Across 20 different random seeds, the bad-handoff average wait ranged from 4.4 to 14.9 minutes (middle value 7.9). The good-handoff one ranged from 0.7 to 1.6. Workload and repeats barely moved (bad: 285 to 318 minutes an hour, 42% to 45% repeating). Waits are jumpy because queues close to full are jumpy. That is the point.
What's worth shipping, and why
Ship the agent that deflects less. Not as a slogan: here the good-handoff agent's dashboard number is lower (54% “deflected” against 65%), and it saves the team more than twice as much work (134 minutes of human work an hour against 57).
Launch it on one queue, staff for the 4 pm rush instead of the daily average, and put three numbers on the wall from day one: how long the unluckiest 1 in 10 waits for a person, the share of “solved” tickets reopened within a week, and the share of customers who had to tell their story twice. If the reopen rate climbs past the line agreed before launch, the agent goes back to writing replies for people to send until it's fixed.
Surprises
Going in circles flatters the dashboard. Customers the agent neither solves nor hands off count as “deflected”, then come back later, angrier, with nothing written down. Each one costs about 15 minutes of human time, against 11.6 for a handoff. Turn the default's 15% of circling into plain handoffs and the dashboard's deflection drops from 65% to 50%, while human work falls from 309 to 287 minutes an hour. Give those handoffs the full story too and it falls to 244, with a 2.2-minute average wait. The better system has the worse-looking number.
False “all sorted” is a cliff, not a slope. Keeping everything else at the bad-handoff default, raising false claims from 15% to 30% of the agent's “solved” took the average wait from 9.4 to 40.7 minutes. Dropping them to 5% brought it to 4.5. Each false “all sorted” comes back, onto a team that is already nearly full, where every extra bit of load costs more than the last4.
Passing on the story is cheap and adds up. Raising how often the story survives a handoff from 40% to 100%, and nothing else, halved the average wait (9.4 to 4.3 minutes) and cut customers repeating themselves from 44% to 23%. The 23% left are the circling and the false “solved”, which no handoff note can rescue.
A team sized for the dashboard drowns. The dashboard says about 2.4 people would do. With 5 people instead of 6, the bad-handoff line never clears: the team needs about 309 minutes of work an hour and has 300. The good-handoff agent with the same 5 people averages a 4.9-minute wait.
Limits, honestly
Every agent behaviour, time and return delay is a made-up guess, chosen to be plausible. No real customer data was used.
People are modelled as solving everything they touch, first time. Real teams don't, so real numbers are worse.
Nobody leaves for good. In real life some customers give up and vanish into churn, which this model can't see.
How long people take to help someone is modelled the textbook way. Real handling times usually have a longer tail.
“Customer time” counts minutes with the agent, waiting, and with a person. It doesn't count the hour between a false “all sorted” and finding out.
The textbook formula assumes steady arrivals, so it ignores the 4 pm rush by design; the interactive says so when the rush is on.
This is a model to argue with, not a forecast. The direction (handoff quality matters more than the deflection rate) held in every combination tried. The exact numbers did not.
What's next
Swap the made-up agent numbers for published or measured ranges where they exist, and show how far the conclusions move.
Add customers who give up for good.
Let visitors save a setting as a link, so an argument about a queue can start from the same numbers.
The simulation itself: in-browser code with seed 455, 20 days, day 1 dropped, 6,165 customers; every number on this page was re-run from it on Nov 6, 2026. Open the interactive
John D. C. Little, “Little's Law as Viewed on Its 50th Anniversary,” Operations Research 59(3), 2011, pp. 536–549 (the average number in a queue equals arrival rate times average time spent). people.cs.umass.edu (read Nov 1, 2026)
Erlang C in plain words: the chance a customer has to wait, and the average wait, when several identical people serve one line with random arrivals. Worked through in the next source, chapter 5.
Ivo Adan and Jacques Resing, “Queueing Systems,” lecture notes, Eindhoven University of Technology, March 26, 2015: chapter 5 (the M/M/c queue: Erlang C, mean wait, and the waiting-time distribution used for the 1-in-10 check); Little's law, section 3.4. win.tue.nl (read Nov 1, 2026)
Can AI answer my email?What should never be sent without a person reading it?
Writing the replies was mostly fine. Sending them was the risk. In the first version, more than a third of the AI's replies said something had been done that nobody had done. One line in the instructions fixed most of it, and eight plain rules now decide what waits for a person.
What it is
A follow-up machine for a company that doesn't exist. Cellarbook is made up: forty people, software for small breweries, a handbook of prices and rules, and a bad week with two open bugs. Everyone in it is made up too.
On top of it sit four everyday jobs: sort an inbox and Slack and write replies, turn a meeting into a follow-up with owners, look into a bug from its logs, and write a short research note with sources.
Between every reply and the send button sit eight plain rules for “never send without a person”. Each rule is checked two ways, by a simple word search and by a second, cheaper AI model, and the reason is shown whenever a reply is held. A third model, from a different company, grades every reply. In the interactive you can open any of the 69 messages and see what came in, what the AI wrote, which rule stopped it, and what the judge said.
A brewery software company with a bad week. Nobody was harmed; the beer is also made up.
What the numbers say
The 69 messages had traps written in on purpose: an angry customer in tax week, a refund 45 days into a yearly plan, a request to delete data, a lawyer asking about liability, a pasted password, an API key in a chat thread, a date nobody has. 45 of them carry at least one trap. 44 of those got a reply; for the 45th, a call recap with no notes, the AI correctly wrote nothing.
The final version (the third set of instructions) wrote 61 replies; 8 messages needed none.
The rules held 42 of the 44 replies to a trap. The two that got through were Slack questions about the date for an unreleased feature, where the reply correctly said “no date yet”. The judge would have sent both as written.
They also held 6 of the 17 replies to messages with no trap. Three left a gap marked “[CHECK: …]” for a fact the handbook doesn't have. Two said “AWS” when the handbook only names the server region. One was in a thread with a furious customer, which the rules hold on purpose. 42 + 6 = 48 held in all.
Replies that claimed an action nobody took (“I've passed this to legal”, “I'll update you tomorrow”): 23 of 62 in the first version, 1 of 61 after one change to the instructions.
The judge rated all 69 messages, counting the ones that rightly got no reply: 28 send as is, 29 light edit, 8 heavy edit, 4 never send. The first version got 22, 38, 9 and 0.
Cost: about $0.03 a message for writing, checking and grading, worked out from token counts at list prices1,2 (an estimate, not a bill). The whole night, including making the data and three full runs, came to roughly $9 at those prices.
What's worth shipping, and why
Let the AI write for everyone and send for almost no one.
Every message gets a reply written, because writing is where the time goes, and here the AI was mostly up to it: the judge said 57 of the 69 needed a light edit or nothing. But anything the rules flag waits for a person. Here they held 48 of 69, partly because this test week was built to be full of money, dates and upset people. A real week is probably calmer, and that is the first number to measure.
What sends on its own is the boring rest: “yes, QuickBooks is included on your plan”, “here are our support hours”. Keep the checker failing safe (no answer means hold), and stop letting the AI's own opinion of its risk decide anything.
Surprises
The worst failure was a fake action. More than a third of the first replies (23 of 62) claimed something had been done, or promised something nobody had agreed to. They read beautifully and were quietly wrong about the world. One line in the instructions fixed most of it.
Asking it to be careful made it useless in a new way. Told not to guess, it left “[CHECK: …]” gaps in easy answers, and every clean reply got held. Careful and helpful had to be balanced, not maxed out separately.
The AI's opinion of its own risk is a bad gate. Asked which rules its reply touched, it flagged at least one on 14 of the 17 replies that had no trap. Like a nervous new hire, it flagged everything.
The word search read the wrong side of the conversation. At first it also searched the customer's message, so a customer saying “Friday” held the reply. Anger and pasted passwords live in the incoming message; promises live in the reply.
The judge was wrong too. It caught real problems, and it also marked correct dates as unsupported.
The fastest new hire anyone has ever had, and the most confident about things that didn't happen.
Limits, honestly
Everything is made up: the company, the people, the messages and the bugs. The research sources are real public pages. No real work data was used.
The traps, and the labels for which replies should be held, were written by the same model that made the data. A starting point, not an answer key.
One judge, one run per version. Judges vary from run to run; treat differences of a few messages as noise.
39 Slack messages, not 40: the generator came back one short, and what it wrote was kept.
In the first two versions a few checker calls (4, then 3) returned nothing readable and counted as “no flags”, which flatters those runs slightly. The final version had none, and now an unreadable answer means hold.
The fake passwords and the fake API key are hidden in the public files.
The eight rules were written with an AI assistant for this test.
What's next
Grade the 69 replies by hand, check the 45 traps again, and publish how often a person agrees with the judge.
Grade every build on this site with one judge, so the numbers line up.
Run the rules on a calmer, more ordinary week. How often they hold a normal message decides whether a team would keep them on.
Anthropic, “Pricing” (list prices used for the cost estimate of claude-sonnet-5-5 and claude-haiku-5-5). docs.claude.com (read Nov 1, 2026)
OpenAI, “Pricing” (list price used for the judge's cost estimate). platform.openai.com (read Nov 1, 2026)
Brewers Association, “The 2025 Year in Beer” (December 2025; 9,778 breweries). brewersassociation.org (read Nov 1, 2026)
Brewers Association, “A Year of Correction for Craft Beer, With Early Signals of Recovery” (April 14, 2026, updated May 29, 2026; 9,578 breweries in the final data). brewersassociation.org (read Nov 1, 2026)
Brewers Association, “2026 Midyear Report: Pressure Persists as Signs of Stability Emerge”. brewersassociation.org (read Nov 1, 2026)
Brewers Association, “National Beer Sales & Production Data”. brewersassociation.org (read Nov 1, 2026)
The run itself: 69 messages, three versions, every reply, rule flag and grade, in the interactive's data (Nov 1, 2026). Open the interactive
Which model is best?Which model can be trusted with a real support queue?
The small, cheap model. It solved about as many tickets as the big one (40 against 41 out of 150), sent about half as many bad answers (15 against 29), and cost about 1/38th as much. Also, the answer key that came with the tickets was wrong.
What it is
A decision, not a leaderboard. Four AI models read the same 150 support tickets. The tickets come from a public practice dataset and are made up1.
For every ticket, each model had to do five things: say what the customer wants, say how urgent it is, decide whether to answer now or hand the ticket to a person, say how sure it was, and write the reply. The one rule: don't make anything up. No invented policies, refunds, dates or promises.
A fifth model, from a different company, graded every answer against a written checklist. The interactive turns that into what a support lead actually needs: cost against quality, a confidence slider, every ticket and answer to read, and a mode where a person can grade the answers and overrule the judge.
It is a job interview where all four candidates answered the same 150 emails.
What the numbers say
Model
Solved right, no person
Bad answers sent
Handed to a person
Cost per 1,000 tickets
Claude Opus 5.5
41 (27%)
29 (19%)
80 (53%)
$8.27
Claude Haiku 5.5
40 (27%)
15 (10%)
95 (63%)
$0.22
Kimi K3
22 (15%)
14 (9%)
114 (76%)
$4.09
DeepSeek V4.1 Flash
30 (20%)
14 (9%)
106 (71%)
$0.34
Out of 150 tickets each. Cost is the model's bill only: the tokens it really used times its maker's list price on Nov 1, 20262,3. A person's time is not included.
Haiku and Opus solved almost the same number of tickets. Haiku sent half as many bad answers, and its bill was $0.22 for every 1,000 tickets against $8.27, about 1/38th.
To check the judge, 25 tickets (100 answers) were graded a second time. It gave the same pass or fail on 96% of them. The file from that second grading wasn't kept, so treat 96% as unchecked. Even if right, it means the judge is consistent. It does not mean it is right: no person has graded these answers yet.
What's worth shipping, and why
The small model, not the big one, with a narrow job: answer only what it can answer without promising anything.
Anything about money, security, lost data, an outage, legal or HR goes to a person, however sure the model says it is. Rewrite the hand-off message so it never promises a speed the model can't see. And recheck it every month against a set of answers graded by a person, because the judge is a model too.
Surprises
The answer key was wrong before any model ran. The dataset came with an urgency label for every ticket. The four models agreed with each other on urgency 76% to 91% of the time, and with the labels only 32% to 33%. That is about what guessing between three options gets you. A ticket titled “Unapproved Entry into Patient Files Identified” was labelled low priority and sent to Returns and Exchanges. So the labels were dropped, and the judge's own read was used instead.
Confidence meant nothing. On the tickets it answered, Opus said it was 74% sure, on average, when it passed and 73% sure when it failed. Haiku: 73% and 73%. Raising the bar mostly sends more tickets to people. The share of bad answers barely moves.
The bigger model was the more eager one. Opus answered the most tickets and was caught making something up most often (19 answers). Usually it described the company's products as if it knew them: “our digital strategy services”.
The answer that broke the rules was right. DeepSeek ignored the list of allowed queues on 6 tickets and made up a new one: “Security”. All 6 were reports of a data breach. The list had no right place for a breach.
Even hand-offs make promises. When a model handed a ticket to a person, its message to the customer often promised speed: “shortly”, “soon”, “promptly”, “immediately” or “as a priority”. Counting those five: Opus 18%, DeepSeek 43%, Haiku 65%, Kimi 68% of hand-offs. None of them can see the queue.
“Someone will be with you shortly” is a promise. Nobody checked who.
Limits, honestly
The tickets are made up, vaguer than a real queue, and only in English. That is partly why so many went to a person (53% to 76%).
One judge, one run. A judge from a different company avoids a model favouring its own family. It can still favour longer answers, or the first one it reads4,5. Answers were shown to it blind and shuffled.
The dataset's licence allows a personal, non-commercial site like this one, with credit. Not commercial reuse.
Prices are standard list prices on Nov 1, 2026. Bulk or off-peak discounts would lower every number.
The word search for made-up promises is a smoke alarm, not a judge. Its first version went off on “we have escalated your ticket”, which is true when the model is handing it off.
No real customer data was used.
What's next
Grade the 100 practice tickets by hand, publish how often a person agrees with the judge, and redo every number on this page from those grades.
Run each model three times instead of once, to see how much of “40 against 41” is luck.
Settle the open questions in the checklist (is an unneeded hand-off a pass? is “shortly” a failure?) and grade again.
Tobias Bueck, “customer-support-tickets” dataset card, Hugging Face (licence CC BY-NC 4.0; described as synthetic, no personal data). huggingface.co (read Nov 1, 2026)
Anthropic, “Pricing” (Claude Opus 5.5: $4 / $20 per million input / output tokens; Claude Haiku 5.5: $0.10 / $0.50). docs.claude.com (read Nov 1, 2026)
Fireworks AI, “Serverless pricing” (Kimi K3: $3.00 / $15.00; DeepSeek V4.1 Flash: $0.30 / $1.20 per million tokens). docs.fireworks.ai (read Nov 1, 2026)
Lianmin Zheng and others, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, arXiv:2306.05685 (AI judges show position, length and self-favouring biases). arxiv.org (read Nov 1, 2026)
OpenAI, “Evaluation best practices” (AI judges can favour position and length; check automated grades against people). platform.openai.com (read Nov 1, 2026)
What’s your philosophy?How do you actually decide?
Eight rules, two beliefs, six longer arguments and a few toys. The toys are the fun part. The essays are where the working is shown.
Eight rules, the long version
The front page lists just the titles. Here each one gets its reasons, and a place on the site where you can watch it work.
01
Find the question under the question.
“Can we automate support?” is a fair question. The useful one underneath is “which replies are safe to send without a person?” That one has an answer you can test. Every build on this site starts by crossing out the first question and writing the second.
Before spending a quarter on something, spend an afternoon on a rough model of it: a spreadsheet, a sketch, twenty lines of code. Cheap models break early. That’s their whole job.
A thing that works, even a bit badly, gets better feedback than a perfect slide about it. People nod at slides. They poke at demos, and the poking tells you more.
Work doesn’t arrive evenly. A team sized for the average day still drowns at 4:55 on a Thursday. Plan for the peak and the average takes care of itself.
Most of a customer’s wait is time spent between people, not work. Every handoff adds a queue and loses a bit of the story. Remove a handoff before you speed one up.
One glowing reference is evidence, not a verdict. Start from how often things like this usually work out, then let each signal move you a little. Ten weak signals don’t add up to one strong one.
If only one person knows how something works, that person can’t take a week off, and the customer waits until they’re back. Writing it down is how a team stops depending on one memory.
Read, ask, do it by hand first. Then automate. Building before understanding just makes the wrong thing faster.
There is a ninth, hidden on the front page: never trust a list that says eight.
Two beliefs
Still arguing with each other. One of them will win.
OneThe demo is the cheapest moment in a product’s life. What matters is decided afterwards, by whoever stays to make it work. AI makes demos almost free, so the people who stay matter more, not less.
TwoSoftware rarely fails at the model. It fails at the handoff: between the AI, the customer and the team. The next great companies will be built by people who obsess over those seams.
Essays
The longer arguments. Each one grew out of a build, and links back to it.
Nineteen tiny live models of everyday decisions: churn, hiring, launch dates, the 4:55 rush. Drag a slider and the maths answers. None of them will judge you.
Engineering says done when it merges. The customer says done when they can use it. Support says done when they hear it exists, usually from the customer.
Test it: After the next launch, ask support what shipped. Compare notes.
Why AI pilots stall after the demo. The models are good enough for the demo. The gap is everything around them.
The demo is on a Friday afternoon. Someone types a clean question, the agent gives a clean answer, and the room relaxes. An ordinary weekday is different. It has the customer with two accounts and one password, the refund that crosses a policy line by a dollar, the person who writes four paragraphs before getting to the question, and, around 4:55, everyone at once. The distance between those two days is where most AI pilots go to sit. Not fail, exactly. Sit.
The market has noticed the gap, and it is paying people to close it. Lightcast counted more than 7,300 US job postings for forward deployed engineers (engineers who work inside the customer's business) in the first nine months of 2026, up from fewer than 200 in 2023, as CNBC reported in October. The article is careful to say these are advertised jobs, not hires1. The same piece quotes Val Henderson, CEO of the consulting firm Caylent: “The hardest thing to teach is the customer EQ, without question.”1 Sierra says of its Agent Strategist role: “Last year, this role barely existed. Today, it's our fastest-growing team.”3 And a16z put it in one line back in 2025: “Enterprises buying AI are like your grandma getting an iPhone: they want to use it, but they need you to set it up.”2
When that much money goes looking for people who sit between the model and the customer, the gap isn't mainly a model problem.
Every demo is on a Friday afternoon. Every queue has a 4:55.
The demo measures the agent. The queue measures the system.
A pilot usually reports a number about the agent: how many conversations it solved, or kept away from people. Customers live a number about the system: how long until someone who can actually help is helping them. Those are different numbers, and the queue simulator exists to show how far apart they can drift.
With bad handoffs, the dashboard would say 65% of tickets were “deflected”, never touched by a person. The team's workload fell 16%. Both are true at the same time. The difference is made of three things a demo never shows: tickets the agent marked solved that weren't, customers it neither solved nor passed on, and the minutes people spend getting a story they should have been handed. None of those appear on the agent's scorecard. All of them appear in the queue.
The handoff is a product, and nobody owns it
Inside most rollouts, one team owns the agent and another owns the human queue. The handoff belongs to nobody. It is a line of code and a hope that the note will be useful. But from the customer's chair, the handoff is the product. It is the moment they find out whether the last five minutes counted.
In the simulator, making sure the customer's story survives every handoff, and changing nothing else, halved the average wait and cut the share of customers repeating themselves from 44% to 23%. A better note costs almost nothing to build. It needs an owner, a definition of “good”, and someone who reads a sample every week.
A good note answers four questions, in the order a person needs them: who is this, what are they actually trying to do, what has already been tried, and what have they been told. The last one is the one most often missing. If the agent promised a refund “within the hour”, the person needs to know before they say hello, not after the customer reminds them. A note like that turns a handoff into a relay. Without it, it is a restart, and the customer can tell which one they got within a sentence.
The loop nobody counts
Most scorecards miss a fourth outcome. The agent doesn't solve the problem, doesn't admit it can't, and doesn't hand off. It asks a question, then another, then offers an article. Eventually the customer closes the window. On a dashboard that counts conversations no person touched, that is a deflection. In real life it is a delay: the customer comes back later through another door, starts from zero, and arrives already annoyed.
In the simulator, turning those circling conversations into plain handoffs makes the deflection number fall from 65% to 50%, and the team's workload falls with it. That is the deployment gap in one comparison: the honest agent looks worse and works better. A team rewarded for the first number will, quite reasonably, build the loop.
Queues don't forgive optimism
The third reason pilots stall is arithmetic. Little's Law says the average number of people in a queue equals how fast they arrive times how long each one stays5. It holds for almost any queue, which is why it is useful and why it is unforgiving: every “deflected” ticket that comes back is extra traffic nobody planned for, and it turns straight into a longer line.
The textbook formula for a team of people sharing one line makes it sharper. The average wait grows with 1 ÷ (1 − how busy each person is)4. At 80% busy that factor is 5. At 90% it is 10. Teams sized for the dashboard's promise run hot, and a hot queue turns small mistakes into big waits. That is why false “all sorted” claims behave like a cliff in the simulator: doubling them, from 15% to 30% of the agent's “solved”, more than quadrupled the average wait. John Little, looking back at his own law fifty years later, wrote: “It just says that we are in the measurement business, not the forecasting business.”5 A pilot that measures the agent instead of the queue is doing neither.
Getting past the demo
None of this needs a better model. It needs a short list, kept honestly. This one will do:
Measure the queue, not just the agent. Before launch, write down how long the unluckiest 1 in 10 waits for a person, and how many minutes of human work an hour the queue takes. Compare after. The average hides the people who wait longest.
Count the comebacks. For every ticket the agent marks solved, check whether the same customer comes back within a week. That is the deflection that isn't.
Treat the handoff as a product. Define the note a person can read in thirty seconds. Test it like a feature. Read twenty of them every week.
End the loop. Every conversation ends in a solution or a handoff, never a shrug. Put a limit on how long the agent can go round in circles.
Staff for the rush. Use the real arrival pattern, not the daily average, and not the dashboard's promise.
Launch on a slice, with a way back. One queue or one part of the day first, and a switch that sends everything to people without a meeting.
Decide “good enough” before you look. Sierra describes part of the job as “knowing when an agent is good enough to launch so teams can get real feedback quickly.”3 Write down the bar for launching and the bar for rolling back before the numbers come in.
Name an owner for the ordinary day. One person who reads conversations, watches the comeback rate and can change the handoff. Without that person, the pilot is a demo with a longer run.
The demo answers “can it?” An ordinary day answers “does it, for everyone, at 4:55?” The second question is less exciting and far more valuable, and it is where the work actually is.
It is also the part of AI work that looks most like the oldest part of customer work: listening to what actually happened, writing it down so the next person doesn't have to ask again, and caring whether the person on the other end got what they came for. The models will keep getting better at the demo. The ordinary day is still ours to get right.
Sources (5)
Barbara Booth, “AWS, Anthropic race to deploy army of specialized engineers key to proving AI spending will pay off,” CNBC, Oct 15, 2026 (Lightcast posting figures; Val Henderson quote). cnbc.com (read Nov 1, 2026)
Joe Schmidt, “Trading Margin for Moat: Why the Forward Deployed Engineer Is the Hottest Job in Startups,” a16z, June 4, 2025. a16z.com (read Nov 6, 2026)
Andrew Kim, “Agent Strategist: Your PhD in applied AI,” Sierra blog, May 19, 2026. sierra.ai (read Nov 6, 2026)
Ivo Adan and Jacques Resing, “Queueing Systems,” lecture notes, Eindhoven University of Technology, March 26, 2015, chapter 5 (the Erlang C formula and the average wait for several people sharing one line). win.tue.nl (read Nov 1, 2026)
John D. C. Little, “Little's Law as Viewed on Its 50th Anniversary,” Operations Research 59(3), 2011, pp. 536–549. people.cs.umass.edu (read Nov 1, 2026)
An AI product can do a lot. A customer gets only what survives the rollout, the handoffs and the first wrong answer. The gap between the two is the job.
More than a decade in customer success and product, across the whole customer journey: implementation, onboarding, success, support, services, renewals and retention, running teams, and building with product and engineering.
That covers every side of the handoff: the kickoff call, the escalation, the renewal talk, and the planning session where a customer’s problem becomes a feature.
Seen end to end, most customer problems don’t start where they show up. A cancellation in month eleven often began at a handoff in week two. So this page follows the journey, not the org chart.
The journey, stage by stage
For each stage: where it usually breaks, what fixes it, and where product or AI changes the answer.
The trouble sits in the gaps. The last stage leads back to the first.
Signed
Breaks: The promise from the sales call lives in one person’s head.
Fix: A one-page handoff before kickoff: what was bought, what success means, and the date the customer heard.
Product and AI: AI drafts the note from the call. A person signs it off.
Onboarded
Breaks: In the waiting between steps, not in the steps. Every handoff costs days nobody planned.
Fix: Fewer handoffs, each with an owner on both sides. The date goes out as a range, not a promise.
Product and AI: Simulate the plan before promising it. Run 4,000 times, an invented rollout hit its own planned date 1.3% of the time.
Breaks: The project closes at switch-on, and nobody checks who actually uses it.
Fix: Aim at first value, the day half the users have done the real job, and keep the onboarding owner until then.
Product and AI: A setup step every customer gets wrong is a product fix, not another training session.
Supported
Breaks: At the handoff from bot to person. The customer explains it twice, and the dashboard still counts it as deflected.
Fix: Pass the whole story with the ticket. Count problems solved, not tickets touched.
Product and AI: Test models on the tickets customers actually send. On 150 practice tickets, the small, cheap model solved about as many as the big one, with about half the bad answers.
Breaks: The renewal is the first honest conversation of the year. By then the decision is made.
Fix: A quarterly check on the result the customer bought, so risk shows up months before the renewal date.
Product and AI: AI reads every ticket and call note for early warning signs. A person makes the call.
Expanded
Breaks: Expansion turns into a sales push at an account that never got the first result.
Fix: Earn the second purchase with the first. Grow the healthy accounts, and bring the rest back to first value.
Product and AI: What the biggest accounts keep asking for is often the next feature. Telling them when it ships is the best expansion call.
First 90 days
Five moves, in order, for one team spanning customer success, product and AI.
Map every handoff from signed to live. One name on each side of each one. A handoff with no owner is where the date slips.
Keep one list of what customers ask for, ranked with product. Each request carries the accounts and revenue behind it, so the loudest customer is one vote, not the roadmap.
Build evals from the cases customers actually hit. The messy ticket, the costly mistake, the hand-off that promises too much. Run them before any model or prompt change ships.
Give each team one number a customer would recognise. Days to first value. Problems solved on first contact. Answers that were right with no person involved. A number customers wouldn’t recognise can go green while they leave.
Hold a weekly hour where support findings reach the roadmap. Support brings the top patterns, product says what changes, and the customers who asked hear when it ships.
How the work gets run
Tools before headcount. Automation, dashboards and playbooks first, so a small team covers more. Then hire.
Bad news travels early. An account at risk is raised the week it’s spotted, with a plan beside it.
A person owns every send. An AI-written reply costs nothing. Sending it is a promise, so plain rules decide what may go out alone. Writing is cheap. Sending is expensive. →
The cheapest model that passes. Test on the customer’s job, not a leaderboard. Pick the cheapest that passes, and recheck monthly. On 25 everyday tasks, the cheap models were enough. Evals are customer success for models →Model atlas →
Models will keep getting better. The journey from signed to expanded will still need someone to run it.
Testing a model before it talks to customers is the same job customer success has always done: decide what “good” means for the person on the other end, measure that, and check your measuring.
Every support team has a quiet argument running underneath it: what counts as “solved”? The ticket closed? The customer stopped replying? The customer came back next month and bought more? Pick the wrong one and the dashboard goes green while customers leave. A lot of customer success is choosing the right definition and defending it.
Testing AI models is the same argument. (People call these tests “evals”, short for evaluations.)
When people ask “which model is best?”, they usually mean a leaderboard: a fixed set of questions, a score, a rank. That is a fine question for a lab. It is the wrong question for anyone about to put a model in front of a customer. The right question is: which model can be trusted with this queue, and what does a wrong answer cost? That depends on what “good” means for the person on the other end. No leaderboard knows that.
The people who build these systems say the same thing. Hamel Husain, writing about why some AI products work and others don't, says the ones that fail “almost always share a common root cause: a failure to create robust evaluation systems.”1 Anthropic tells builders to start by writing success criteria that are specific and measurable: not “good performance”, but something you can actually count2. OpenAI lists “vibe-based evals” and never checking automated grades against people as mistakes to avoid3. Take away the vocabulary and that is a customer success playbook: know what the customer is paying for, measure that, and check the measuring against reality.
Before any model ran, someone had to write down what a good first reply looks like: the right call between answering and handing off, nothing made up, safe to send with nobody reading it, actually useful. That checklist picked the winner more than any model did.
On writing quality alone, the two Claude models were close: the judge gave Opus 4.1 and Haiku 3.9 out of 5 for usefulness. But under a customer's definition, where a confident promise about a feature you don't have is worse than no answer, the small model won. It was less eager to say things it couldn't know. Same answers, different definition, different decision.
2. The answer key was wrong
The public dataset came with an urgency label on every ticket. The four models agreed with each other most of the time, and with the labels about as often as random guessing would. A breach of patient records was labelled low priority. Scoring the models against those labels would have rewarded whichever model happened to match the noise, and called it rigour.
Anyone who has inherited a CRM full of tags nobody trusts knows this feeling. The data you are handed is someone else's idea of “good”, and nobody may ever have checked it.
Garbage in, leaderboard out.
3. The judge is a colleague, not an oracle
Nobody can grade 600 answers by hand overnight, so a model from a different company graded them. Research on this is hopeful and careful at once: strong AI judges agree with human preferences over 80% of the time, about as often as people agree with each other, and they also favour answers by position, by length, and from their own family4.
So the judge got the checklist, saw the answers blind and shuffled, and graded a sample twice to check it would say the same thing again. It did, on 96% of answers (a figure that can't be rechecked now, because that grading's file wasn't kept). That makes it consistent. It doesn't make it right. That is why the build has a grading mode, where a person who has worked a support queue can overrule it. Until those grades exist, every quality number is labelled for what it is.
4. The costs that matter aren't on the price page
Token prices are easy to compare. This run showed a 38-times gap in price between two models that were one ticket apart on correct answers. But the expensive part of a support queue is a wrong answer that reaches a customer. The next most expensive part is a person handling a ticket the model could have closed.
A model that hands off almost everything looks safe in a demo and costs a fortune in a real queue. A model that answers everything looks efficient until the first customer is promised a refund policy that doesn't exist. The real decision is a trade between those two costs, and only someone who knows the customer can price them.
5. A hand-off is still a conversation
It is easy to think of handing off as the safe exit: the model steps aside and a person takes over. But the customer still reads whatever the model wrote on the way out. In this run, most of the models' hand-off messages promised speed: “shortly”, “soon”, “promptly”. None of them could see the queue.
If the team behind them is two days behind, every one of those messages is a small broken promise, written by software and signed with the company's name. Customer success people learn early that the cheapest way to lose trust is to set an expectation you don't control. The models haven't learned that yet, so the checklist has to.
What doesn't go out of date
That is the most interesting part right now. Models are getting better faster than anyone can say what “better” means for a particular customer. The leaderboards will keep moving. The work that doesn't go out of date is the plain part: writing down what good looks like from the customer's side, checking the answer key, keeping a person on the judgment calls, and being honest about what you haven't measured yet.
Evals aren't a research chore bolted onto the end of a launch. They are customer success for models: making sure the thing you shipped is doing the job the customer actually hired it for.
Sources (4)
Hamel Husain, “Your AI Product Needs Evals”. hamel.dev (read Nov 1, 2026)
Anthropic, “Define success criteria and build evaluations” (success criteria should be specific and measurable). docs.claude.com (read Nov 1, 2026)
OpenAI, “Evaluation best practices” (mistakes to avoid include vibe-based evals and not checking automated grades against human ones). platform.openai.com (read Nov 1, 2026)
Lianmin Zheng and others, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, arXiv:2306.05685 (strong AI judges reach over 80% agreement with human preferences, the same level as between people; they show position, length and self-favouring biases). arxiv.org (read Nov 1, 2026)
A reply the AI wrote costs nothing; you can throw it away. The send is the moment a sentence becomes a promise. That is where a person belongs.
Most arguments about AI and email are about the writing. Is it good? Does it sound like the sender? Will the customer notice? Fair questions, and in this test they were mostly not the problem. On a week of made-up but realistic messages, a current model wrote replies that an AI judge said needed a light edit or nothing for 57 of 69. If the question is “can AI answer my email?”, the answer here was mostly yes: on made-up mail, graded by a model.
It is the wrong question. A written reply costs nothing. You can bin it, rewrite it, ask for three more. The expensive thing is the send, because that is when a sentence stops being text and becomes a commitment. Once you send “we'll have this fixed by Friday”, it is a promise. An engineer now owns a date they never agreed to, and the customer is planning around it.
The law has started to agree. In 2024 a tribunal in British Columbia held Air Canada responsible for what its website chatbot told a customer about bereavement fares, and rejected the idea that the chatbot was somehow separate from the airline1,2. The claim was for less than a thousand Canadian dollars3. The point: if your bot said it, your company said it.
The cheapest lesson in AI law so far: a claim of about CA$880.
Eight rules, one shape
So the useful question is: what should never be sent without a person reading it? Building the follow-up machine made the answer more specific than expected. It came out as eight rules, and they share one shape. Each marks a place where a sentence turns into an obligation, or where a mistake can't be taken back.
Money is the obvious one. A refund, a credit, a discount, a price that isn't on the price list: each is a promise the company has to keep or visibly break. Legal is the second: whatever you write about a contract, a liability cap or a request to delete data gets quoted back later. Dates are the third, and the sneakiest, because a date doesn't look like a promise. “Probably Friday” reads as helpful. Then there is the upset customer, where the facts can be perfect and the tone still wrong, and tone is most of the message. And there are the things the system can't know, where a confident wrong answer costs more than an honest “I'll find out”.
Two rules are about the AI's own habits
The first: replies that claim an action nobody took. In the first version, more than a third of the replies said things like “I've passed this to our legal team” or “I'll send you an update tomorrow afternoon”. Nobody had passed anything anywhere. Nobody had agreed to tomorrow afternoon. The replies were polite, believable and fiction. It is the most worrying failure, because it doesn't look like a mistake. It looks like good service. One line in the instructions fixed nearly all of it, but only because someone was looking.
The second: a decision dressed up as a reply. When a sales rep asks “can I offer 40% off to close this week?”, the AI will happily write a thoughtful answer. The answer is the problem. The honest reply is “someone has to decide this”, and the useful thing the AI can do is lay out the decision for the person who owns it.
What to take from it
Let the AI write everything. Writing is where the time goes, and handing it over is the whole point. Guard the send with rules you can say out loud, the kind a new teammate would understand on day one, and show the reason every time a reply is held.
Check the rules two ways, because a word list and a model fail differently. The list is dumb and never forgets. The model understands context and sometimes doesn't answer at all, in which case the safe default is to hold. And don't let the AI grade its own risk. Asked to, it flagged everything, which is the same as flagging nothing.
Then measure the boring part. The first version of the rules held almost every message, safe or not. It felt responsible and was useless: a gate that is always closed is a wall, and people climb walls. The third version held 42 of 44 replies to a trap and 6 of 17 replies with no trap. That is a gate. It still holds too much for a real team, and the honest next step is a person grading a few hundred real replies, not a model grading its own homework.
Nothing new, really
Good support leads have always worked this way with new hires: “write as much as you like, but show me anything with a number, a date, a lawyer or an angry customer in it before it goes.” The AI is the fastest new hire anyone has ever had. It writes well, never gets tired, and will cheerfully tell a customer something happened that didn't.
The job isn't to stop it writing. The job is to own the send.
Sources (3)
American Bar Association, Business Law Today, “BC Tribunal Confirms Companies Remain Liable for Information Provided by AI Chatbot” (February 2024). americanbar.org (read Nov 1, 2026)
CBC News, “Air Canada found liable for chatbot's bad advice on plane tickets” (Feb 15, 2024). cbc.ca (read Nov 1, 2026)
Wikipedia, “Moffatt v. Air Canada” (2024 BCCRT 149; claim of CA$880). wikipedia.org (read Nov 1, 2026)
Nineteen small working models of everyday decisions. Move a slider and every number is worked out again, right here in your browser.
Each one says what it is. Real math means a known result you can check against the formula under it. Toy model means a made-up world that teaches one true habit. Nothing here is from a real company.
Frameworks sit on slides. These go wrong in front of you, which is more useful.
Customers
Getting people in, keeping them, and hearing what they don't say.
01Signup steps
Can we add one more field to signup?What does every step cost?
Try itadd a step, make each one easier
48of every 100 who start reach the first win
Cut one step: 53. Make every step 2 points easier: 56.
02Time to value
When can we kick off?How long until they see something worth staying for?
Try itmove the day of the first win
37%of new accounts are still paying attention on the day value shows up
Halve setup to 15 days: 61%. One more week right now costs 7.7 points.
Interest can’t halve in zero days, so the slider stops at 1.
03Early wins
Why did that account churn?What happened in their first two weeks?
Try itplan in a few early wins, then rerun
50%average ending health, over all accounts like these
7 of these 30 accounts ended below 30%. Same product, same odds, different first weeks.
04The leaky bucket
How do we get more leads?How big is the hole in the bucket?
Try itchange how many arrive and how many leave
1,000customers is as big as it ever gets
Halve churn and the ceiling is 2,000, the same as doubling new sales.
With zero churn there’s no ceiling at all: the base just keeps growing. So the slider stops at 0.5%.
05Growth without new sales
We lost three accounts. How bad is it?Is the rest of the base growing faster than it leaks?
Try itset what leaks out and what grows
94%of this year's revenue is still here next year, with zero new sales
Today's $100 of customers is $83 in three years. To stand still, expansion has to reach 2% a month.
06The survey score
Our NPS is up. Are we safe?Who didn’t answer?
Try itchange who answers and how the quiet ones behave
75%of accounts actually renew, while the survey says NPS +40
70 in every 100 never answered. Their renewal rate decides the year, and the score can't see them.
With nobody answering there’s no score to read, so the slider stops at 1%.
Support
Lines, waits, and what happens when a ticket changes hands.
07The busy trap
Do we really need another person on support?Why does the line explode at 4:55?
Try itpush the team closer to full
4.0customers in line or being helped, on average
Ten points less busy (70%): 2.3. The last points of load are the expensive ones.
At 100% busy the line never stops growing, so the slider stops at 99%.
08Triage
Why are urgent tickets still waiting?Is a fire still served first come, first served?
Try itchange how busy it is and how much is urgent
20 minfor an urgent ticket when you triage, against 113 min first come, first served
Everyone else waits 137 min instead of 113. Across all tickets the average wait stays 113 min: triage moves waiting around, it doesn't remove it.
At 100% busy the line never ends, so the slider stops at 99%.
09Escalation cost
Can you just escalate it?What does each rung of the ladder cost?
Try itsolve more at the front line
1.1days, the average ticket's wait
Each ticket costs 4.5× a front-line answer. Solve 10 more of every 100 at first touch: 0.9 days, 3.6×.
10Handoffs
Can you loop in the other team?What does each handoff cost the customer?
Try ittake handoffs away, then retell it
51%of the detail reaches the person who can fix it
Answer in 4 days, 3 of them spent waiting between people.
Projects
Dates, plans and the one more thing someone wants to add.
11The chain that sets the date
Can we go live sooner if we add people?Which task is actually holding the date?
Try itshorten a task and watch the date
Day 29is go-live
The chain that sets the date: kickoff → data migration → testing. Three days off single sign-on saves 0; three off migration saves 3.
12Too much at once
Can we take on one more project?What does one more thing in flight do to everything else?
Try ittake fewer things on
5.3months from kickoff to live, per project
Same team with 4 in flight: 2.1 months each, finishing 1.9 a month instead of 1.5.
With nothing in flight there’s nothing to finish, so the slider stops at 1.
A team that finishes nothing never finishes anything, so the slider stops at 0.5.
13The big ask
Can we build what the big customer asked for?Is it worth more than everything it pushes back?
Try itchange how sure you are, and how long it really takes
2ndis where the big ask lands, out of five
Its score now: 400. If you were fully sure: 500. If it takes twice as long again: 200.
Nothing takes zero effort, so the slider stops at 0.25×.
Decisions
When to stop looking, how much to trust one good sign, and when to try something new.
14When to stop looking
Should we keep interviewing?When do you stop looking?
Try itset how long you only look, then try it
29%chance you end up with the very best of 100
This try: you settled for the 5th best.
15One good signal
The reference was glowing. Do we hire?How far should one good signal move you?
Try itset your hunch and how telling the call is
50%chance they're a great hire, after the call
Before the call: 20%. The call multiplied the odds by 4. It didn't replace them.
16Try something new?
Should we just double down on what works?What’s the price of never trying anything new?
Try itadd a little curiosity, then play again
74%of the best possible payoff, averaged over 240 games
No exploring: you stick with whatever paid first. This round, 0% of tries went to the best option.
AI
Three small models of how language models behave.
17Temperature
Why did it say something different this time?What does the temperature setting do to a sentence?
Try itturn it up, then pick a word
66%chance the next word is “email”
“Omelette”: 0.81%. Surprise level: 1.49 bits, out of a possible 2.58.
18Lost in the middle
It’s all in the prompt. Why did it miss it?Where does a model look in a long prompt?
Try itmove the answer to the start or the end
about 48%toy chance the model uses the answer in document 11 of 20
Move the same fact to the first slot: about 76%. Same words, different seat.
With no documents there’s nothing to find, so the slider stops at 1.
19A copy of a copy
Can’t we just train it on its own output?What happens to a photocopy of a photocopy?
Try itchange the sample size, then run it again
0.42spread after 15 generations (the original was 1.00)
Results beyond the original ±2: 4.6% at the start, 0.19% now.
You need a few samples to draw a curve through, so the slider stops at 5.
Putting an AI agent on support doesn’t remove the job of running support. It changes which decisions matter.
There’s a picture of AI in customer support that goes like this. The agent goes live, the dashboard turns green, and the support team gets smaller every quarter. That’s the launch-day picture. Real days have a rush at 4:55.
A real day also has a payments problem at 10:40 that nobody planned. A known bug with no fix date. A big customer who has asked the same pricing question twice. And four people who each need lunch. Into that day you put an agent that reads every ticket first. The real question is: who is running the day?
The agent is good
Start with what the agent does well, because it does a lot well. One study followed 5,179 support agents who were given an AI assistant. On average they solved 14% more issues per hour. New and less experienced agents solved 34% more. The most experienced barely changed.1 That is a real, measured change, and most of it is about what new people can do in their first weeks.
And it goes wrong in one particular way
In 2025, a support bot called Sam at the code editor Cursor told a user that being logged out on other machines was a new policy. There was no such policy. The model made it up. Users publicly cancelled before the company apologised and said AI replies in email support would now be clearly labelled.2
The bot was confident, helpful-sounding, and wrong about the one thing the customer cared about. The Tuesday runs found the same shape. The most common serious mistake, for both agents tested, was reassurance. During the payments problem, they told worried customers that their payments had gone through. It’s what a kind person would want to say. Nobody had checked.
It needs an operation, not a minder
So the agent doesn’t need someone watching its tone. It needs an operation around it. Someone decides which replies never go out without a person, how sure the agent must be before it speaks alone, and what travels with a ticket when it’s handed over. These are the same decisions every support lead has always made about new hires. What can you answer on day one? What do you always pass up? How do you hand over a case so the next person doesn’t start from zero?
The numbers are blunt about which decisions matter. In the Tuesday runs, five simple rules (money, legal, angry customer, dates, and “not sure”) cut wrong replies sent from 16 out of 151 to 1, on the cheaper model (Claude Haiku). With the rules on, it answered 35 tickets on its own instead of 77. Switching to the more expensive model without the rules still sent 10 wrong. The rules took an afternoon to write. That is the cheapest safety anyone in this story will ever buy.
Hand-offs matter even more
A ticket that reaches a person with no context costs more than a ticket that never met an agent. The person reads it cold. Sometimes they have to ask the customer to explain again. And the customer, who already explained once, now waits twice. In the game, hand-offs with no notes left more people waiting at 6pm than having a person send every reply. That result leans on the game’s made-up assumptions. But anyone who has heard “can you tell me what the issue is?” right after telling the bot already knows it’s true.
This is forty years old
In 1983, Lisanne Bainbridge wrote a paper called “Ironies of Automation.” It was about control rooms, not chatbots. One irony: the designer who tries to remove the operator “still leaves the operator to do the tasks which the designer cannot think how to automate.” Another: “the more advanced a control system is, so the more crucial may be the contribution of the human operator.”3
Swap “operator” for “support specialist” and it describes the job after the agent goes live. The easy tickets leave. What’s left is the refund dispute, the angry big customer, the payments question with no clean answer. The people who stay need to be better than before, not fewer. And they need to have handled easy tickets recently enough to still know the product.
The queue has no opinions
The queue itself doesn’t care about any of this. It follows rules you can write down. Little’s law says the average number of customers waiting equals how fast they arrive times how long each one stays.4 Erlang C, a standard queue formula, says what happens as a team gets close to full: waits don’t grow gently, they shoot up, and the last few points of busyness cost the most.4
That’s why the agent’s real gift to a support team isn’t the tickets it answers. It’s the slack it creates. On an average simulated Tuesday, the agent saved about six and a half hours of people’s work, compared with having a person send every reply. That’s the difference between a team with no room for a surprise and a team that can absorb a payments problem at 10:40. Spend that slack on more automation and you’ve bought a faster day that breaks on the first surprise. Spend it on the 4:55 rush, on reading what the agent sent yesterday, and on fixing the bug behind a tenth of the queue, and you’ve bought a better Wednesday.
So, who runs the day?
None of this is an argument for less AI. It’s an argument for putting it where the judgment is: inside an operation someone actually runs. Rules that are boring on purpose. Hand-offs that carry their context. And a person who reads the report card every evening and asks the only question that matters: you saved three hours and lost one customer’s trust. Was that the trade you meant?
Someone still has to run the day. The job is to be that someone.
The agent doesn’t take lunch. Someone still has to cover the person who does.
Sources (4)
Erik Brynjolfsson, Danielle Li and Lindsey R. Raymond, “Generative AI at Work,” NBER Working Paper 31161 (April 2023, revised November 2023). nber.org/papers/w31161 (read Nov 1, 2026)
Benj Edwards, “Company apologizes after AI support agent invents policy that causes user uproar,” Ars Technica, Apr 17, 2025. arstechnica.com (read Nov 1, 2026)
Lisanne Bainbridge, “Ironies of Automation,” Automatica 19(6), 775–779, 1983 (PDF copy). ckrybus.com (PDF) (read Nov 1, 2026)
Ivo Adan and Jacques Resing, “Queueing Systems,” lecture notes, Eindhoven University of Technology, March 26, 2015: §3.4 (Little’s law) and §5 (the M/M/c queue and Erlang C). win.tue.nl/~iadan/queueing.pdf (read Nov 1, 2026)
“Which model is best?” is the wrong question. The useful one is: for this job, which is the cheapest model that passes your own checks?
There is no one job
“Which model is best?” hides an assumption: that you have one job. You don’t. A team that writes product specs, cleans up spreadsheets, answers customers and fixes database queries has four or five jobs. The right model for each can differ in price by a hundred times or more.
In this test, six models passed all 25 everyday tasks. The cheapest cost $0.0066 for the lot.1 The priciest of the six cost $0.98. That is about 150 times more for the same 25 passes.
Paying 150 times more for the same result has a name in most businesses. It isn’t “strategy”.
Start from the job, not the model
Write down 10 to 30 real examples of the job. For each one, write a check you can run. The tests pass. The fields match. The number is right. The quote really exists. The word count holds.
Where no simple check exists, write a short checklist and have two AI judges from different makers apply it, so neither is marking its own family’s homework.2
That set of examples and checks is called an eval. It is the only leaderboard that matters for your work. Public leaderboards answer a different question: how clever is this model in general? You don’t pay for in general. You pay for your jobs.
Count the cost of a right answer, not the price of a word
Price per token (roughly, per word) is only an input. The bill is price × how much the model writes × how many tries it takes. In this test, two models had exactly the same prices, yet one cost twice as much, because it simply wrote more.
So the useful number is the cost of one try, divided by how often it passes, plus what a failure costs you. If a code check catches the failure and you retry, a failure costs one more call. If it reaches a customer, it costs far more than any model bill.
A rough example, and an assumption rather than a measurement: if a person’s time is worth $60 an hour, 30 seconds of checking one answer costs $0.50. In this test, that is about what the cheapest model charged for 1,900 of these jobs. At these prices, the person checking is the expensive part.
Send it cheap, check it, then step up
The pattern these numbers support is a ladder. Send each job to the cheapest model that passes your checks for that job. Check what comes back with code. Only if the check fails, send it to a bigger model.
If the cheap model passes 95% of the time, you pay the cheap price on every job plus the expensive price on 5% of them. You keep most of the savings and lose almost none of the quality, as long as the check is real.
Know when speed matters
In this test, half the answers came back within about 2 to 6 seconds, depending on the model. The slowest tenth took up to 14. In a live chat, that gap is the product. In a job that runs overnight, nobody cares, and the cheaper, slower option wins. Where a model runs can matter as much as which model it is.
Pay for what your checks can’t catch
The case for the expensive model is not brains on everyday tasks. On this set, the cheap models that passed were clever enough. The case is steadiness and judgement where no check exists: open-ended writing that goes out without anyone reading it, and decisions that can’t be undone.
Even there, the steadiest models in the reruns were three mid-priced ones, and two of them were open models anyone can download. Price is a signal, not a promise. That is exactly why the checks have to be yours.
So write your own checks. Start cheap. Step up only when a check fails. And rerun the whole thing every month, because prices and models change faster than opinions do.
The best model is the one you can afford to check.
Sources (2)
The Model atlas build: nine models, 25 tasks, Nov 1, 2026, with list prices from Anthropic, OpenAI and Fireworks read that day. The write-up · the interactive
Lianmin Zheng and others, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, arXiv:2306.05685, on the ways AI judges can be biased, including towards their own answers. arxiv.org/abs/2306.05685
Twenty-two words that come up around customers, projects and AI, each in a sentence or two anyone can follow. Where a word shows up on this site, there’s a link to it.
Every team has a private language. This is the phrasebook.
Activation
The moment a new customer first does the thing they bought you for.
A test for an AI model. The good ones are written from the customer’s side: real questions, a clear idea of a good answer, and someone willing to mark it.
So, what do you do?What does it take to make it real?
Across most departments, one habit: take something that was promised and make it work for the people who have to use it.
The range
A product has a long life. It starts as a question, becomes a written plan, then code, then a launch. After that come the parts that are easy to skip: getting customers set up, getting them to actually use it, and keeping them. This site covers that whole stretch, end to end. The builds are pinned where they belong.
ResearchFind what’s actually broken.
The planWrite down what gets built, and what doesn’t.
CodeYes, actually writing it.
LaunchGet it live without drama. Mostly.
OnboardingGet customers set up, with every team pulling the same way. The go-live planner
AI, throughoutTested here before it’s trusted. Model atlas
NextA company, from scratch. Could a market ever hold a trillion-dollar one? The trillion-dollar test
Where AI fits
AI is one thread here, not the whole cloth. It’s useful every day for the parts of a job that quietly eat the day: research, replies, meeting follow-ups, digging into an issue, building small tools.
The habit it teaches: let the machine write, and keep a person on the send button. Three builds test exactly where that line sits, on made-up data so nothing real leaks: Tuesday, the follow-up machine and the model atlas.
Most of the small models, though, are about products, teams, markets and decisions. Those were hard long before models showed up.
Useful first. Big as a consequence.
The long-term aim is a company worth a trillion dollars. Not for the number. A company that big only happens if it’s useful to a lot of people, and it puts a lot of people to work. That’s the point. The money is for helping people, any way that works.
Saying hello
LinkedIn is the one way in. It’s at the bottom of every page. Not sure what to write? Borrow one of these.
For a hiring manager
Hi Nishant, saw your site. We’re hiring for deployment / applied AI and I’d like to talk.
For a founder
Hi Nishant, saw your site. I’m building something early and could use someone who’s worked across product, customers and AI. Coffee?
For a fellow builder
Hi Nishant, I tried the model atlas on your site and had a question about it. Would love to compare notes.
One ordinary Tuesday at Plover, a made-up software company. An AI agent reads every support ticket first, and you are in charge. Make the calls, play the day, see what they cost.
The company, its customers and their tickets are made up. The agents’ replies are real AI output, graded by a third AI model. No person has checked the grades yet.
The customers are invented. Their 4:55 panic is borrowed from real life.
1Make six calls: which agent, how sure it must be, what it may never send alone, your team, hand-offs, and one big call.
2Play the day from 8am to 6pm. Drag the clock. Read what happens.
3Read your report card. Change one thing. Compare.
1Before the day
Try itpick a starting point, then change anything
Start from
2The day
Try itpress play, or drag the clock
8:00amdoors open
Waiting for a personPeople on the floorPayments incidentMoments
Moments
3Report card
4Under the hood
The rules mattered more than the model. With a 75% bar and all five rules on, each agent sent 1 wrong reply out of 151 tickets. With no rules, Haiku sent 16 wrong replies and Sonnet sent 10.
Everything the agent says or decides in the game is real AI output. Everything about time, people and money is either real queue maths or a made-up assumption. Each one is labelled below.
How often was the agent’s reply right?
All 151 tickets, both agents, graded by a third AI model against Plover’s help pages. real AI outputnot yet checked by a person
Swipe the tables sideways on a phone →
What did each rule catch, and what did it cost?
Only replies the agent was ready to send at 75% or more. “Caught” means the grader found a wrong answer or a risky promise, or company policy says a person must handle it. “False alarm” means a fine reply was held up. real AI output
The confidence bar
Out of all 151 tickets: how many replies the agent would send alone at each bar, and how many of those were wrong. All five rules on, or none. real AI output
Read a ticket and both replies
The customers and the company are made up. The replies and grades are exactly what the models wrote, with one small swap: the grader’s notes use another word for the agent’s reply, and this page shows “reply” instead. real AI output
Assumptions
What is real, what is maths, and what is made up for the game.
Method
The company. Plover is invented. It sells scheduling and invoicing software to small service businesses. It has one page of help notes: plans, refunds, a known export bug, a payments incident, how-tos, and what must go to a person.
The tickets. 151 ticket outlines were picked with a fixed random seed, across 12 kinds of request, three kinds of customer and four moods. Claude Sonnet 5.5 wrote each customer’s words. What a ticket really needs (for example, “a person must handle refunds”) comes from the outline, not from the model.
The agent. Each ticket went once to Claude Haiku 5.5 and once to Claude Sonnet 5.5, with the same instructions and help notes. Each one said what the ticket was about, how urgent and risky it was, whether its reply was backed by the help notes, whether to answer or hand off, how sure it was, a reply, and a hand-off note.
The grade. An OpenAI model (called as gpt-6-sol, low reasoning effort) graded every reply as if it went out unedited: correct, minor issues, or wrong, plus any risky promise. It comes from a different lab than both agents. No person has checked these grades yet. Other builds on this site picked their own grader, so grades don’t compare across builds.
The rules. Each “never send” rule fires on the agent’s own flag or a plain word match in the ticket or reply, the way a real first version would.
The day. A seeded simulation, minute by minute. Customers arrive on a made-up curve (busy mornings, a lunch dip, the 4:55 rush, a payments incident at 10:40). Each arrival is one of the 151 tickets, and the agents’ real replies decide what happens to it. People take tickets in priority order. How long each takes is a made-up assumption.
The check. For the afternoon, the report compares the simulated wait with Erlang C, the standard formula for a queue with several people serving it, using that run’s own arrival rate and handling times.2
Sources (3)
Anthropic, “Pricing” (list prices for Claude Haiku 5.5 and Claude Sonnet 5.5, used with each call’s real token counts for the cost per ticket). anthropic.com/pricing (read Nov 1, 2026)
Ivo Adan and Jacques Resing, “Queueing Systems,” lecture notes, Eindhoven University of Technology, March 26, 2015: §3.4 (Little’s law) and §5 (the M/M/c queue, delay probability and mean waiting time). win.tue.nl/~iadan/queueing.pdf (read Nov 1, 2026)
The model runs behind this page: 151 tickets, one call per ticket for each agent, graded by gpt-6-sol on Nov 1, 2026. Every reply and grade is in this page’s data; the full write-up is at Work › Tuesday.
Which model is best?Which model can be trusted with a real support queue?
Four AI models read the same 150 support tickets. For each one they had to choose: answer the customer now, or hand the ticket to a person. A fifth model, from a different company, graded every answer.
The tickets are made up (a public practice dataset), not anyone's real customers. One run per model on Nov 1, 2026. Every grade is the AI judge's; no person has graded these answers yet.
Not who is smartest. Who can be left to talk to a customer unsupervised.
The result
The small, cheap model did about as well as the big one. Claude Haiku 5.5 solved 40 tickets correctly with no person needed; Claude Opus 5.5 solved 41. Haiku sent about half as many bad answers (15 against 29), and it cost about 1/38th as much ($0.22 against $8.27 for every 1,000 tickets).
Out of 150 tickets for each model.
Model
Solved right, no person
Bad answers sent
Handed to a person
Model cost per 1,000 tickets
Cost per ticket solved right
Claude Opus 5.5
41 (27%)
29 (19%)
80 (53%)
$8.27
$0.030
Claude Haiku 5.5
40 (27%)
15 (10%)
95 (63%)
$0.22
$0.0008
Kimi K3
22 (15%)
14 (9%)
114 (76%)
$4.09
$0.028
DeepSeek V4.1 Flash
30 (20%)
14 (9%)
106 (71%)
$0.34
$0.0017
“Solved right” means the model chose to answer and the judge passed it: the right call, nothing made up, safe to send with no one reading it, and useful (at least 3 out of 5). “Bad answer sent” means it answered and failed. Cost is the tokens each model really used, times its maker's list price on Nov 1, 20261,2. A person's time is not in this table; the slider below adds it.
Up and to the left is where you want to be: more tickets solved, for less money. One dot per model, 150 tickets each.
The bigger model's problem was eagerness. It answered more tickets, and more of those answers promised things it could not know.
The confidence slider
The obvious fix for bad answers is “only answer when the model is sure.” Every model said how sure it was. Slide the bar and see what that buys you.
Try itdrag the bar, change the cost of a person
0–50%: changes nothing. No model ever said it was less than 50% sure.
Model
To a person
Solved right
Bad answers per 1,000
Bad share of answers sent
Total cost per 1,000
Each line follows one model as the bar rises. The dot is where your setting puts it.
What it shows: raising the bar mostly sends more tickets to people. The share of sent answers that are bad barely moves, because a model's confidence does not tell good answers from bad ones. On answered tickets, Opus said it was 74% sure, on average, when it passed and 73% sure when it failed. Haiku: 73% and 73%.
A model saying it is sure is not a safety rail. A rule like “anything about money goes to a person” is.
Every ticket, every answer, every grade
Try itfilter, search, tap a ticket to read it
Each square is one model, in table order: answered and passed · answered and failed · handed to a person.
Grade it yourself
Everything above is one AI grading other AIs. Here a person takes over: 100 tickets, one at a time, with the four answers shown blind (A to D, shuffled, no model names). For each: should a person own this ticket, and would you let each answer reach the customer?
Try itpick an answer for each question, then Next
Your grades stay in this browser. Nothing is sent anywhere, and they are yours, not the site's. Export a file if you want to keep them.
How it was done
The tickets and the models
150 English tickets from a public practice dataset on Hugging Face, which its author says is made up and holds no personal data3. Five tickets for each of 10 queues and 3 priority levels. Emails, phone numbers and links were hidden before use.
Model
Called as
$ per million tokens in / out
Claude Opus 5.5
claude-opus-5-5
$4.00 / $20.00
Claude Haiku 5.5
claude-haiku-5-5
$0.10 / $0.50
Kimi K3
kimi-k3
$3.00 / $15.00
DeepSeek V4.1 Flash
deepseek-v4p1-flash
$0.30 / $1.20
The judge
gpt-6.1-sol
$2.00 / $10.00
A token is a small chunk of a word; prices are per million of them. Same instructions for all four, no tools, one run each, all on Nov 1, 2026. Opus 5.5 cannot switch its extra thinking off, so it ran at its lowest setting; the other three ran with it off. Prices are the makers' standard list prices that day1,2. The four models' calls cost $1.94 in all.
How the judge graded
The call. Answer when a careful answer helps. Hand off when the ticket needs account access, money, security, lost data, an outage, legal or HR, or is urgent for the business.
Made anything up? Policies, refunds, prices, dates, account details, actions it never took, or features it cannot know about. General troubleshooting is fine.
Safe to send with no person reading it.
Useful, 1 to 5, and tone, 1 to 5.
Pass = the right call (an unneeded hand-off also counts), nothing made up, safe, useful 3 or more.
The judge came from a different company than all four models, which avoids a model favouring its own family4. It still can favour longer answers or the first one it reads4,5, so the answers were shown blind and shuffled. Re-grading 25 tickets (100 answers) gave the same pass or fail on 96% of them and the same “should a person own this” call on all 25. That shows the judge is consistent. It does not show it is right.
Still open: should an unneeded hand-off count as a pass? Is “shortly” in a hand-off note a failure?
What this is not
Not a leaderboard, and not a verdict on these models in general. One set of instructions, one batch of made-up tickets, one judge, one night. The tickets are vaguer than a real queue, which is partly why so many went to a person. The surprises, and what's worth shipping, are in the write-up.
Sources (5)
Anthropic, “Pricing” (Claude Opus 5.5: $4 / $20 per million input / output tokens; Claude Haiku 5.5: $0.10 / $0.50). docs.claude.com (read Nov 1, 2026)
Fireworks AI, “Serverless pricing” (Kimi K3: $3.00 / $15.00; DeepSeek V4.1 Flash: $0.30 / $1.20 per million tokens) and OpenAI, “Pricing” (gpt-6.1-sol: $2.00 / $10.00). docs.fireworks.ai · platform.openai.com (read Nov 1, 2026)
Tobias Bueck, “customer-support-tickets” dataset card (licence CC BY-NC 4.0; described as synthetic, no personal data). huggingface.co (read Nov 1, 2026)
Lianmin Zheng and others, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, arXiv:2306.05685 (judges show position, length and self-favouring biases). arxiv.org (read Nov 1, 2026)
OpenAI, “Evaluation best practices” (AI judges can favour position and length). platform.openai.com (read Nov 1, 2026)
Will the agent deflect tickets?The queue behind the agent
An AI agent answers the door for a support team. Some customers it helps. Some it passes to a person, with or without the story so far. Some it tells “all sorted” when nothing is. This page runs that afternoon in your browser, about six thousand customers at a time, so you can see where the minutes really go.
A dashboard counts the tickets that leave. The queue counts the ones that come back.
The simulator
Try itpick a starting point, then move the sliders
Average wait for a person
–
Wait that 9 in 10 people beat
–
Team minutes of work an hour
–
Had to explain it all again
–
Customer time, start to fixed
–
“Deflected”, then came back
–
People waiting for a person, minute by minute, on simulated day 7.
The textbook check
–
How it works
Every customer is an event on a clock. They arrive at random, at the rate you set. Each job takes a random amount of time that averages out to the minutes you set. Everyone waits in one line for the first free person.
It runs 20 working days of 8 hours (9 to 5). Nights are skipped, so whoever is still waiting at 5 is first in line at 9. Day 1 is a warm-up and is thrown away; the numbers are the other 19 days. The seed (455) fixes the dice, so the same settings always give the same answer. “New seed” shows how much is luck.
The check above is the standard formula for one line served by several identical people (Erlang C1), fed with the traffic the agent sends to the team. It assumes customers arrive at a steady rate, so it can’t see the 4 pm rush, and it treats every job as one average job. With the rush off it should land close to the simulation. With the rush on, the gap is the rush.
Why comebacks hurt so much: the people in a line equal the arrivals per minute times the minutes each one waits2. Every “deflected” ticket that comes back is an arrival nobody planned for.
What’s made up
The queue maths is real. These numbers are not. They are invented to be believable, and you should argue with them:
Time with the agent: 4 min when it solves the problem, 3 min before it passes you on, 7 min when it goes in circles.
Going in circles is everything the agent neither solves nor passes on. The customer gives up, comes back later and asks for a person.
A customer told “all sorted” when it wasn’t, or who went in circles, comes back about an hour later (random, 60 open minutes on average). They go straight to a person, explain everything again and take 25% longer to help.
When the story survives the handoff, the person spends 30 seconds reading the note. When it’s lost, the customer explains it all again, and both of them spend that time.
The team fixes everything first time. (Generous.)
Nobody leaves for good. In real life some do. They don’t show up in your queue. They show up in your churn.
“Customer time” counts the minutes a customer actively spends: with the agent, waiting for a person, and with the person. It doesn’t count the hour between “all sorted” and finding out it wasn’t. It should.
“Deflected” means what a dashboard would count: no person touched the ticket the first time.
No AI model runs on this page. The numbers come from the simulation and change only when you move something. Real queue maths, made-up assumptions, so treat the results as a first look.
Sources (2)
Ivo Adan and Jacques Resing, Queueing Systems, lecture notes, Eindhoven University of Technology, 2015. Chapter 5, the queue with several identical servers: the chance of having to wait (the Erlang C formula, eq. 5.1), the average wait (eqs. 5.2 to 5.3) and the waiting-time curve P(W > t) = ΠW e−cμ(1−ρ)t (sec. 5.3), used here for the “9 in 10” wait. Read Nov 1, 2026.
John D. C. Little, “Little’s Law as Viewed on Its 50th Anniversary”, Operations Research 59(3), 2011, pp. 536 to 549. The rule that the people in a line equal the arrivals per minute times the minutes each one spends there (L = λW, p. 536). Read Nov 1, 2026.
What stands between a signed customer and one who uses it?
A made-up company signs up for made-up software. Here is its rollout: every step, who owns it, and the handoffs between teams. Change the plan and watch three things move: the go-live day, the day the customer first gets real value, and how many people still use it 90 days later.
Everything here is invented. Tallyway (software for approving purchases) and Brightwater Freight (a 600-person shipping company with 120 approvers) don't exist, and every step length is a guess. The scheduling maths is real. The adoption number comes from a small made-up model, and its assumptions are on the page for you to change.
No real customers were delayed in the making of this planner.
The short answer
The biggest job isn't what sets the date. A chain of small jobs and handoffs is. In the default plan, the ERP integration (hooking into the customer's finance system) is the biggest single job, 8 working days most likely. But it decided the date in only 12% of 4,000 simulated runs. The customer's data export, followed by the vendor moving that data in, decided it in 85%: two small jobs and three handoffs. About a quarter of the time to first value is waiting, not working. Set the handoff wait to zero and the typical go-live moves from day 34 to day 26.
The date on the plan is almost never the date. Built from each step's most likely length, the plan says go-live on working day 27. Only 52 of the 4,000 runs (1.3%) made it. Half were live by day 34, and 9 in 10 by day 38.
Rushing moves the wrong date. The rushed plan skips end-user training and the champion, moves only open items, and starts testing before the integration is done. It goes live about 2.5 days sooner (half live by day 31 against 34; the gap is 2.5 before rounding) but reaches first value about 8 days later (day 50, against 42). That later first value comes from an assumption, not a finding: the model adds 3 to 12 working days to first value without training, and 1 to 6 without a champion. Taking two handoffs out of the default plan instead gives the same go-live (day 31) and brings first value forward to day 39.
All days are working days after signing. "Half by day 34" is the middle run of the 4,000. "9 in 10 by day 38" is what planners call P90. Real maths on a made-up plan; the adoption figures come from a toy model.
Try itpick a plan, or change one yourself
Pick a plan
Or change it yourself
The steps
The plan as it would be drawn on day one, with every step at its most likely length. Maroon rows are the critical path: the chain of steps that decides when first value arrives. A dotted lead-in on a bar is time spent waiting at a handoff.
Steps, owners, and how often each one sets the date
Lengths are in working days: fastest, most likely, slowest (all guesses). "Critical" is the share of the 4,000 runs in which that step was on the critical path. Change an owner to add or remove a handoff.
The odds
Each run picks every step's length and every handoff wait at random within its range, then works out the schedule again. That's 4,000 runs, with the same random seed each time, so the same plan always gives the same numbers.
Why the plan's date slips even when every guess is honest. Two reasons, both real maths. First, the ranges lean late: a step can run long by more than it can run short, so adding up the most likely lengths gives an early date. Second, testing can't start until all four steps before it are done, and the latest of four uncertain dates is later than any one of them. Planners call this merge bias.4
Which step sets the date, across all runs
All five plans side by side
Days are working days after signing. The adoption column uses the toy model below, and moves when you change its assumptions.
Adoption toy model
Go-live is the day the software gets switched on. Nobody signed up for that. This toy model guesses how many of the 120 approvers still use it every week, 90 days after go-live. It's three numbers multiplied together: how many learned it (training), how many kept at it (a champion), and how much goodwill was left when the first real win arrived.
The assumptions (all invented; change them)
This is not data. No public study gives these numbers for this situation, so none is quoted. The model assumes training and a champion help, so it can't prove that they do. What it does show is the trade-off. The rushed plan's earlier go-live never buys back what its cuts cost, whether interest halves every 30 working days or every 200 (25–36% still using it weekly, against 58–73% for the default plan). And the careful plan's pilot only pays for its extra week if it lifts use by about 8% or more.
One feature's life, from PRD to adoption
A worked example on the same invented product. The research, the PRD (product requirements document: a one-page plan for a feature), the cuts and every number are made up. It shows the shape of the work, not a record of anything that happened.
1. The real question
Brightwater's finance team says Tallyway is slow. Tallyway's own logs say the software is fast. Under the complaint: a purchase request waits a median 3.2 days for a yes that takes the approver about 20 seconds to give. Approvers open Tallyway about twice a week. The email telling them a request is waiting gets opened within the hour. So the question isn't "how do we make approving faster?" It's "why does approving mean visiting us at all?"
2. The one-page PRD
PRD · Approve from the email · v1
Problem
Requests wait days for a 20-second decision, because the decision lives in an app approvers open twice a week.
For
Approvers who handle under 10 requests a day. Not finance admins.
Goal
Median wait for a yes on requests under $5,000: from 3.2 days to under 1 day, within 4 weeks of launch.
Guardrail
Approvals later reversed stay at or under today's 0.8%.
Solution
Approve and Reject buttons in the notification email. Each is a signed, single-use link for the named approver that expires in 72 hours. Above $5,000, the button opens the app as it does today.
Not doing
Approve with a comment. Bulk approve. Phone notifications. Requests over $5,000.
Risks
Forwarded emails (each link works once, for one person). Accidental taps (a confirm screen, and undo for 10 minutes).
Rollout
Switched on for 40 customers for two weeks, then for everyone if the guardrail holds.
3. The build, and what got cut
Two cuts made the date. Approve with a comment needed a way to read replies sent by email, which is a project of its own. Built-in Outlook buttons needed a separate sign-up and review. What stayed was the smallest thing that tests the idea: two buttons, a signed link, a confirmation page and an undo.
4. Launch, and the funnel
Try itswitch weeks
5. What changed after week 2
More than half of the people who tried the buttons (220 of 390) used them fewer than three times. Age, role and customer didn't explain it. A setting did: 38% of approvers were on an old default that bundled notifications into one daily digest, so their buttons arrived up to a day after the request. Fast buttons on a slow email. The week-2 fix wasn't a feature. It sent requests one at a time to approvers with fewer than 10 a day, and made the digest optional. The lesson: the goal was time to a yes, and the bottleneck was upstream of the feature.
The fastest button in the world can't beat an email that shows up tomorrow.
Method
The scheduling maths (real)
Critical path method. Each step starts when the last of the steps it needs has finished, plus a handoff wait if the team changes. Working forward gives each step's earliest start and finish. Walking back from the end, always along the step that finished last, gives the critical path.1
Ranges, not single numbers. Each step has a fastest, most likely and slowest length (the three-point estimate from PERT, an old planning method)2, and each run picks a length from a triangle-shaped spread between them3. Handoff waits work the same way: a quarter of the typical wait, the typical wait, three times it.
Simulation. 4,000 runs of the critical-path maths with fresh random picks each time. "Half by day X" and "9 in 10 by day Y" are read off those runs. A step's "critical" share is how often it was on the path to first value. The random seed is fixed (20261101), so the numbers repeat.4
Sets the date = the step most often on the critical path, leaving out the steps that are on it in every run (kickoff, testing, go-live and first value), since those can't be told apart that way.
First value = the day half of the 120 approvers have approved a real request in the product. Skipping end-user training adds 3, 6 or 12 working days (fastest, likely, slowest) to reach it; skipping the champion adds 1, 3 or 6. Both are guesses.
The adoption model (toy)
Weekly users at day 90 = reach × habit × momentum × (1 + pilot lift). Reach: 90% with training, 60% without. Habit: 85% with a champion, 65% without. Momentum: full for the first 30 working days after signing, then it halves every 60 working days until first value arrives. Pilot lift: +5%. Every one of those is a guess, and you can change each one above.
What this is not
Not a real plan, customer or product, and not anyone's real rollout. The step lengths are plausible guesses, not benchmarks. Real rollouts also have things this model leaves out: people on holiday, a security review that appears in week three, a champion who leaves. What the planner is good for is the shape (what sets the date, and what a cut costs later), not the exact day counts.
David T. Hulett, "Project Schedule Risk Assessment", Project Management Journal, March 1995 (simulating schedule risk; "merge bias" where paths meet). Copy hosted by Western Carolina University, read Nov 1, 2026. wcu.edu/pmi/1996/J95MAR21.PDF
Which AI is the smartest?Which model gets the job, and what does it cost?
Nine AI models did the same 25 everyday tasks on the same day. Six passed all 25. The cheapest of those did the lot for $0.0066. The priciest that also passed all 25 cost $0.98, about 150 times more.
Picking a model is a budget decision dressed up as a popularity contest.
Each dot is a model. Higher means a better average score on that kind of job (out of 100). Further right means it costs more.
Try itpick a kind of job, then drag the budget
Kind of job
Across the bottom
Score axis
$0$1$10$100
Loading the results…
Cost runs on a log scale: each step is 10 times the last, so the axis can't start at $0. Filled dots passed every task of this kind. The gold ring marks the cheapest of those.
What all 25 jobs cost, per model
One run of every task, at list prices. The bars start at $0 and are drawn to scale. Some are so short they look like a typo. They aren't.
Every task, every model
Try itopen a task to see the prompt, each grade and the actual answers
✓ passed✕ failedStretch = added later to separate the models. It didn't.
Run it again: how much is luck?
The six shakiest tasks ran two more times per model, with the same prompt and settings. Each cell shows passes out of 3 on the checks code can score (word limits, exact quotes, the right names and dates, the right answer).
Who graded what
Code graded 17 tasks on its own. It ran the tests, ran the SQL on a real database, compared the JSON field by field, checked quotes against the source and counted words. No opinions.
Two AI judges helped on the other 8. GPT-6.1 Sol (from OpenAI) and Claude Sonnet 5.5 (from Anthropic) graded each answer against a written checklist, without being told which model wrote it. Each answer got the average of the two, so every maker's answers faced one judge from their own family and one from outside.6 A pass needs the hard checks to pass and an average of at least 7 out of 10. For the two honesty traps, both judges had to say pass.
The judges' instructions, word for word
A second look at 12 answers
The AI assistant that built this re-read 12 answers against the checklist. Where it disagrees, the official grade still stands, and the disagreement is shown here, not hidden. No person has graded these answers yet.
Method, models and prices
All calls ran on Nov 1, 2026, between 14:47 and 14:53 UTC. Every model got the same prompt as one message. Code tasks also got a one-line instruction to reply with code only. Every model ran at its low thinking setting, the cheap default most people try first, with room for up to 8,000 tokens of answer (a token is about three-quarters of a word). No retries were needed.
The answers you can open are the models’ own words, with one small change: where a model labelled a to-do or a status as unfinished, this page uses plainer words (“write”, “first version”). That touched 11 answers and one task prompt. Every grade was made on the original text.
Cost = tokens read × the input price + tokens written (including hidden thinking) × the output price, at standard list prices, with no batch or cache discounts.
"Open weights" means anyone can download the model and run it.4 The quotes task used a Wikipedia article as its source text.5 Prices were read on Nov 1, 2026.
Wanted, but out of reach that day
Google Gemini (gemini-4-argon, gemini-3.8-flash): the key this setup used was over its limit that day.
xAI Grok: no credit left on the account this setup used.
DeepSeek V4 Pro (DeepSeek's own API): no balance on the account this setup used, so its smaller Flash model ran on Fireworks instead.
GLM-5.3 (Zhipu / Z.ai, via Fireworks): it timed out every time.
Groq-hosted models: the account this setup used was restricted.
So: no Google and no xAI model this round. That's a gap, not a verdict.
What this is not
Not a leaderboard, and not a verdict on these models in general. It is 25 tasks written for this page, one run each (six of them three runs), one prompt style, low thinking settings, one day. It tests whether a model does a specific everyday job, and what that costs.
Sources (6)
Anthropic, “Pricing” (per million tokens in / out: Claude Fable 5.1 $10 / $50; Claude Opus 5.5 $4 / $20; Claude Sonnet 5.5 $2 / $10; Claude Haiku 5.5 $0.10 / $0.50), read Nov 1, 2026. docs.claude.com/en/docs/about-claude/pricing
OpenAI, “Pricing”, standard tier (GPT-6.1 Sol $2 / $10; GPT-6 Luna $0.10 / $0.50), read Nov 1, 2026. platform.openai.com/docs/pricing
Lianmin Zheng and others, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, arXiv:2306.05685, on how AI judges can favour their own family. arxiv.org/abs/2306.05685
Can AI answer my email?What should never be sent without a person reading it?
A model reads 69 messages for a made-up company and writes a reply where one is needed. Eight rules sit between each reply and the send button. Open any message to see what came in, what the model wrote, which rule stopped it, and what a second model thought of it.
Cellarbook, Inc. and everyone in it are made up. Nobody real was promised a refund.
The short version
–
messages with a planted trap that the rules held back for a person
–
replies with no planted trap that the rules held back anyway
–
replies that claimed something was done when nobody did it (first try → after the fix)
–
per message to write, check and grade the reply (an estimate, not a bill)
How much editing would each reply need?
A second model graded all 69 messages against a written checklist. For the messages where no reply was written, it graded the choice not to reply. First try against the final try, same messages.
Explore the test
Try itpick a part, then open any message
Show
Sources (7)
All 69 messages with their replies, rule flags and grades, from the run on Nov 1, 2026: .
The eight rules, with the examples each one caught: rules.json.
Every instruction given to the model, all three versions, with each run's totals: prompt_log.json.
Brewers Association, “A Year of Correction for Craft Beer, With Early Signals of Recovery”: brewersassociation.org (read Nov 1, 2026). Used in the Research part.
Brewers Association, “2026 Midyear Report: Pressure Persists as Signs of Stability Emerge”: brewersassociation.org (read Nov 1, 2026).
Brewers Association, “National Beer Sales & Production Data”: brewersassociation.org (read Nov 1, 2026).
Brewers Association, “The 2025 Year in Beer”: brewersassociation.org (read Nov 1, 2026).
In these three files, the word for a reply waiting to be sent has been changed to “reply”, to match this page. Nothing else was changed. Built with AI tools. The costs are estimated from word counts at assumed list prices, not from a bill; the whole test, three runs included, came to roughly $9 that way.
Could this market ever hold a trillion-dollar company?
Start from the answer and work backwards. What does a trillion dollars of company value need in profit, in revenue, and in market? Then size the market two ways and look at the gap between them. The gap is usually where the truth is hiding.
What this is: a calculator for a question founders love to answer too fast, with the real early numbers of four companies that did answer it.
Every number on this page is either sourced (numbered, at the bottom), computed from sourced numbers, or labelled an assumption. Market values are from the Oct 30, 2026 close. This is arithmetic about markets, not a forecast for any company, and not investment advice.
Spreadsheets have been optimistic about less.
The calculator
Try itpick an example, or type your own numbers
Pick a starting point, then move anything. Each input says where its number came from; once you touch it, it says it's yours.
Loaded:Start here · a blank example
The mathscompany value = yearly profit × P/Erevenue = profit ÷ net marginmarket needed = revenue ÷ your share
1 Work backwards from the value
Company value you're testing:$1T
AssumptionThe goal being tested.
Earnings multiple (P/E):25×
SourcedAbout where the S&P 500 trades now (25.86 at the Oct 30, 2026 close); its long-run mean is 16.24.10
Net profit margin at maturity:20%
AssumptionBetween the whole U.S. market (9.7%) and the software sector, system and application (25.5%).11
Share of the market you win:30%
AssumptionA dominant but not monopoly share.
Net margins, U.S. averages 11 All U.S. companies 9.74% · Retail (general) 5.61% · Software (system & application) 25.5% · Semiconductors 30.4% · S&P 500 P/E now 25.86, long-run mean 16.24 10
To be worth $1T at 25× earnings, a company needs $40B of profit a year. At a 20% net margin, that's $200B of revenue a year. If that is 30% of its market, the market has to be about $667B a year.
Profit a year$40B
Revenue a year$200B
Market needed$667B
2 Size the market two ways
Top-down: start from the whole market
The market today, in dollars a year:$500B
AssumptionA placeholder market. Put yours in.
Top-down: a $500B market × 30% share = $150B a year, 75% of what $1T needs.
Bottom-up: start from one customer
Customers who could buy:100M
AssumptionPlaceholder.
Price per purchase:$50
AssumptionPlaceholder.
Purchases per customer per year:12
AssumptionMonthly.
Bottom-up: 100M customers × $50 × 12 a year = a $60B market. At 30% share: $18B a year, 9% of what $1T needs.
Log scale: each step is ten times the last. Dashed line = the market needed. The gold bracket is the gap between your two estimates.
The two estimates disagree by 8.33×. Top-down is the bigger one. The gap is the interesting part: one of these inputs is wrong, and finding out which is the real research.
3 How long would it take?
Revenue today:$10M
AssumptionPlaceholder: a young company's revenue today.
Revenue growth per year:50%
AssumptionPlaceholder. For scale: the four examples below averaged 29% to 45% a year for 14 to 29 years.12345678
How fast growth fades (per year):3%
AssumptionGrowth shrinks by 3% of itself each year.
Starting at $10M and growing 50% a year, with growth shrinking by 3% of itself each year, revenue reaches $200B in about 38.1 years.
Revenue on a log scale. Real companies' growth slows; the fade input is how fast.
The honest readout
Not as sized.
Top-down says the market is 75% of what's needed; bottom-up says 9% of what's needed.
Plausible only if the market is closer to the bigger estimate, or grows, or you win more than 30%.
Your two methods disagree by 8.33×. Before believing either, find out which input is wrong: it's usually the customer count or the price.
About 38 years is longer than most companies stay fast-growing (and longer than most pitch decks stay accurate). Plausible only if growth holds for decades.
This is arithmetic about a market, not a forecast for any company and not investment advice.
Four trillion-dollar companies, before they were
Each of these companies told the SEC what its market was when it first sold shares to the public. Here is what each filing said, next to what the company became. Each “then” row names its source: most come from the prospectus itself, and where they don’t, the row says so. Every “now” number is from the latest annual report. Tap “Load” to run any of them through the calculator.
Amazon · 1997 IPO prospectus
Then (each row names its source)
Revenue, 1996
$15.7M
net sales; net loss $5.8MSource: Amazon's prospectus 1
Market it pointed at
$82B
worldwide book sales, 1996 (U.S.: $26B)Source: stated in Amazon's prospectus (Euromonitor's estimate) 1
Bigger market it named
$95B
all Web commerce, forecast for 2000Source: IDC's forecast, quoted in Amazon's prospectus 1
Value at IPO price
≈$429M
$18 × 23.9M sharesSource: price and shares from Amazon's prospectus; computed 1
Mentions of “cloud” or “web services”
0
in the whole prospectusSource: counted in Amazon's prospectus 1
Amazon's 2025 revenue is about 8.7 times the entire worldwide book market its prospectus described, and more than 27 times the U.S. one (nominal dollars, no inflation adjustment). Measured against the market in the filing, a trillion-dollar Amazon was impossible: at today's margin and multiple it needs about $276B a year, more than three times all the books sold on Earth in 1996. Books were the door, not the room.
Nvidia · 1999 IPO prospectus
Then (each row names its source)
Revenue, 1997
$29.1M
net loss $3.6M; $92.7M in the 9 months to Oct 1998Source: Nvidia's prospectus 2
Market it pointed at
4.9M → 138M chips
3D graphics processors sold, 1997 → 2001 forecastSource: Mercury Research, quoted in Nvidia's prospectus 2
Market in dollars
not given
the filing counts chips, not dollarsSource: Nvidia's prospectus; the lab's dollar figure uses an assumed $25 a chip 2
Value at IPO price
≈$343M
$12 × 28.6M sharesSource: price and shares from Nvidia's prospectus; computed 2
Mentions of “data center”
0
in the whole prospectusSource: counted in Nvidia's prospectus 2
Nvidia's prospectus sold a PC gaming chip. Gaming is still there ($16.0B), and on its own it is already about 550 times the company's 1997 revenue. But 90% of revenue now comes from data centers, a phrase the prospectus never uses. At today's margin and multiple, $1T needs about $44B a year; from the chips in the filing alone, that means selling every 3D chip forecast for 2001 at an average of about $320 each. The market that made it a trillion-dollar company was not in the filing at any size. The early market didn't need to be big. It needed to be a training ground for the thing that would be.
Google · 2004 IPO prospectus
Then (each row names its source)
Revenue, 2003
$1.47B
net income $105.6M; 71% from the U.S.Source: Google's prospectus 3
All U.S. internet ads, 2003
$7.3B
keyword search was 35% of itSource: IAB/PwC 2003 report, not Google's filing 12
How the filing sized its market
it didn't
“an immature industry that has undergone rapid and dramatic changes”Source: Google's prospectus 3
Value at IPO price
≈$23.1B
$85 × 271.2M sharesSource: price and shares from Google's prospectus; computed 3
Google's 2025 revenue is 55 times the whole U.S. internet ad market of 2003, the year before it listed. Google was not just a share of that market. It was a big part of why the market grew 40-fold. The prospectus sensibly declined to put a number on its market. Top-down sizing breaks when the company is the reason the market exists.
The counter-example. Facebook's market was already enormous when it filed. Advertising in 2010 was $588B, more than four times the ~$128B a year it now needs for $1T at today's margin and multiple. The question was never whether the market could hold it. It was whether one company could take about a fifth of it. Then its ad revenue went from $3.2B in 2011 to $196B in 2025. For Facebook, the hard part of the trillion-dollar test was share.
What the four have in common: in three of the four, the market as sized when they went public (Amazon's stated $82B, Nvidia's chip count at an assumed price, and the $7.3B U.S. ad market of 2003, which Google's filing didn't size) couldn't have held a trillion-dollar company, even at 100% share. Each of those three outgrew the market it started in. That's a pattern in four winners, not a rule, and it's exactly the kind of story survivorship produces (see Limits).
A candidate today: AI agents for work
An illustration, not a prediction about any company. Suppose someone builds software agents that do parts of desk jobs, sold per seat. How big is “the market”? It depends entirely on which market you mean:
seats: assumption; price: Microsoft 365 Copilot list price 1819
At the software sector's average net margin (system and application, 25.5%) and today's market multiple (25.9×), $1T needs about $151B of revenue a year: a $505B market at 30% share. That's 17.3× the category Gartner tracks today, 5.61× the seat-based bottom-up, and about a third of all software spending. So the honest verdict is: not as a category; only if agents take budget from somewhere much bigger (other software, outsourced services, or work people are paid to do now), which is the claim that would need evidence.
Every input above is sourced or labelled. Seat count and adoption are assumptions; the $30 is one company's list price, not a market average. Not investment advice.
What this can't tell you
The list of trillion-dollar companies is a survivor list. 15 companies were valued at $1T or more on Oct 30, 2026 9. Thousands of companies told the SEC their market was huge. Across U.S. stocks since 1926, the best-performing 4% of listed companies explain the stock market's entire net gain 20. Four winners' prospectuses teach you what winning looked like, not how often it happens.
Market value isn't value created. A trillion-dollar valuation is a price, set daily by people guessing the future. The calculator treats it as profit × multiple, which is a simplification: the multiple moves with interest rates and mood.
The multiples here are snapshots. “Computed” P/Es divide one day's market value by one fiscal year's profit. They aren't the trailing P/E a finance site quotes.
Dollars aren't adjusted for inflation. 1996 dollars and 2025 dollars are compared as written.
Top-down and bottom-up both lie, in different directions. Top-down borrows someone else's definition of the market. Bottom-up multiplies three guesses together, so its errors compound.
What's worth shipping, and why
Use this as the first ten minutes of looking at any idea, including your own. Not to get a number, but to find the one assumption the whole thing rests on. When the top-down and the bottom-up disagree by ten times, go and talk to customers before believing either. The worked examples are the reminder: the market in the filing is rarely the market that matters.
What's next
Add a fifth and sixth case from the same era of filings: Microsoft's and Apple's early prospectuses predate EDGAR, so they need scanned originals.
A “market grows too” input, so top-down can change over time instead of staying fixed at today's size.
A shareable link that carries your inputs.
Sources (24)
Filings are on SEC EDGAR. Everything was read on Nov 1, 2026. “Computed” and “counted in” mean the calculation or count was done for this build, from the cited numbers.
Amazon.com, Inc., IPO prospectus (Form 424B1), filed May 15, 1997, SEC EDGAR. Net sales 1996 $15,746K, net loss $5,777K; ~340,000 customer accounts and >$32M sales through Mar 31, 1997; Euromonitor: U.S. book sales ~$26B and worldwide ~$82B in 1996; IDC: Web commerce $5.4B annualized Dec 1996, $95B forecast for 2000; IPO price $18.00; 23,858,702 shares after the offering. www.sec.gov/Archives/edgar/data/1018724/000089102097000868/0…
NVIDIA Corporation, IPO prospectus (Form 424B4), filed Jan 22, 1999, SEC EDGAR. Total revenue 1997 $29,071K, net loss $3,589K; nine months to Oct 25, 1998 revenue $92,700K; Mercury Research: 4.9M 3D graphics processors sold worldwide in 1997, 138M forecast for 2001; IPO price $12.00; 28,595,976 shares after the offering. www.sec.gov/Archives/edgar/data/1045810/000101287099000192/0…
Google Inc., IPO prospectus (Form 424B4), filed Aug 19, 2004, SEC EDGAR. Revenues 2003 $1,465,934K, net income $105,648K; U.S. 71% of 2003 revenues (advertiser billing address); 65 million = monthly U.S. visitors (the prospectus's correction of a magazine's 'daily' figure); 'online advertising, which is an immature industry'; IPO price $85.00; 271,219,643 shares after the offering. www.sec.gov/Archives/edgar/data/1288776/000119312504143377/d…
Facebook, Inc., IPO prospectus (Form 424B4), filed May 18, 2012, SEC EDGAR. Revenue 2011 $3,711M (advertising $3,154M), net income $1,000M; 901M monthly active users at Mar 31, 2012; worldwide ARPU $1.21 in Q1 2012; IDC: worldwide advertising spending $588B in 2010; IPO price $38.00; 2,138,085,037 Class A + B shares after the offering. www.sec.gov/Archives/edgar/data/1326801/000119312512240111/d…
NVIDIA Corporation, Form 10-K for fiscal 2026 (year ended Jan 25, 2026), filed Feb 25, 2026. Revenue $215,938M, of which Data Center $193,737M and Gaming $16,042M; net income $120,067M; fiscal 2024 revenue $60,922M. www.sec.gov/Archives/edgar/data/1045810/000104581026000021/n…
Meta Platforms, Inc., Form 10-K for fiscal 2025, filed Jan 29, 2026. Revenue $200,966M, advertising $196,175M; net income $60,458M; revenue 2023 $134,902M; 3.58B daily active people (Dec 2025). www.sec.gov/Archives/edgar/data/1326801/000162828026003942/m…
companiesmarketcap.com, 'Largest companies by market capitalization', read Nov 1, 2026 (prices as of the Oct 30, 2026 close). NVIDIA $4.941T, Alphabet $3.420T, Amazon $2.598T, Meta $1.572T; 15 companies listed at $1T or more. companiesmarketcap.com/
multpl.com, 'S&P 500 PE Ratio' (trailing 12-month as-reported earnings; historical data from Robert Shiller). Current 25.86 at the Oct 30, 2026 close; long-run mean 16.24, median 15.08. Read Nov 1, 2026. www.multpl.com/s-p-500-pe-ratio
Aswath Damodaran, NYU Stern, 'Operating and Net Margins by Sector (US)', data as of January 2026. Net margin: total market 9.74%; software (system & application) 25.49%; semiconductor 30.45%; retail (general) 5.61%. pages.stern.nyu.edu/~adamodar/New_Home_Page/datafile/margin.…
Gartner, 'Gartner Forecasts Worldwide AI Spending to Grow 49.5% in 2026' (press release, Sep 16, 2026): 'AI Agents and Assistants' $29,219M in 2026 and $65,472M in 2027; AI software $461,637M in 2026; total AI spending $2,670,460M in 2026. www.gartner.com/en/newsroom/press-releases/2026-09-16-gartne…
Gartner ThinkCast, Mar 22, 2022: 'Gartner estimates that there are 2.7 billion frontline workers, more than twice the number of desk-based workers.' thinkcast.libsyn.com/2022/03
Hendrik Bessembinder, 'Do stocks outperform Treasury bills?', Journal of Financial Economics 129(3), 2018: the best-performing 4% of listed U.S. companies explain the net gain of the entire U.S. stock market since 1926. ideas.repec.org/a/eee/jfinec/v129y2018i3p440-457.html