Fictional company (Kestrel Supply Co.). SYNTHETIC data. ILLUSTRATIVE prices. Proof of concept.

AI Spend and Value Tracker: case study

Fictional company (Kestrel Supply Co.). SYNTHETIC and ILLUSTRATIVE data. Proof of concept.

The problem

Companies buy AI seats quickly and review them slowly. A usage report shows who used a tool, not what the work was worth. A leader still has to decide which tools to keep, which to trim and which to question. I wanted to build that decision for a made-up company, show my working, and be clear about how sure the answer is.

What I built

I built a generator that makes 52 weeks of synthetic usage for 80 staff, with problems hidden in it on purpose. I built a seat classifier that sorts each of the 120 seats into a state. I built a range model that turns hours saved into a return, always as low, base and high. A set of rules gives each of the six tools one answer, and says how sure it is. I built a static page where a reader can change the assumptions. It draws the seats as stools in a members' cafe. I also built a one-page memo. It is a proof of concept on synthetic data. It has not been used on real data.

What I expected

I wrote these expectations before I saw the page. They are in preregistration.md, committed on 2026-10-26 in commit cd2e04a, and copied here without a change.

Written 26 October 2026, after I had seen the verdict table and both challenge rounds.

1. What will confuse a first-time viewer most on the dashboard?

Terminology. How to read the dashboard if they don't know the terms. Too many sidebars and menus to click into. They'd need a clean, clutter free dashboard with adjustments that is all on one page - that can be scrolled up and down.

2. Will the memo number-trace fail on its first run, and why?

Yes. If people round up - for example, “about $4,300” when the file says $4,285. They calculate new figures “roughly a quarter of seats” that nothing stores. They reword. They aren't meticulous and don't enter data verbatim.

3. If a reviewer sets hours saved to 0.5 a week, which verdicts do you expect to move?

Dollar amounts.

4. On a real company’s data, what would break in this method first?

Messy exports. Date formats, missing user IDs, one person with two emails.

Out-of-date HR records. The leaver check depends on the roster being right.

Seats that don’t map to one person. Team accounts, contractors and service accounts.

Change management. Reclaiming seats feels unfair to the people who lose them.

5. What would change your mind about the transcriber’s “investigate” verdict?

Moving to keep: people read the emailed summaries and never need the full transcript, so “opened” understates the value.

Moving to cut: interviews show nobody uses it, and nothing else depends on it.

Something else entirely: a records or compliance requirement means you have to keep it regardless.

The numbers

Every figure below is read from the file named beside it by a script. All money is for one month, and prices are ILLUSTRATIVE.

Seats classified. The classifier scored 120 seats (results/recovery.json). 116 are paid for and 4 were already cancelled (results.json). Of the 116 paid seats: 85 active, 13 light use, 9 idle, 2 never used, 3 owner has left, 4 new starter (results.json).

Money that is not earning its keep (results.json). Idle seats (certain): $480 to $520 a month, from 13 to 14 of 116 seats. Consolidation savings (a choice): $0 to $315 a month, up to 9 seats. The two are never added together.

An answer for each tool (results.json). Break-even hours are the hours a week each active user must save for the tool to pay for itself. They run from the dearest hour to the cheapest. An hour of time costs $45, $65 or $90 (low, base, high), and a month is 4.33 weeks (assumptions.yaml). The hours saved are my guesses.

Recovery of the problems I planted (results/recovery.json). Precision is the share of what was flagged that was planted. Recall is the share of what was planted that was flagged. 6 Retail seats and 4 Finance seats were set as traps, because their quiet spells are normal. None was flagged idle (0 trap failures, 0 edge-case failures).

Hard cases, before and after (results/challenge_baseline.json, results/challenge_after.json, results/challenge2_baseline.json and results/challenge2_after.json). The hard cases were suggested by Claude in chat and approved by me. I ran the checker on them first without changing it, then fixed what it missed. The fixes are in the log below.

What failed

The entries below are copied word for word from the failure log. Every entry is in the technical log. The page has a short version in its "What failed" part (site/index.html#failed). After each entry, I say what it taught me and which of my expectations was wrong.

2026-10-26 07:23 Phase 2: the traps did not bite my first classifier (no failure to report)

• What happened: The first run of the cycle-aware classifier recovered every planted problem with precision and recall of 1.0, and passed the Retail and Finance trap tests and the edge-case tests. No check failed. The same trap tests do fail for a flat 30-day rule (all 10 trap seats flagged idle), for a classifier with no business calendar, and for classifiers that ignore leave dates or start dates.
• What I expected: The brief said to expect the traps to bite a first version and to log what happens.
• Why (root cause, or best hypothesis labelled HYPOTHESIS): HYPOTHESIS: I wrote the cycle logic before the first run, and I wrote the generator, so I knew where the traps were and which calendar entries described them. A perfect score is therefore partly built in, as rules.yaml says. It is a test of the code, not of the world.
• What changed: Nothing. I added over-forgiveness tests (a Finance seat that misses the latest close is idle, an idle seat moved into Finance stays idle, a Retail seat also used off-season is not seasonal) so the cycle rule cannot pass by forgiving everything.
• Was any check or threshold changed? No. Thresholds in rules.yaml were set before the first run and not changed afterwards.

What it taught me, and which expectation was wrong. The Retail and Finance traps did not trip my first classifier. It scored perfectly, which told me little. I wrote the data and the checker, and set the limits with the data in front of me. A perfect score on my own data tests the code. It does not test the world.

2026-10-26 07:44 Challenge baseline: most of the passes prove little

• What happened: 17 of 28 assertions pass at baseline, and 8 of those are the controls. Of the 9 case passes, most are not evidence that the classifier handles the situation: C1-returned, C2-active, C2-one-user, C2-handover, C3-missed, C3-irregular, C4-active, C4-department and C4-one-seat all pass because the classifier counts seats rather than labels, never flags anything, calls every quiet seat idle, or happens to keep the last roster row. Only the controls are a real test of the unchanged behaviour.
• What I expected: A pass to mean the classifier handles the case.
• Why (root cause, or best hypothesis labelled HYPOTHESIS): HYPOTHESIS: I wrote the cases and the assertions, including the negative controls, around what a fix must not break. A control cannot fail for a classifier that never does the thing it guards against. They matter after the fixes, not before.
• What changed: Nothing. The baseline is reported as 17 of 28, with this caveat.
• Was any check or threshold changed? No.

What it taught me, and which expectation was wrong. 17 of 28 sounds respectable, but 8 of those passes were controls, which only show the checker works. The hard cases told the real story. I now ask what a pass could have missed before I trust a score.

2026-10-26 07:44 Challenge case 5 (missing telemetry): a documented blind spot is read as idleness, and evidence strength does not drop

• What happened: The seat of a known mobile user whose tool cannot see mobile use (SEAT-SUP-C050, last logged use 105 days ago, telemetry note present) is classified idle and counted as idle spend. Three assertions missed: status should be "unknown, telemetry gap" (got "idle"), the seat is counted as idle spend (1 where the key says 0), and the Support-desk AI add-on verdict keeps evidence strength High with no reason recorded.
• What I expected: Not idle, status "unknown, telemetry gap", and evidence strength for that tool's verdict dropped with the reason stated (challenge/CASES.md, Case 5).
• Why (root cause, or best hypothesis labelled HYPOTHESIS): The telemetry notes file is never loaded. The classifier treats silence in the log as evidence of non-use for every tool, and evidence strength is computed only from verdict flips and data completeness.
• What changed: The telemetry notes file is now loaded (optional; the main dataset has none). A quiet seat, or one never seen, on a tool with a documented gap gets the status "unknown_telemetry_gap" instead of idle or never activated, with a recommendation to ask the user or administrator. It is not counted as idle spend. Seats explained by a cycle stay active. The tool's evidence strength drops one step (High to Medium, Medium to Low), and each tool's result now carries evidence_strength_reasons that say why, for every rating. Result: all three Case 5 assertions pass (Support-desk AI add-on goes from High to Medium with the telemetry reason). Tests show the note is the cause: without it the seat is idle again and the tool is High. The rule applies to every quiet seat on the tool, not only the user named in the note, because the log cannot tell any of them apart. The note's affected_users list is only used to add a line to the reason.
• Was any check or threshold changed? No threshold added or changed and rules.yaml is untouched. The one-step drop in evidence strength is a design choice, not a tunable threshold. No expected answer was changed.

What it taught me, and which expectation was wrong. A silent log is not the same as an idle person. When a tool cannot record some of its use, a quiet seat proves nothing. My first version called those seats idle and stayed confident. I now treat missing records as a reason to be less sure, not as evidence.

2026-10-26 14:34 Challenge cases 1 to 8 were labelled as Chris's, and Chris did not write them

• What happened: challenge/CASES.md, its tests and the round 2 generator header said cases 1 to 8 were authored by Chris, and my report at Gate 1c and the round 2 commit message said cases 6 to 8 were "Chris's". Chris has corrected this: all eight were proposed by Claude in chat and approved by Chris.
• What I expected: To be able to tell, from the file, who wrote each case, so that "independent of the classifier's author" could be judged honestly.
• Why (root cause, or best hypothesis labelled HYPOTHESIS): I treated the cases as Chris's because they arrived as instructions in his messages. I did not check who had written them. HYPOTHESIS: the cases came out of an earlier chat with Claude, which would make the independence of the challenge weaker than the files said: the cases and the classifier both trace to Claude.
• What changed: CASES.md now says "Claude in chat, approved by Chris" for cases 1 to 8, and explains the three allowed labels. "Chris" is reserved for a case Chris wrote, and none is marked that way. Variants and controls remain "Claude Code". The round 2 generator header and one test comment were corrected. The test that checked authorship used to assert that cases 6 to 8 were "Chris"; it now asserts the new label. The original wording survives in the git history (commits 950a5fd and 25d6cec) and in the Gate 1c report. Claims of independence in the earlier FAILURES entries should be read with this correction.
• Was any check or threshold changed? One test assertion about authorship labels was changed, because the label it asserted was wrong. No detection check, expected answer or threshold was changed.

What it taught me, and which expectation was wrong. The hard cases were labelled as mine, but Claude suggested them in chat and I approved them. That matters, because who writes a test changes what it proves. Cases I only approved are not independent evidence either. I now say plainly who did what.

2026-10-27 06:14 First version of the page failed Chris's review

• What happened: Chris reviewed the Phase 4 page in a browser and it failed his review. This is a design review finding, not a code failure: every check and test passed on that version (commit 68e1f98). His findings, as given: the page is about 11,000 px tall (I measured 11,467 px at a 1,024 px window); there are 22 sliders on the main data; the copy is heavy with jargon; the text cells are large; there is no navigation; and details about the fictional company are mixed into the main flow.
• What I expected: The preregistration asked for one clean scrolling page with the adjustments on it and the terms explained. I read "one page" as "everything open on one page", and read "clean" as "no menus", and I checked the page against my own tests (every chart has a table, no network, page agrees with the rules) instead of against how a first-time reader would move through it.
• Why (root cause, or best hypothesis labelled HYPOTHESIS): HYPOTHESIS: I built to the checks I had written. None of them measure length, reading difficulty, how many controls are on screen at once or whether a reader can find their place, so the page passed everything and still failed the person it was for. I also kept every table and card open by default and exposed all 22 sliders at once, which I treated as completeness and Chris reads as clutter. The page also had no WCAG review at all, and no check that measured any of this.
• What changed: Nothing yet in this entry. The redesign, a WCAG 2.2 AA pass, a plain-English rewrite, worked examples and the new checks follow on the same branch in small commits, and each will log what it found.
• Was any check or threshold changed? No.

What it taught me, and which expectation was wrong. My first version of the page was long, dense and full of jargon. I reviewed it and sent it back. A correct page that a newcomer cannot read does not help a decision. The redesign cut the length, the sliders and the jargon. I now judge a page by whether a newcomer could explain it after a short read.

2026-10-28 13:15 The animation rows were marked verified before Chris could see the animation

• What happened: On 2026-10-28 I marked the animation rows in docs/ACCESSIBILITY.md as "verified by Chris, Edge" (commit c8e72be, now on master). I did that on the word that he had checked the page in a real Edge window and the animation passed. When he looked, Windows animation effects were off, so Edge reported "reduced motion" and the page correctly showed only the still picture, with no Play button. He could not see the animation until he turned animation effects on. So the doc claimed a check that had not happened yet. I also marked "no flashing" and "the button worked" on the same basis.
• What I expected: A row marked verified to rest on something that person had actually watched.
• Why (root cause, or best hypothesis labelled HYPOTHESIS): I accepted a summary of a test as the test. I also had a warning I did not use: every browser on my own machine, and the in-app pane, reported reduced motion, which meant a real Edge window on a Windows machine with animation effects off would show the still picture too. I told Chris this only in the log of the earlier failure, not when I wrote the rows.
• What changed: The section now says what happened. Rows are marked "verified by Chris, Edge on Windows, 2026-10-28" for what he then watched: it starts still with a Play button, Play and Pause, up to 4 walkers, the marks following the hours slider, the card following the hourly cost, no flashing and no glitches. Reduced motion is marked verified too: he saw the still picture, no Play button and the note with effects off. Rows he has not seen stay "to be verified by Chris by hand": colour and shapes, the picture's contrast and forced colours, its text alternatives and the status message. The tally of 55 criteria is unchanged. The correction is on master and on branch phase-5-memo.
• Was any check or threshold changed? No. A mark of "verified by Chris" is a statement in a document, and no check reads it.

What it taught me, and which expectation was wrong. I told Claude Code the animation had passed my tests. It had not. Windows had animation effects switched off, so I had only seen the still picture, and the log recorded "verified" for something I had not watched. The page's own note gave it away. Now "verified" means only what I have seen myself.

What the method cannot tell you

Activity is not value. A busy tool can be low value. A quiet tool can be critical. Return figures rest on my assumptions. The interactive page lets you change them.

What I would do differently

I would run this against real data, with the right approvals and a real administrator who can act on a seat. I would interview heavy users, and the teams behind quiet seats, before trusting any verdict. I would have someone else write the hard cases from the start, so the person who wrote the classifier does not also mark it. I would add a check that looks at the finished page from day one, because layout problems reached me before any check looked.

Limits

Synthetic data with problems I seeded myself. The analysis finds what I planted, which is a test of the code, not of the world. Not a production tool. The classifier, the data, the checks and the dashboard were all built by one person with AI help.