AI evaluation
An experiment: how often does an AI assistant answer questions about this data correctly with raw data, with tools, and with tools plus the semantic layer?
The question
Does a governed semantic layer make an AI assistant's answers about this data more reliable? The same 30 questions are put to the same model under three conditions, and every answer is graded in code against the correct value, computed with the app's own metrics from a frozen snapshot of the data (taken 2026-10-16, last logged day 2026-10-16). No AI judges the answers.
- Raw data
- The public data document pasted into the prompt, with a short note on its format. No tools.
- MCP tools
- The MCP server's tools on the snapshot, with the semantic layer switched off: no definitions, no
explaintool, no provenance. - Tools + semantic layer
- The same server with the semantic layer: every definition in the prompt, the
explaintool, and provenance (which definitions a number comes from) in every answer.
Questions come in three kinds: lookup (a stored fact), metric (a defined figure where a plain reading usually works) and definition (where the precise definition matters: lifetime cost per wear, unworn garments, ownership periods, Sportswear left out of daily cost…).
Results
claude-sonnet-5-5
| Condition | All questions | lookup | metric | definition | Avg tokens | Avg tool calls |
|---|---|---|---|---|---|---|
| Raw data in the prompt | 23 % 7/30 | 67 % 4/6 | 27 % 3/11 | 0 % 0/13 | 176,708 | 0.0 |
| MCP tools | 97 % 29/30 | 100 % 6/6 | 100 % 11/11 | 92 % 12/13 | 52,574 | 2.7 |
| MCP tools + semantic layer | 100 % 30/30 | 100 % 6/6 | 100 % 11/11 | 100 % 13/13 | 54,754 | 2.4 |
Findings: first run (16 October 2026, Claude Sonnet 5.5, each question asked once)
- Raw data in the prompt: 7 of 30 correct (23 %). With the whole log in the prompt (about 177,000 tokens per question), the model has to add up thousands of records by hand. It got 4 of 6 lookups and 3 of 11 metrics right, but none of the 13 questions where the precise definition matters. In 16 of its 23 misses it used up the 4,096-token answer limit without reaching an answer. The others were miscounts (407 wears instead of 408) or figures defined differently (turnover 371 % instead of 50 %; Sportswear named instead of Shirts as the most expensive category).
- MCP tools: 29 of 30 correct (97 %), with less than a third of the tokens. Most of the gain comes from governed computation. The server computes every figure with the app's own tested metrics, so the model doesn't have to re-derive definitions; it only has to pick the right tool and date range.
- MCP tools plus the semantic layer: 30 of 30 correct (100 %), with about 4 % more tokens. The single difference was "What was the cost per wear of the North Bend black vest in 2024?". Without the definitions, the model saw the lifetime figure of €0.05 but chose to divide the price by the 2024 wears instead (€0.26). With the semantic layer, it applied the definition: cost per wear is always a lifetime figure (decision D-2). This is exactly the kind of interpretation error the semantic layer is meant to prevent.
- Caveats. This is one run with one model. The difference between tools and tools plus semantics is a single question, so it is not statistically meaningful. The raw-data condition was held to the same 4,096-token answer limit as the others. Next steps: repeated runs (
--repeat), a higher answer limit for raw data, and more questions where the tools leave interpretation to the model.
Questions and correct answers
| Question | Kind | Correct answer | Metrics |
|---|---|---|---|
| What was the purchase price of the garment named "North Bend black vest", in euros? | lookup | 50 EUR (± 0.01) | |
| In which year was the "Barbour Classic Tartan navy" bought? Answer with the year. | lookup | 2025 (± 0) | |
| Which garments were worn on 15 June 2025? List their names, separated by semicolons. | lookup | Marmot crossover shorts blue; Ballin Amsterdam; Pier One black 01 (43-46); Nike M Tanjun black 2; Zalando Essentials black 27; Jack & Jones Jacsummer Mid blue; SOC Tech black melange S 02; Under Armour Vanish woven shorts M; Contra regular boxer Khaki green 03; Tesma Coolmax 03; Nike Air Zoom Vomero 15 | |
| How many wears in total were logged in March 2025 (one garment on one day = one wear)? | lookup | 408 (± 0) | |
| How many different garments were worn at least once in 2025? | lookup | 250 (± 0) | |
| How many garments were divested (got a divest date) in 2024? | lookup | 2 (± 0) | |
| How many garments were active (owned) on 1 October 2026? | metric | 316 (± 0) | Active items |
| What was the total inventory value (sum of purchase prices of active garments) on 31 December 2025, in euros? | metric | 15589.2 EUR (± 1) | Inventory value |
| What was the inventory value of the Jackets and hoodies category on 1 January 2020, in euros? | metric | 1426 EUR (± 1) | Inventory value |
| Which garment was worn the most times in 2025? Give its name. | metric | Diesel black matte 02 | Times worn |
| How many times was the "North Bend black vest" worn in 2023? | metric | 82 (± 0) | Times worn |
| In 2025, on what share of days was at least one pair of shoes (category Shoes) worn? Answer as a percentage. | metric | 89.041 percent (± 0.5) | Days used |
| How many garments tagged Hanko were active on 1 October 2026? | metric | 49 (± 0) | Active items |
| What share of the garments active at the end of 2025 were bought second hand? Answer as a percentage. | metric | 5.994 percent (± 0.5) | Second hand share |
| What was the average weight of an outfit (garments worn per logged day, those with a known weight) in 2025, in kilograms? | metric | 3.928 kg (± 0.02) | Average outfit weight |
| How many kilograms of wool did the wardrobe contain on 30 September 2026 (garments active then, weight × material composition)? | metric | 1.862 kg (± 0.1) | Fibre mass |
| What share of the mass of clothes worn in 2025 (each wear counting the garment's weight) was synthetic fibre? Answer as a percentage. | metric | 33.201 percent (± 1) | Synthetic share (worn), Fibre mass |
| What was the cost per wear of the "North Bend black vest" in 2024, in euros? | definition | 0.05 EUR (± 0.002) | Cost per wear (garment) |
| What was the cost per wear of the Shirts category in 2025, in euros? | definition | 1.296 EUR (± 0.01) | Cost per wear (category) |
| What was the cost per wear of the Socks category in 2025, in euros? | definition | 0.139 EUR (± 0.01) | Cost per wear (category) |
| What was the cost per wear of the Belts category in 2025, in euros? | definition | 0.263 EUR (± 0.01) | Cost per wear (category) |
| What was the cost per wear of the Blazers and vests category in 2025, in euros? | definition | 13.178 EUR (± 0.01) | Cost per wear (category) |
| What was the yearly cost of the Pants category in 2024, in euros? | definition | 170.959 EUR (± 1) | Yearly cost, Cost per wear (category), Days used |
| What was the monthly cost of the Shoes category in 2025, in euros? | definition | 22.73 EUR (± 0.1) | Monthly cost, Yearly cost |
| What is the inventory turnover of the Underwear boxers category over the whole period from 1 January 2018 to 30 September 2026? Answer as a percentage. | definition | 49.584 percent (± 0.5) | Inventory turnover |
| Which category had the highest yearly cost over the whole period from 1 January 2018 to 30 September 2026? Give the category name. | definition | Shirts | Yearly cost |
| Over the whole period from 1 January 2018 to 30 September 2026, on what share of days were socks worn? Answer as a percentage. | definition | 80.72 percent (± 0.5) | Days used |
| What was the average daily cost (the cost per wear of everything worn that day, Sportswear excluded) over the days with wears in September 2025, in euros? | definition | 5.27 EUR (± 0.05) | Daily cost |
| How many times per month, on average, has the "North Bend black vest" been worn over its whole ownership up to 30 September 2026? | definition | 8.891 wears per month (± 0.05) | Wears per month |
| In September 2025, what share of the worn fibre mass was synthetic? Answer as a percentage. | definition | 35.755 percent (± 1) | Synthetic share (worn) |
Code: mcp/evals (questions, runner, grading). Run it yourself with an Anthropic API key: cd mcp && npm run evals -- run.