The lab

Experiments.

Tests to see whether AI can improve performance and capability in different situations, including the ones where it didn't.

Hybrid wins · 12Model wins · 18Rule wins · 17Mixed · 3
01 Rule wins

Pathfinding

Decision
Which gap in the wall to route through
Raced against
Dijkstra, A* (Manhattan), weighted A*, a no-model gap rule
Setup
Every searcher expands nodes at the same rate, in its real order. The model only picks the gaps; A* does the rest.
Result
About half the node expansions of A*, but paths 13.8% longer and 15.5 ms of model time against 0.16 ms for the whole A* search. A no-model gap rule does better.
02 Rule wins

Sudoku

Decision
Which cell to branch on
Raced against
MRV, fewest options first
Setup
The model sees the same four-cell shortlist as the no-model rule, written as words.
Result
MRV: 12/12 solved, 1.06 s total. Model: 4/12. The 1970s heuristic keeps its belt.
03 Hybrid wins

Snake

Decision
Next move
Raced against
Planner + cycle-safety shield
Setup
The planner labels every move with a verdict. The model picks one, and a cycle-safety shield can overrule it.
Result
Planner labels the moves, model picks, shield vetoes. The pattern that works.
04 Hybrid wins

Minesweeper

Decision
Which cell to click
Raced against
Solver verdicts + mine shield
Setup
Each option carries the solver's verdict. A shield blocks any cell that is a proven mine.
Result
16/20 boards with the shield, 13/20 without. The shield stepped in 8 times in 1,286 clicks.
05 Hybrid wins

Agent command gate

Decision
Allow, confirm or block
Raced against
Regex rules
Setup
Classification only, nothing is executed. Commands are a small hand-labeled set, some written after the regex was frozen.
Result
Regex alone waved 5/10 destructive commands through. Regex + model: zero.
06 Model wins

Tool router

Decision
Which tool gets the request
Raced against
Keyword rules
Setup
Hand-written requests routed to eight tools. The model generates no text, it only scores the options.
Result
83.7% vs 78.8%. On tricky phrasing, 72.5% vs 47.5%. 3.8 ms, zero tokens.
07 Rule wins

Cache eviction

Decision
Which key gets evicted
Raced against
LRU, LFU, Belady
Setup
The model picks a victim from the four least-recently-used keys, with no hint about which one LRU would evict.
Result
Matches LRU within 0.2 points. LFU still wins two traces at ~5,000× less cost per eviction.
08 Rule wins

Tetris

Decision
Which of four placements
Raced against
Placement heuristic
Setup
Both boards get the same pieces. The options are the planner's top four placements, with no 'best' label.
Result
Heuristic: 278 lines, never topped out. Model: 51 lines, topped out 5/5. D'oh.
09 Rule wins

Traffic signal

Decision
Keep green or switch
Raced against
Fixed-time, actuated, longest-queue
Setup
Every policy sees the same seeded arrivals. The model only gets the queues described in words.
Result
Longest-queue rule: 6.34 s mean wait. Model alone: 41.5 s. The model is still learning to read queues.
10 Mixed

Algorithm arena

Decision
Zero-shot vs trained classifiers
Raced against
SVM, LR, kNN, naive Bayes, keywords
Setup
Small hand-written data sets. Trained models are scored on out-of-fold predictions.
Result
Six data sets, significance tests on screen. 4 wins, 2 losses against the best trained model.
11 Model wins

Multilingual router

Decision
Which tool a request in any of 11 languages needs
Raced against
English keyword rules
Setup
One English tool schema, 16 hand-translated requests per language. The model reads the request as written, no translation step.
Result
86.4% vs 30.7%. Keywords are perfect in English and lost everywhere else.
12 Model wins

Car radio

Decision
Which of six stations to tune
Raced against
Exact English genre words
Setup
Requests describe a mood or a genre in eight languages, never an artist. The dial needle swings to whatever each side picks.
Result
86% vs 13.3%. The genre rule found no genre word in 130 of 150 requests.
13 Model wins

Dragon rider

Decision
Which of six moves the dragon flies
Raced against
Six exact English commands
Setup
Shouts in nine languages and plenty of slang. A ghost dragon flies the exact-command parser next to the model's.
Result
77.7% vs 6.4%. The ghost dragon mostly hovered.
14 Model wins

Fireworks

Decision
Which of six fireworks to launch
Raced against
English colour, shape and sound words
Setup
Crowd shouts in nine languages over a pixel skyline. First keyword match wins for the baseline.
Result
80.6% vs 25.7%. Keywords matched nothing in 107 of 144 shouts.
15 Model wins

Neon signs

Decision
Which of six signs a shop gets
Raced against
English business words
Setup
Owners describe their business in nine languages. Keywords first, then the model, scores higher than either.
Result
87.3% vs 40.7%; keywords then model, 92%.
16 Model wins

Pirate ship

Decision
Sail, fire, anchor, left, right or flag
Raced against
Exact English nautical words
Setup
The captain bellows in nine languages on a rolling sea. Same orders for both ships.
Result
75.7% vs 24.3%. Keywords matched nothing in 109 of 144 orders.
17 Model wins

Starship bridge

Decision
Shields, warp, fire, hail, stop or scan
Raced against
Exact English command words
Setup
Orders in nine languages, played out on the viewscreen. A fairer keyword-stem rule is reported too.
Result
78.5% vs 9%. Even English keyword stems only reach 38.2%.
18 Model wins

Model railway

Decision
Which station to set the points for
Raced against
English keyword dispatcher
Setup
Passengers say where they're going in any language. Options are station names plus two plain words each.
Result
87.3% vs 30.7%. On non-English requests: 72 of 78 vs 1.
19 Model wins

Paint studio

Decision
Which landscape element to paint next
Raced against
English keyword lists
Setup
Viewers suggest trees, a mountain, a lake, the sun, a cabin or clouds in any language. Both canvases paint whatever they read.
Result
93.1% vs 36.8%. Keywords got 0 of 72 non-English suggestions.
20 Model wins

Departures board

Decision
Which service-desk queue a stranded passenger needs
Raced against
English keyword rules
Setup
Messages in eight languages for a fictional airline, shown on split-flap boards.
Result
86% vs 36.7% at 3.9 ms a message.
21 Model wins

Flood SMS

Decision
Rescue, medical, water, food, shelter or info
Raced against
English keyword rules
Setup
Hand-translated flood texts in eight languages on a city board. Rescue and medical routes still go to a person.
Result
79.2% vs 29.2%.
22 Model wins

Tavern

Decision
Which item an adventurer wants, and are they haggling
Raced against
Keyword, random and all-knowing shopkeepers
Setup
40 customers a day in seven languages, 40 seeded days. The model sees only the request text.
Result
Beat the keyword shopkeeper on 40 of 40 days.
23 Model wins

Dock control

Decision
Which dock a ship gets
Raced against
Keyword, random and oracle controllers
Setup
Pilots radio in seven languages. A mayday rule always runs after the model, whatever it says.
Result
Beat the keyword controller on 38 of 40 watches.
24 Model wins

Tycoon

Decision
Bug, refund, restock, hiring, press or spam
Raced against
Keyword, inbox-blind and oracle managers
Setup
30 runs of 80 messages into a fictional company inbox, each run starting with $10,000.
Result
Beat the keyword manager in 24 of 30 runs.
25 Model wins

Text adventure

Decision
Which verb and object a command means
Raced against
Classic two-word parser
Setup
A six-room adventure you can play in any language. About two typed questions per command.
Result
Verb and object both right: 79.6% vs 14.8% for the two-word parser.
26 Model wins

Robot arm

Decision
Which action, on which colour block
Raced against
Keyword lists, plus a stop rule
Setup
Typed commands in eight languages. A stop rule and move checks sit in front of the arm either way.
Result
Action and target both right: 76% vs 31.3%.
27 Model wins

Chat plays

Decision
Which of six buttons a chat message votes for
Raced against
Exact commands only
Setup
200 messages: votes, emotes, slang, typos and plain junk. A wrong vote counts against you.
Result
125 real votes captured vs 14, at the price of 16 false votes (exact matching made none).
28 Hybrid wins

Data quality alerts

Decision
Which team owns an alert
Raced against
Ownership map of table names
Setup
Alerts in ten languages with fictional table names. Hybrid: the map when it matches, the model otherwise.
Result
Model 88.7%, map 36.3%, together 91.9%.
29 Hybrid wins

Tab filer

Decision
Which group a browser tab belongs in
Raced against
Domain rules
Setup
200 synthetic tabs, 62 of them on mixed-use domains where the URL says little.
Result
75% each on their own; 88% together. On tricky tabs, 67.7% vs 21%.
30 Hybrid wins

Recipe tags

Decision
Which canonical tag a messy user tag means
Raced against
Alias list (English and loanwords)
Setup
Hand-written messy tags, many tricky or non-English. The alias list runs first; the model reads what it misses.
Result
Aliases 61.4%, model 77.9%, aliases then model 93.1%.
31 Hybrid wins

Jet cabin

Decision
Which cabin system a spoken request is for
Raced against
English keyword rules
Setup
Classify only, nothing is switched. Rules first; the model handles what they can't read.
Result
Rules 70%, model 79.2%, rules then model 94.2%.
32 Hybrid wins

Complaint desk

Decision
Which review panel a complaint goes to
Raced against
Keyword clerk
Setup
Non-identifying complaints for a fictional city, many deliberately tricky.
Result
The keyword clerk beat the model, 76.5% to 65.7%. Together: 92.2%.
33 Hybrid wins

Forum sorter

Decision
Which of six subforums a post belongs in
Raced against
Moderator keyword rules
Setup
Baking-forum titles. The rules couldn't read 32 of them; those go to the model.
Result
Rules 72.3%, model 67.3%, rules then model 89.1%.
34 Hybrid wins

SafeChat

Decision
Which approved preset a kid's message becomes
Raced against
Keyword presets
Setup
Kids type anything; only presets are ever sent. Block list, then keywords, then the model.
Result
Model 66.7%, keywords 70.6%, the pipeline 83%. Unsafe messages: 15 of 15 caught.
35 Hybrid wins

Terminal errors

Decision
Which fix family an error message needs
Raced against
English regex table
Setup
Errors in ten languages. Classified, never executed.
Result
Regex 36.2%, model 78.7%, regex then model 83.5%.
36 Hybrid wins

Bank feed

Decision
Which budget category a card descriptor is
Raced against
Merchant keyword rules
Setup
Messy card descriptors, cleaned up before the model reads them. On its own it misfiled $3,151 of a $10,305 month.
Result
Rules 80%, model 54.5%, rules then model 82.8%. The rules did the heavy lifting.
37 Rule wins

Match ticker

Decision
Goal, shot, foul, sub, set piece or play
Raced against
Keyword rules
Setup
152 lines of football commentary. Team and minute come from the feed, not the model.
Result
Keywords 73% vs 57.2%. On the 20 tricky lines, 40% vs 20%.
38 Rule wins

Snippet filer

Decision
Shell, SQL, regex, config or git
Raced against
Syntax regex
Setup
Dev notes in ten languages, each with the first 48 characters of code.
Result
Regex 97.5% vs 70%. Code has syntax; syntax has regex.
39 Rule wins

Bilingual log

Decision
Home language, English or mixed
Raced against
Common-English word list
Setup
A tide-pool game tagging parents' log lines by language.
Result
Word list 91.5% vs 31.4%, worse than guessing blind (41.5%). The model reads meaning across languages; it doesn't name them.
40 Rule wins

Spelling patterns

Decision
Which error pattern an early reader's misspelling shows
Raced against
Rule-diff planner
Setup
The planner diffs the word against the target. The hinted model sees the planner's verdicts.
Result
Planner 99%. Model hinted 77.1%, plain 33.3%.
41 Rule wins

Lab notebook

Decision
Which experiment type an entry records
Raced against
Cue-word planner
Setup
Notebook entries with invented compound codes, plain and tricky.
Result
Planner 94.1% vs 67.8%. Hints only lifted the model to 69.5%.
42 Rule wins

Clickbait meter

Decision
Which of four styles an episode title is
Raced against
Style rules: caps, '!!', hype words, length
Setup
Titles in nine languages, one editor's subjective labels.
Result
Rules 73.3% vs 50.8%. Clickbait shouts in a way regex can hear.
43 Rule wins

Star trader

Decision
Will a headline push a price up or down
Raced against
Keyword, news-blind and oracle traders
Setup
30 games of 60 turns from fictional galactic headlines. The model holds when it's unsure.
Result
Beat the keyword trader in only 8 of 30 games.
44 Rule wins

Commodity pit

Decision
Buy or sell on a crop, weather or shipping report
Raced against
Keyword, news-blind and oracle traders
Setup
30 sessions of fictional wire reports, 21 of them with no real effect.
Result
Beat the keyword trader in 8 of 30 sessions. Keep your day job.
45 Rule wins

Frontier explorer

Decision
Which unexplored frontier to drive to
Raced against
Nearest frontier, information gain
Setup
Same hidden map for every rover. The model sees up to four frontier clusters described in words.
Result
1,955 steps to 95% coverage vs 1,397 for info-gain and 1,490 for nearest.
46 Rule wins

Elevator dispatch

Decision
Which car takes each passenger
Raced against
Nearest car, ETA, random car
Setup
243 seeded passengers, 12 floors, 3 cars. The model gets worded car state, no hints.
Result
37.6 s mean wait vs 15.9 s for ETA. A random car did 21.8 s.
47 Rule wins

Pac-Man

Decision
Which direction to go
Raced against
Flee/forage rule
Setup
Directions described as ghost-danger and pellet words, with a shield against walking into ghosts.
Result
333 pellets vs 928, and the shield stepped in 259 times.
48 Rule wins

Lane change

Decision
Keep lane, go left or go right
Raced against
MOBIL-lite, keep-lane
Setup
Gaps described verdict-first in words. A shield rejects unsafe changes.
Result
75.7 km/h vs 87.8 for MOBIL-lite, with 350 shield overrides.
49 Mixed

Live typing

Decision
Which tool, re-asked on every keystroke
Raced against
Its own answer on the finished sentence
Setup
104 requests typed out character by character, 3,780 routing calls in all.
Result
3.8 ms a keystroke and right on 87 of 104, but it needs about 55% of the sentence before it settles.
50 Mixed

Log triage

Decision
How bad a log line is, and which subsystem
Raced against
Regex rules
Setup
Synthetic log templates, some the regex was tuned on and some held out.
Result
Loses on the regex's home turf (53% vs 70%), wins on held-out lines (46% vs 25%).

The model in these clips is Laya, an open decision model that runs on the device (about 4 ms a call on the Apple Neural Engine). The benchmarks are small and built by hand, so read them as examples rather than general accuracy numbers.