Testing and release
Unit tests, release gates and on-device checks that AI-written code goes through before each release, with the real defects those checks caught along the way.
Notes
A shared LLM cache could have frozen a prompt injection: the blind A/B that let us stop sending context
A shared LLM cache keyed without the context could serve one injected answer to every user. A blind A/B showed the context added nothing: 96.0% vs 96.6%.
AI coding agents and secrets: one rule in chat, logs and token permissions, and the two exceptions a copied masking regex missed
Three ways secrets leak around an AI coding agent: a key in a session log, two exceptions a copied masking regex missed, a deploy plan asking for a wider token.
What "100% E2E pass" really meant: deleting seven runners with zero expect() calls
Seven runners behind a "100% E2E pass" since Aug 2025 had zero expect() calls; five never called the app. The night we deleted them, a real E2E logged 7 bugs.
We built an edit feature and shipped it switched off: the HWP save gate that caught a form-field bug
A gate that diffs edited HWP files record by record caught a form-field bug. 1.1.0 shipped with edit behind a flag; 1.3.0 turned it on nine days later.
Our contrast checker passed while skipping 40% of the text: make it print how much it measured
A contrast check that passed in three themes had silently dropped 3,762 of 9,318 text fragments (40%). Now it prints what it measured and fails on a shortfall.
3,066 green tests hid a line that breaks opening the database: checking tests by deleting the guard
A P2P chat app passed 3,066 tests while one SQLite pragma, traced through plugin source, kept its database from opening. Now we judge tests by deleting guards.
The agent said it was tested. The game crashed on the first tutorial tap: a missing VIBRATE permission and the fresh-install monkey test
Our AI agent called a puzzle game crash-free, then it died on the first tutorial tap: a missing VIBRATE permission. adb monkey on clean data hit it at event 15.
What to check before shipping an AI-built Android app: the release gate and an R8 crash
Our Android app ran fine in debug but crashed at launch in release after R8 removed a constructor. Now every upload passes three pre-release checks.