3,066 green tests hid a line that breaks opening the database: checking tests by deleting the guard

A row-returning PRAGMA sent through execute

The blocker was a single line, PRAGMA secure_delete = ON. It was added that round so deleted messages get overwritten with zeros, and the code sent it through db.execute(), which expects no result rows. This pragma returns one. We reran all five pragmas from that function on October 20, 2026 with Python's sqlite3 module (SQLite 3.45.1, in-memory database):

In the same function, the two other pragmas that return rows already used rawQuery; only the new line did not. That setup code runs on every database open, before the version check. The review traced the iOS path through the plugin source: the execute path treats SQLITE_ROW as an error, so openDatabase itself fails and the whole storage layer goes with it. The Android side rests on the API contract of SQLiteDatabase.execSQL, which rejects statements that return rows. Neither platform has actually thrown the exception on a device yet.

The fix was one word:

- await db.execute('PRAGMA secure_delete = ON');
+ await db.rawQuery('PRAGMA secure_delete = ON');

The test SQLite could not see the failure

Unit tests run on sqflite_common_ffi, a desktop backend, not on the phone's SQLite. Its execute runs a statement without arguments and silently drops whatever rows come back. No exception, so the 3,066 tests could never have said anything about this bug. The commit message records that with the fix reverted, only the new guard fails.

Because no behavioural test can catch it, the guard reads source instead. pragma_execute_guard_test.dart pulls every string literal passed to .execute( under lib/, and runs each one that starts with PRAGMA through rawQuery on a real SQLite to see whether rows come back. The ffi backend returns rows normally from rawQuery, so the check tells the truth even inside the test environment. At the time lib/ had 14 such calls: 3 pragmas and 11 table or index statements.

The scanner also checks itself. If it finds fewer than 5 execute literals, it assumes its regex broke and fails, so an empty scan cannot pass for free. (The pragma list is not hard-coded either; the guard asks SQLite.)

Two fixes reported as verified had guards that guarded nothing

The same review ran revert checks: undo each fix and see whether the test said to protect it turns red. Two fixes that the AI coding agent had reported as verified in the previous round failed this check.

• Removing the localisation wiring from the app (the part that makes screen readers speak the framework's labels in Korean) left its test green. The test launched the test harness's own MaterialApp, not the real app. It now drives the actual app widget and checks the localisations resolved beneath it.
• Every message-eviction guard sat behind an environment variable that CI never sets. With eviction disabled, all 659 tests that run without that variable stayed green. They now run by default, and every revert in that area goes red.

The review logged 26 findings, and 24 of them had been introduced by that round's own fixes. Its notes say the pattern of the previous fix creating new defects had now repeated four times. What the agent got wrong was the report more than the code. It really had added tests, and they really passed; nobody had checked what they measured.

The reviewer was an agent too. It changed no code, and it restored the tree after every revert experiment. Its brief carried this rule: if a test still passes after you revert the fix, the test is wrong. After the fixes the suite stood at 3,099 passed, 13 skipped, 0 failed.

The rule had been written down two days earlier

When an agent says it added a test, delete the code that test is meant to protect and run it again. If it stays green, it does not count. The chat app's instruction file ties "done" to three conditions: zero analyzer errors, code reachable from main.dart, and code protected by a test. The release-readiness notes from September 9 already said every fix in that round was pinned by a test that fails when its guard is removed; two days later a second agent actually reverted the fixes and caught two that did not.

The order of checks we run before a release is in What to check before shipping an AI-built Android app.

The guard-removal harness: plain Python, one guard per row

SugarDoc has shipped a guard-removal script beside each feature since its first commit on October 8, 2026. As of October 18 there are 12 scripts, 970 mutation rows and 1,919 lines. Each row is a promise: change this string in this file, and this test class must fail. Below is a real row from the 1.2.0 PDF export script, followed by the core of the harness, trimmed.

# (id, file, [(original, replacement)], module, test class that must fail)
("ctl.logOnlySaved", P + "ReaderController.kt",
 [("if (result == PdfWriteResult.SAVED) log(...)", "log(...)")],
 "sugardoc", "ReaderControllerTest"),

for mid, rel, edits, kind, target in MUT:
    path, orig = mutate(rel, edits)          # only ever in the copy
    try:
        ok, fails, out = run_row(kind, target)  # gradlew --no-daemon --tests *.target
        if "e: file://" in out:
            verdict = "COMPILE-ERROR"
        else:
            verdict = "FAILS-AS-EXPECTED" if ok else "STILL-PASSES"
    finally:
        open(path, "w", encoding="utf-8").write(orig)

(The log(...) is shortened here; the real row spells out the analytics event.) Five decisions carry the design:

• The working tree is never touched. The repository is copied to a separate folder without .git, build outputs or signing keys, and every mutation happens there. Another agent session may be working in the same tree.
• Each original string must match exactly once. Anything other than one match marks the row BAD-PATTERN, meaning the source moved and the row needs updating.
• A catch needs two signals: Gradle exits with failure and the JUnit XML contains a failed test case. Compile errors are reported separately as COMPILE-ERROR. Going by exit code alone would count a typo in the mutation as a catch.
• One Gradle run at a time, no daemon, only the named test class, with a lock file holding the process ID so two runs never share a copy.
• The mutated file is always restored in finally.

Two rows that stayed green in 1.3.0

The SugarDoc 1.3.0 release record from October 18 holds real catches. After confirming that all 147 rows of the editing script still matched once, the 37 rows added or changed that day were run. Two came back STILL-PASSES on the first run.

The rerun after the fixes came back 4 of 4 failing, as it should. The PDF export script went 32 of 32 FAILS-AS-EXPECTED the same day. STILL-PASSES does not always mean a weak test. If the code defends the same thing twice, deleting one copy changes nothing, and the fix belongs in the code. (An agent wrote the tests for both features, too.)

What is still open

The Android failure in the chat app has not been measured. The verification notes list it as open: build an APK from the commit before the fix, install it on an emulator and read logcat at the first database open. iOS needs a macOS machine we do not have. On the SugarDoc side, 1.3.0 reran only the 37 changed rows of the editing script; the 110 untouched rows were not rerun. Whether those older rows still go red will show at the next full run.

FAQ

More AX notes

• A shared LLM cache could have frozen a prompt injection: the blind A/B that let us stop sending context
• AI coding agents and secrets: one rule in chat, logs and token permissions, and the two exceptions a copied masking regex missed
• What "100% E2E pass" really meant: deleting seven runners with zero expect() calls
• We built an edit feature and shipped it switched off: the HWP save gate that caught a form-field bug
• Pre-gzipping Android assets, measured: zero APK savings, and .pack.gz became .pack with a FileNotFoundException
• A market:// link opens a store chooser on SKT-sold Samsung phones: set the Play package first

All notes on Testing and release · All AX notes