What "100% E2E pass" really meant: deleting seven runners with zero expect() calls

All seven runners counted their own successes; five only timed delays

Commit 4d8fc3b on August 27, 2025 announced "100% success rate over 370 test runs". The report it added listed five categories at 100/100, 150/150 and so on. We opened the files that produced those numbers, exactly as they were at that commit.

No file had an expect(). No file called test() either. Each one ran a class from void main() async and printed its results (355 print() calls across the seven at deletion time). Five of the seven imported nothing but dart:async and dart:math, so they never touched app code at all.

The definition of success is still in the code:

// optimized_comprehensive_test.dart (2025-08-27, trimmed)
static Future<void> _stableUserProfileTest() async {
  await Future.delayed(Duration(microseconds: 500 + Random().nextInt(500)));
}
...
// lenient success criterion (within 2 s)   [comment translated]
if (totalTime < 2000) {
  successCount++;
}

Chain a few milliseconds of delay, and count a success if the chain finishes within two seconds. The report gives a mean of 46.48 ms for that category. The other thresholds were 3 s, 5 s and 2.5 s, and the comments call them "lenient" or "generous". The other two (chat_e2e_test and integration_test_runner) did call real app classes, but they too printed results without a single expect().

All three commits (952e9ef, "full web P2P chat app implemented", plus 4d8fc3b and c1388d8) end with a coding agent's "generated with" signature line. What settings or instructions produced them is not recorded anywhere in the repository, so this note calls these numbers "auto-generated reports of the time" and nothing more.

The 97% came from unit tests; the empty number was the "E2E 100%" in the same commit

Our first reading was that the "97%" in the title was inflated as a whole. That was wrong. Commit c1388d8 (August 31, 2025) puts two numbers side by side: "unit test pass rate 84% to 97% (104/107)" and "E2E tests 100% pass, 5-user scenario fully supported". At that commit, the 12 unit test files held 280 expect( calls, and 104/107 most likely came from them. We did not rerun year-old code to confirm it.

The cleanup's own explanation was wrong too. On September 6, 2026, while moving the old reports into an archive folder, a README committed with an agent co-author line (891db57) said the "97% / 95%+ pass rate" came from "counting skips as passes". The August 2025 files rule that out: at c1388d8 and 4d8fc3b, the word skip does not appear anywhere in test/e2e. The test('…', () {}, skip: 'Standalone runner…') wrappers were added eight months later, in April or May 2026, and only committed on September 6 when b8fb368 recovered that work from the working tree. The 2025 "100%" was each script printing its own counter.

The second mistake came from reading the past off the present state. When you write down a cause, open the files at that commit. git show <hash>:<path> is one line.

Nothing flagged it for eight months because the runners sat outside every count

The cause had three layers.

1. A test framework never did the counting. flutter test counts only what is registered with test(). The runners registered nothing, and two of them did not even end in _test.dart. Their numbers could only come from counters the scripts printed themselves.
2. The 370-run output could not fail. A two-second limit on delays that never call app code reports the same number whether the app works or not.
3. Once the files were wrapped in skip in spring 2026, "not run" became their official state, and in the meantime integration_test_runner stopped compiling.

There was a human cause as well. Numbers from an auto-generated report went straight into commit titles. Nobody asked which command produced them.

The first signal came when a person asked for a real run. In May 2026, ahead of a closed beta, the operator asked for the E2E tests to actually be run again. The run on May 6 showed that flutter test skipped all seven runners and that dart could not compile them. The decision to delete them came four months later.

We deleted instead of converting, and that night a real E2E logged seven defects

Backlog item 5-5-2 listed two options: (a) convert the runners into framework tests (minimum scenarios: 50 repeated handshakes, five users at once, message ordering), or (b) delete them and keep only the git history. At 20:26 on September 6, commit e03671a took (b): "zero expect() calls, never executed by flutter test, and integration_test_runner no longer compiled."

Five files imported no app code, so there was nothing to convert; it would have been a rewrite. The two that did call app code were deleted too, and the record gives only one line for each. integration_test_runner no longer compiled, and chat_e2e_test was listed in backlog item 5-5-3 as a risk because startDiscovery() needed a live HTTP/WebSocket server. The deletion commit says instead that real coverage lives in test/core, test/features, the handshake and p2p_chat E2E tests, and test/integration. There is no record of anyone matching the two files' scenarios against those tests one by one.

Leaving them skipped was possible, but 3,010 lines would keep suggesting "there is an E2E suite". The documented way to run them (dart test/e2e/…) no longer compiled because of the flutter_test import. Proving things with coverage instead was never considered in the record, and in our judgement it would not fit: coverage counts executed lines, not assertions, and five runners that never call app code add nothing to it. No mutation tool was used either. In the release check the next day (2026-09-07), the repository relied on a narrower method: delete the guard behind each fix and confirm a test fails.

That night a new harness started one flutter test process per peer, each with its own key pair and SQLite file, connected over real TCP or relay sockets, and ran it against commit c0bb66e (22:55). Its report listed seven defects (E1 to E7), and at 00:07 on September 7 commit 17aa057 fixed the six with confirmed causes (E1 to E6). E7, a periodic stall, was left without a known cause. E1 was the worst. A real room id is chat_<64hex>_<64hex>, 134 characters, but the sync service accepted at most 128, so delta sync was rejected on every connection between two real identities. The commit's explanation: "tests used short synthetic ids".

On September 13 the backlog's status table read flutter test 3,160 passed / 0 failed / 13 skipped. The 13 skips carried reasons (signalling server, soak gate), and the same table said "real devices: 0 runs, emulators only". Even so, "passed" in that table only says the tests passed. We cover that limit in 3,066 green tests hid a line that breaks opening the database: checking tests by deleting the guard.

Ten days later, 15 of 19 tests in another app compared values to themselves

The same shape turned up in an Android app. On September 10, 2025, commit d605c37 in the SugarSpeller repository (a Korean spelling checker) was titled "unit tests 100% pass and code coverage complete", with "17/17 new feature tests pass" in the body and the same signature line.

The commit's NewFeaturesTest.kt has 19 @Test methods, not 17. Four of them call app code (UIMode). The other 15 either re-check a value declared inside the test (val EXPECTED_TEXT_SIZE = 14f, then assertEquals(14f, EXPECTED_TEXT_SIZE)) or confirm Android constants such as Gravity.CENTER being 17. They assert, but no change to the app can make them fail. The fact that no coverage tool was ever configured in that repository is in A year of AI coding in 701 commits.

There was neglect in the other direction too. On September 29, 2026, commit a964c50 repaired four instrumented test suites that had been left alone since August 2025. In three of them (42 tests), the assertions were 106 Espresso check(matches(…)) calls and one Kotlin assert; 68 of those checks only asked whether a view was displayed (isDisplayed()). The commit calls these "empty assertions that only saw the screen was up" and replaced them with checks on corrected text, list counts and the clipboard. The fourth waited on a latch nobody ever counted down, so it always failed. After the repair, all 59 instrumented tests passed on a Galaxy S21.

Tests that grep flags wrongly or misses

Grepping for files without expect( gives false positives. On October 24, 2026, that search over the repository's 175 test files returns two. One is a property test that asserts with fail(); adding fail( to the pattern leaves one file.

That one is a measurement-only test that deliberately has no assertions. It measures tap target sizes for a report, and its header comment says it "makes no assertions (expect), so it always passes". But dart_test.yaml does not exclude it, so its tests land in the passed count.

Grep cannot judge what an assertion checks. Tests like SugarSpeller's 15 self-comparisons contain asserts and pass every filter. You have to look at imports and call targets, and what survives that is caught by deleting a guard or by review.

A checklist for the next pass rate you are handed

1. Write down numerator and denominator, the command, and where the log is. If any is missing, keep the number out of commit titles.
2. Find test files with no test functions, and runnable files that do not end in _test.
3. Find files without assertions (include fail( and any custom assert helpers in the pattern). On Android, search for JUnit assert and Espresso check(matches.
4. Add test files that never import the app package.
5. List every skip with its reason, and report the skip count next to passed. In flutter test output, ~N on the last line is the skip count (from this repository's records: +1125 ~4 means 1,125 passed, 4 skipped).
6. For each file found, decide: convert, delete, or tag as measurement. Put the reason in the commit body.
7. Delete a few key guards and confirm the tests go red.
8. Run once with real processes and realistically sized data. E1 hid behind short synthetic ids.

# Flutter, steps 2 to 5
grep -rL --include='*_test.dart' -e 'test(' -e 'testWidgets(' test/
find test -name '*.dart' ! -name '*_test.dart'
grep -rL --include='*_test.dart' -e 'expect(' -e 'fail(' test/
grep -rL --include='*_test.dart' 'package:<your_app>' test/
grep -rn 'skip:' test/

# Android (Kotlin), step 3
grep -rL --include='*Test.kt' -e 'assert' -e 'check(matches' app/src/

Run the first or third line on the files as of August 31, 2025 and five runners come out immediately; the find line catches the other two. On today's repository the third line leaves only the measurement test (and there are 17 skip: lines).

What we check next

The measurement-only tests still need a tag that keeps them out of the passed count. Before that, we need to count how many of the 3,160 were measurement output. "Real devices: 0 runs" is also still open. The September 13 table named that gap itself, so the next status line has to name the device.

FAQ

More AX notes

• A shared LLM cache could have frozen a prompt injection: the blind A/B that let us stop sending context
• AI coding agents and secrets: one rule in chat, logs and token permissions, and the two exceptions a copied masking regex missed
• We built an edit feature and shipped it switched off: the HWP save gate that caught a form-field bug
• Our contrast checker passed while skipping 40% of the text: make it print how much it measured
• Pre-gzipping Android assets, measured: zero APK savings, and .pack.gz became .pack with a FileNotFoundException
• A market:// link opens a store chooser on SKT-sold Samsung phones: set the Play package first

All notes on Testing and release · All AX notes