Three agent designs, three judges: the market judge alone would have picked the one that crashed on every KakaoTalk file

59 minutes before any code: research, refutation, three designs, judging, critique

HWP and HWPX are the file formats of Hangul, the word processor Korean schools and government offices use, and many people receive these files through KakaoTalk, Korea's dominant messenger. On the evening of October 4 we had market research and an engine spike for a viewer (the engine numbers are in Building an Android HWP viewer: rhwp in a WebView, measured on 222 documents) and zero lines of code. At 21:34 KST the main session wrote one Claude Code Workflow script and started it.

The script has five stages. Twelve agents research twelve areas separately: competitors, the HWP engine, the PDF engine, intents and entry, the received-documents inbox, monetization, UX, the codebase, store search, quality, security and scope. Two reviewers per area then try to break each report. Any finding that at least one of the 24 verification reports marked refuted was deleted before design; there were 16. From what remained, three design agents wrote designs that differ only in priority, three judges scored them, and one critic checked the final plan made from the winner plus ideas grafted from the others. A last agent was assigned five documents: analysis, PRD, screen contract, architecture and implementation plan.

Research ran from minute 0 to 16 and verification, starting on each area as soon as its research finished, from 7 to 26. The three designs took 6 minutes, judging two and a half, and synthesis, critique and document writing 25.

The three design prompts differed by one sentence

Every agent got the same shared context: the repository layout (minSdk 29, targetSdk 36), where the earlier research lived, the goal, a rule to tag each claim as verified, inferred or unverified, and a ban on editing files, installing packages, committing or touching devices. Each stage returned JSON against a fixed schema.

The three priority sentences: speed, "reach the internal track in two to three weeks, inside the October demand window, without giving up never-fails-to-open and no ads on documents"; quality, "open everything correctly and fast on low-end phones, with the strongest regression and crash guardrails"; growth, "maximize installs, return visits and revenue within the trust rules". The judges were a Korean mobile market veteran who had shipped top-10 utility apps, a staff Android engineer responsible for crash-free rate and APK size on low-end Samsung phones, and a Google Play and AdMob policy and privacy reviewer. The script picked the winner by the sum of the three judges' totals. A majority vote would also have chosen quality, the pick of judges 2 and 3; with only the speed design on the table, the NoDisplay entry would have gone straight to implementation.

The speed design's entry point: async copy in a NoDisplay activity

The speed design planned nine milestones totalling 20.5 days. All document intents went to one EntryActivity with Theme.NoDisplay. On Dispatchers.IO it would take the URI, sniff the format from the first 64 KB, copy the file into app storage while hashing it with SHA-256, start the reader and finish, showing progress and a cancel button after one second.

A NoDisplay activity has to call finish() before onResume() completes. For apps targeting API 23 or higher on Android 6.0 or later, one that does not crashes with IllegalStateException and the message "did not call finish() prior to onResume() completing". A CommonsWare blog post flagged it in November 2015 and noted that activities doing their work in onCreate() and finishing right there seem to be fine. Our app targets API 36, and an async copy finishes after onResume(), so as designed, every "open with" from KakaoTalk would have crashed the app.

At design time there was no code to run. Written as designed, it would compile; it shows up only in a test that runs the activity lifecycle, such as Robolectric, or when a device opens a file through a VIEW intent. Judge 2 called it easy to fix but fatal if built as written. Judge 3 said to switch to Theme.Translucent.NoTitleBar.

The final plan has the reader activity take VIEW intents directly, and a Translucent SendEntryActivity hand shared files on and finish inside onCreate(). The architecture document lists the NoDisplay entry as a rejected option, and the PRD acceptance criteria include "zero Theme.NoDisplay activities".

Judge 1 picked the design with the crash

Judge 1 listed ten fatal flaws, none of them NoDisplay. The three judges gave speed 8, 7 and 8 for win potential. Quality got 9 for reliability from all three, but its milestones added up to 44 days, and all three said it would miss the October demand window. Growth (39 days) contradicted itself: one quality gate required switching to a preview within 6 seconds while its state machine showed a notice at 8 seconds and terminated at 30.

Judge 3 also caught that the app lock in the speed and growth designs used BIOMETRIC_STRONG|DEVICE_CREDENTIAL, which API 28 and 29 do not support. With minSdk 29, that paid feature would not have worked on Android 10 phones. The judges' 25 fatal flaws were merged into nine rows in the analysis document, each with its fix.

23 gaps in the merged plan: a schedule that did not add up and a gate that could not run

The synthesis agent started from the quality design, grafted in the speed design's scope cuts and parallel work, and fixed every fatal flaw. The critic's checklist found 2 blockers, 10 major and 11 minor gaps in the result.

Blocker 1 was arithmetic: the milestones on the critical path summed to 23 working days, so even with no weekends the internal-track date was already missed. Blocker 2 was a release gate that could not run. It required a shell script to pass on a Firebase Test Lab device, and Test Lab gives you no adb shell. The gate was split in two, and the first build milestone (M1) got a spike for instrumented tests on a release build.

One major gap concerned password-protected files. For a fast first screen, the plan showed the preview image stored inside the HWP file before rendering, and a research-stage experiment had found that image in encrypted documents too, so the first page would show before the password. It became one functional requirement and one architecture section. The analysis document has a table mapping each of the 23 gaps to where it was fixed.

The NoDisplay entry never reached git, and the ban test closed two bypasses

Early on October 5 the first implementation workflow (M1 and M2) added a NoDisplay ban to ViewerArchitectureTest. A review agent in the same workflow found two ways around it: the <application> theme, which activities inherit, and a style item android:windowNoDisplay=true. The test, which never launches an activity, now reads every themed element in the manifest plus style parents and that item, and each bypass was planted once to confirm the test fails. A guard-removal script also swaps the SendEntryActivity theme for NoDisplay and expects the test to fail.

git log -S NoDisplay returns one commit: ca1ade4 on October 8, internal build 1.0.0 (2), which brought in the ban test and the design documents. On October 9 the owner sent HWPX, HWP5, PDF and HWP3 files from another KakaoTalk account to a Galaxy S23 and all four opened (3a3dfc6).

The NoDisplay entry was first proposed by a refuting reviewer

The intents research report never mentioned NoDisplay. The entry came from the fact-checking reviewer of that report, as its suggested replacement: "VIEW and SEND are received by a very light trampoline (Theme.NoDisplay, excludeFromRecents, noHistory)", with the trampoline starting the 64 KB sniff and the copy. That text went into the dossier and the speed design followed it. The quality and growth designs did not use the NoDisplay theme.

Judge 3 tagged the NoDisplay flaw "[verified, Android 6.0 behavior change]", but that agent made one tool call, the one that submitted its result, so the tag was added without opening anything. The claim was right; we confirmed it against the CommonsWare post while writing this. The three design agents and judge 3 worked only from the dossier pasted into the prompt; judges 1 and 2 looked at the git history and spike files once or twice.

The schedule estimate was wrong in the other direction. The analysis said the original November 3 production date could not be met and set the internal track for November 2, production for November 9 and November 13 to 16 as the worst case. The internal track came on October 8 and the public release on October 10. Some milestone exit criteria, such as purchase restore, the app lock on API 29 and TalkBack checks on a device, were still open at release. We had told the critic to judge estimates for one developer plus Claude; the build then ran as separate workflows, one or two milestones each.

The document-writing agent was assigned five files and wrote four. The harness blocked it from writing the analysis report as a file; it did not work around the block and returned the text in its answer, which the main session saved; the workflow record lists all 45 agents as done.

How to run it: the script skeleton and the order

A trimmed skeleton of the real script: prompt text removed, and helper names such as dropRefuted, argmax and sumTotals are illustrative (the original does this inline). agent, parallel and pipeline come from Claude Code Workflow scripts.

const verified = await pipeline(AREAS,
  a => agent(CTX + a.task, { schema: REPORT }),
  (r, a) => parallel(['fact-check', a.lens].map(l => () =>
    agent(CTX + LENS[l] + JSON.stringify(r), { schema: VERDICT }))))
const dossier = dropRefuted(verified)          // 16 findings removed
const designs = await parallel(ANGLES.map(an => () =>
  agent(CTX + an.priority + dossier, { schema: DESIGN })))
const judged = await parallel(JUDGES.map(j => () =>
  agent(CTX + j + designs + dossier, { schema: SCORE })))
const winner = argmax(sumTotals(judged))       // sum, not one judge
const final = await agent(CTX + winner + designs + judged + dossier, { schema: DESIGN })
const gaps = await agent(CTX + CHECKLIST + final + dossier, { schema: CRIT })

1. Give every research area one fact-checking reviewer and one reviewer with an area lens, and drop refuted findings before design.
2. Have every design use the same dossier and the same schema, changing only the priority sentence. Put day estimates in the schema so schedules can be compared.
3. Use three judges from different angles, such as market, engineering and policy, and decide by the total. Ask for fatal flaws and graft ideas separately.
4. Run a checklist critique on the final plan, and keep a table of where each gap was fixed.
5. Pin each fatal flaw with a rule test, such as an architecture test that parses the manifest and styles, and remove the guard once to see the test fail.
6. Keep checks only a person can do, like opening files on a real phone, on a separate list and close them there.

Review after the code exists is a different process; we ran one where every finding faces three refuting agents, described in 84 agents reviewed a release: three refuters per finding, and only the 20 survivors got fixed.

When 8.8 million tokens and an hour are wasted

Verification used the most tokens, 2.95 million. The nine design, judge and synthesis agents made 21 tool calls between them and used 3.84 million tokens, because each prompt carried the whole dossier as JSON. We did not record a dollar cost.

If the design has nothing to argue about, say one more screen or a feature whose API is already chosen, this is too much. Here a new app, two rendering engines, the entry path, monetization and Play policy were all in play at once. Every agent was the same model reading the same dossier, so a flaw they all miss stays missed, and the scores are not calibrated: the same speed design got anywhere from 29 to 34.

What to check next

As of the October 9 record, opening files received through Naver Mail (a major Korean email service) and Gmail had not been tested on a device.

FAQ

More AX notes

• Several Claude Code sessions, one repo: the file owner table, checked against 13 merges
• Commits without a Co-Authored-By trailer: the agent made 26 of the 29 we could check
• Running an AI marketer on a schedule four times a day: the permission table that leaves money, accounts and policy to a person
• 84 agents reviewed a release: three refuters per finding, and only the 20 survivors got fixed
• Pre-gzipping Android assets, measured: zero APK savings, and .pack.gz became .pack with a FileNotFoundException
• A market:// link opens a store chooser on SKT-sold Samsung phones: set the Play package first

All notes on Working with agents · All AX notes