A 20% staged rollout went out at 100%: a completed release in the Play Publisher API cannot go back to staged

The command got 0.2 and sent completed

The agent ran play3.py promote 97 "97 (1.6.029)" notes29.json production 0.2. The trailing 0.2 was the rollout fraction. This is how the script picked it up:

a=sys.argv; FR=None
if a[1] in ("upload","promote") and len(a)>(8 if a[1]=="upload" else 7): FR=a.pop()
...
**({"status":"inProgress","userFraction":float(FR)} if FR else {"status":"completed"})

That command produces seven entries in sys.argv, counting the script name. The check wanted more than seven before it would take the fraction. Off by one, FR stayed empty, and the default for "no fraction" was completed. The upload branch was off by one the same way.

The script was never in the repository. The agent had made it earlier in the same session by patching an older script with sed, and since the internal-track upload passed no fraction, nothing had exposed the bug. (vc97 was itself a same-day rebuild. A deep check on a Galaxy S21 that afternoon found an old bug where launching with the screen off left the app stuck on a black splash, so vc96 was dropped and the fix went into a new build.)

The PUT back to 20% got 403 PERMISSION_DENIED

The agent opened a new edit straight away and tried to set the production track to "vc97 inProgress 0.2, previous vc94 completed". Play answered:

before [('97 (1.6.029)', 'completed', ['97'], None)]
put [('97 (1.6.029)', 'inProgress', 0.2), ('94 (1.6.027)', 'completed', None)]
HTTP 403 {
  "error": {
    "code": 403,
    "message": "You cannot rollout this release because it does not allow any existing users to upgrade to the newly added APKs.",
    "status": "PERMISSION_DENIED"
  }
}

The edits.tracks reference defines completed as a release that "will have no further changes" and is served to all users. userFraction can only be set when the status is inProgress or halted, and must sit between 0 and 1. Nothing in the docs describes putting a fraction back on a release that has already gone to everyone, and we found no paragraph explaining this particular 403 message.

The agent printed completed and committed anyway

There were two mistakes, and the second was bigger than the off-by-one above. The script printed the track from the PUT response as track production [('97 (1.6.029)', 'completed')] and went on to commit in the same run without checking it.

Afterwards the agent tried to fix the check with sed, and Claude Code's automatic permission check blocked the edit. The agent told the operator it would stop using that script and wrote the incident into the release record, which is commit e64cd83 at 15:52.

That record says a hotfix was the only way to stop the rollout. The docs describe one more route. Per the "Halt a completed rollout" section of Google's APKs and Tracks guide, a completed release can be set to halted if the track already has an earlier published completed release; new users and existing users who update then get that earlier release. The completed release before 1.6.029 was vc94. We never tried this, so we do not know whether Play would have accepted it. What we did instead was add +1 day and +3 day stability rows to the monitoring doc. The late check on October 15 found no regression signal, though Play's crash numbers were out of reach because the reporting API lacked permissions.

Internal track first, then promote the same build

One rule was already two days old when this happened. On October 9, 1.6.026 went straight to 100% of production through the Publisher API, and the operator asked that next time the build go to internal testing first and be promoted from there. A sideloaded APK cannot show the Play path end to end: Play signing, billing, Play's automatic protection. The bundle is uploaded once, and production gets the same versionCode in a new edit.

1.6.029 did follow that order, but the fraction was lost in the second step, the promotion from internal to production. The two versions after it kept the fraction too.

The 20% day for 1.6.030 did not collect enough real users to compare anything. Most of the 1.6.030 users in GA4 were Play pre-launch report test devices and our own development phones. So the 100% call rested on zero crashes and a diff showing no ad code had changed.

The promotion script now never reads the fraction by position

From 1.6.031 on, each version folder has its own release_production.py, and promotion goes through it. Trimmed to the core:

commit = '--commit' in sys.argv          # dry run unless given
fraction = float(sys.argv[sys.argv.index('--fraction') + 1]) if '--fraction' in sys.argv else 0.2
edit = call('POST', f'{API}/edits', {})['id']
done = False
try:
    internal = call('GET', f'{API}/edits/{edit}/tracks/internal')['releases']
    new = [r for r in internal if r.get('versionCodes') == [NEW]]
    if len(new) != 1:
        sys.exit(f'vc{NEW} is not on internal')
    prod = call('GET', f'{API}/edits/{edit}/tracks/production')['releases']
    old = [r for r in prod if r.get('versionCodes') == [OLD] and r['status'] == 'completed']
    if len(old) != 1:
        sys.exit(f'no completed vc{OLD} on production')
    call('PUT', f'{API}/edits/{edit}/tracks/production', {'track': 'production', 'releases': [
        {'versionCodes': [NEW], 'status': 'inProgress', 'userFraction': fraction, ...},
        old[0]]})
    call('POST', f'{API}/edits/{edit}:validate')
    if commit:
        call('POST', f'{API}/edits/{edit}:commit'); done = True
finally:
    if not done:
        call('DELETE', f'{API}/edits/{edit}')   # dry run or failure: throw the edit away

(The real script's messages are in Korean; they are translated here.) Four things changed:

• status is hard-coded as inProgress. A missing fraction cannot fall through to completed, and the fraction only changes through the named --fraction flag.
• The internal testing track must hold exactly one release with that versionCode. A promotion that skipped internal stops here.
• Production must hold exactly one completed release of the previous version. That is what the other 80% keep getting while the new release sits at 20%.
• The default is a dry run: it validates and then deletes the edit. The only run that changes anything is the one with --commit, after the operator has checked. 1.6.031 shipped that way on October 18, after the operator confirmed.

The PUT body has the same shape as the request that got the 403 on October 11: new release inProgress, previous release completed. For 1.6.031 the new release had not been served to anyone yet, and this request passed validate and commit.

The rule now, and a check not yet in code

If you give an agent permission to ship to production, do not let it use a script that takes the rollout fraction as a positional argument. Code that defaults to completed when the fraction is missing turns one slip into a full release. The rule now: uploads go to the internal testing track only, the operator installs from Play and checks, and then the same build is promoted. Promotion runs as a dry run first, sits at 20% of production for a day, and only then goes to 100%.

One line is still missing from the code. The project notes say to confirm, before commit, that the PUT response shows status inProgress and the right userFraction. The current script builds the PUT body itself and never reads the response. The next version of the script gets this check:

tr = call('PUT', f'{API}/edits/{edit}/tracks/production', body)
mine = [r for r in tr['releases'] if r.get('versionCodes') == [NEW]]
if len(mine) != 1 or mine[0]['status'] != 'inProgress' or mine[0].get('userFraction') != fraction:
    sys.exit(f'PUT response does not match: {mine}')   # stop before commit

We will also write down the halt procedure from the docs at the next staged release, and check then whether a stray completed could send new installs back to the previous version without waiting for a hotfix build. Our release checks, in order, are in What to check before shipping an AI-built Android app, and the app itself is on the SugarSpeller page.

FAQ

More AX notes

• A month of 'temporary error': our subscription details screen never waited for products, and 1 of 16 users started checkout
• GA4 said 19 installs yesterday and 8 were real: filtering out test phones and Play pre-launch and review devices
• Pre-gzipping Android assets, measured: zero APK savings, and .pack.gz became .pack with a FileNotFoundException
• A market:// link opens a store chooser on SKT-sold Samsung phones: set the Play package first
• A splash that never moved on: addOnCompleteListener(this) and onStop
• A reverse-construction generator and the gate behind 1,100 arrow-puzzle levels

All notes on Play Store and monetization · All AX notes