A month of 'temporary error': our subscription details screen never waited for products, and 1 of 16 users started checkout

The 90-day funnel: 45 on the plans screen, 16 on details, 1 at checkout

On September 26 we pulled the subscription events for our Korean spell checker, SugarSpeller, counted by user, over 90 days. 45 users saw the plans screen, 16 went on to the details screen where you pick a billing period, and 1 started checkout. Fifteen of the sixteen (94%) left the details screen without starting a purchase. The details-screen events split 18 premium and 2 basic, spread across seven versions between 1.6.016 and 1.6.024.

The "temporary error" in the review is the app's generic retry toast, which in Korean reads roughly "A temporary error occurred. Please try again."

The function that requested products judged an empty list one line later

addSubsItems() on the details screen launched a coroutine in lifecycleScope that waited on the product flow with firstOrNull(), and then, further down the same function, checked the list size straight away. Trimmed, the old code looked like this:

getProductDetails(repo.premiumProductDetails)   // inside: launch { flow.firstOrNull() }
if (recyclerAdapterSubscription!!.mItemList!!.size <= 0) {
    Toast.makeText(..., R.string.string_common_error_retry, ...).show()
    startActivity(main screen)
}

If the billing manager's StateFlow already held the product, firstOrNull() returned on the spot and the list filled. If not, the coroutine suspended, the function moved on, and the size check saw an empty list. Every user whose products were not already cached got the toast and landed back on the main screen. The size check came in with a commit on January 4, 2024; the firstOrNull() call with one on October 3, 2024.

9290f91 turned the check into a wait. The screen now shows a progress dialog and waits inside withTimeoutOrNull(8_000); only if nothing arrives in 8 seconds does it show the toast and go back. The code comment gives the reasoning: connecting to billing and querying products usually takes 1 to 2 seconds, so wait several times that. We did not measure a latency distribution to pick 8 seconds.

The 2024 belief that Play 'sometimes splits the response' came from our own for loop

BillingClientManager stored the products Play returned only if(newMap.size >= 2), that is, only when basic and premium arrived in the same response. The commit that added the line, on October 4, 2024, says the product details response "intermittently comes in separately or together", so only store it when both arrive at once.

The split came from the app. Since December 30, 2023, queryProductDetails() had called queryProductDetailsAsync inside the loop that built the product list, once per iteration. With two products, that sent a one-product query and then a two-product query, and the responses came back as a one-item list and a two-item list. The 2024 fix hid the symptom by throwing away the one-item response. In exchange, whenever Play returned only one product (for example when one product's base plan is turned off), the map stayed empty for good.

9290f91 queries once and merges whatever arrives into the existing map. In Play Billing Library 9.1.0, QueryProductDetailsResult reports products Play could not return in unfetchedProductList; we now log it, which separates Play Console setup problems from app problems.

Three of the agent's first five tests stopped at Mockito

The fix was made with an AI coding agent (Claude Code), tests first: three Robolectric screen tests (waits while products are missing, lists plans when they arrive late, errors only when they never arrive) and two billing manager tests (single-product response, split responses merged). All five failed on the first run, but three of them were not reproducing the bug:

java.lang.NullPointerException: Cannot invoke "org.mockito.plugins.MockMaker$TypeMockability.mockable()" because "typeMockability" is null

The agent had mocked ProductDetails with Mockito. The class is final, and this project's test classpath mixes in mockito-android, so final mocks could not be created, and we dropped the mock. ProductDetails has a package-private constructor that takes a JSON string, so a test helper, TestProductDetails, lives under the same package name, com.android.billingclient.api, and builds real objects from the JSON shape Play sends (productId, type, subscriptionOfferDetails, pricingPhases). On the next run all five failed for the intended reasons: the screen test with "sent the user to main even though products simply had not arrived yet", the merge test expecting basic and premium and getting an empty set. After the fix they passed. A trimmed version of the helper, with product IDs and price replaced by placeholders:

package com.android.billingclient.api   // test sources; the constructor is package-private

fun subscriptionMonthly(productId: String, basePlanId: String): ProductDetails {
    val phase = JSONObject().put("priceAmountMicros", 1_000_000_000).put("priceCurrencyCode", "KRW")
        .put("formattedPrice", "KRW 1,000").put("billingPeriod", "P1M").put("recurrenceMode", 1)
    val offer = JSONObject().put("basePlanId", basePlanId).put("offerIdToken", "token-$basePlanId")
        .put("offerTags", JSONArray()).put("pricingPhases", JSONArray().put(phase))
    val json = JSONObject().put("productId", productId).put("type", "subs")
        .put("title", productId).put("name", productId).put("description", "test")
        .put("subscriptionOfferDetails", JSONArray().put(offer))
    return ProductDetails(json.toString())     // a real object, not a mock
}
// usage: subscriptionMonthly("premium", "premium-monthly")

On a Galaxy S21 the screen still bounced, and a rebuild from the Sep 26 evening code did the same

On September 28 we opened the details screen on a Galaxy S21. The progress indicator spun, the screen closed, and the app went back to main. The first suspect was the changes from the past two days, so we built a release APK from the Sep 26 evening code (which already had 9290f91) and installed it on the same phone. It failed the same way, after 8 seconds. The billing connection's error was In-app billing API version 3 is not supported, and the phone's Google Play services build was from December 2024.

The phone was the cause, but two app defects showed up with it. When billing setup ended in failure, no disconnect callback followed, so nothing ever reconnected; until the app was restarted, the screen waited the full 8 seconds every time and then showed a "please try again" that had nothing to do with the cause. Commit 023dd3c the same day made the billing manager expose a connection state (connecting, ready, or unavailable with a response code). In 1.6.025 the wait looks like this:

repo.ensureConnected()                       // reconnect if an earlier attempt failed
popupDialog.showDialog(POPUP_TYPE.PROGRESS, true)
lifecycleScope.launch {
    val outcome: Any? = try {
        withTimeoutOrNull(PRODUCT_LOAD_TIMEOUT_MS) {   // 8_000L
            merge(source, repo.connectionState.filterIsInstance<BillingConnectionState.Unavailable>())
                .firstOrNull()                        // products or a failed connection, whichever comes first
        }
    } finally { popupDialog.stopDialog() }
    // Unavailable: (on BILLING/SERVICE_UNAVAILABLE) "check the Play Store" + subscription_unavailable / null (over 8 s): retry toast / else: list the plans
}

On BILLING_UNAVAILABLE it skips the wait, tells the user to check the Play Store, and sends a subscription_unavailable event to GA4.

The day after 'ready to ship', the operator's own test on an S23 led to seven more billing defects

On September 29, version 1.6.025 was logged as ready to ship (the review process that day is in 84 agents reviewed a release). The next morning the operator installed the latest build on a Galaxy S23, tried subscribing, reported that it was not working, and said the release could not go out like this. That prompted another round on the S21 with agent cross-checking: three analysis agents, two refutation passes, three fix agents and two adversarial reviews. It found seven billing defects that had been in the app since 1.6.024 or earlier:

1. When the purchase sheet failed immediately, the progress indicator never went away, and a background thread kept one CPU core busy.
2. The billing setup callback wrote OK into the same global value as purchase results, which read as "subscription complete".
3. A PENDING purchase was treated as a success.
4. Every return to the app created a new BillingClient (a leak), and the Classic screen saw an empty purchase list before the query finished and saved subscribers as free users.
5. The Chat screen did not apply benefits after a purchase.
6. Starting checkout without knowing the current purchases let an existing subscriber buy a second subscription.
7. A screen's onDestroy closed the billing connection the whole app shared.

a839c87 moved purchase results into their own flow, separate from connection state, and read purchases from a signal that fills only after a successful query, so a user is downgraded only when the query succeeded and found no active subscription. 6eb49a1 re-queries purchases right before checkout and does not open the purchase sheet when they are unknown. Counting every fix that day, unit tests went from 250 to 292.

We never read the S23's logs, but the same test on the S21 showed the problem was how we were testing. A sideloaded release build that was never uploaded to Play gets Play's "Google Play billing is not available for this version" message, and the debug build, whose package name ends in .debug, has no products registered in Play. The only way to run a real purchase end to end is to upload to the internal testing track and install from Play with a license tester account, so that check became a step in the release procedure (b88e9bf).

Judge now, wait forever, or wait 8 seconds

Judging right away, the old behavior, treats "the list is empty the moment the screen opens" as an error, so it went. Waiting forever was already half there in the old code: firstOrNull() never returns if the product never comes, so deleting only the size check would have left users staring at an empty screen. We also wrote down the next move in case the 8-second wait was the wrong fix: if checkout starts from details were still 0 to 10% 30 days after release, the product query itself is failing, and the next step is retrying the billing connection.

Do not judge asynchronously loaded product details the moment the screen opens. Set a ceiling on the wait, and when the failure already has a known cause, such as an unavailable billing connection, report it right away instead of waiting. The same two rules now sit in the subscription section of the agent instructions (CLAUDE.md) in this app's repository.

4 of 13 is an interim number, and device-side billing failures stay

The other half of the September 26 rule: 30 days after release, if more than 30% of details-screen users start checkout, the cause is confirmed. The interim 31% on October 17 clears the bar, and two purchases went through, but it rests on 13 users. A log entry on October 18 found that Play's pre-launch robots had inflated version 1.6.030's subscription screen numbers; whether any of these 13 were robots is not recorded. (How we filter those devices in GA4 is in GA4 said 19 installs yesterday and 8 were real.)

On the same day, subscription_unavailable fired for 2 of the 13 (15%). Users whose device cannot bill, through outdated Play services or a sign-in problem, are still there after the fix, and the most the app can do is say so clearly. We have only used TestProductDetails with Play Billing Library 9.1.0, and the Robolectric tests never see a real purchase or Play's signature check.

Checking whether your own subscription screen turns users away

1. Put the plans-screen, details-screen and checkout-start events side by side in GA4, counted by user. If checkout starts from details are in single-digit percent, open the details screen code first.
2. In any function that launches a collector on the product flow, look for a list-size check further down the same function.
3. Check whether queryProductDetailsAsync runs inside a loop and whether the response handler stores products only when all of them arrive. Merge what arrives, and log unfetchedProductList.
4. Have the details screen wait with a progress indicator and a ceiling, and end immediately with a "update Play" message when billing setup finishes with BILLING_UNAVAILABLE or SERVICE_UNAVAILABLE.
5. Create a com.android.billingclient.api package in your test sources with a helper that builds ProductDetails from JSON. Make the three screen cases (waiting, late arrival, never arrives) and the two manager cases (single and split responses) fail first, then fix.
6. Verify real purchases through the internal testing track and a license tester.
7. Before release, write down the decision rule with a date: above what rate the cause is confirmed, below what rate it is something else.

What the October 31 check has to answer

The final check is due on October 31, 30 days after 1.6.025 shipped. First we check for pre-launch robots among the details-screen users. Then we look at two numbers: checkout starts against the 30% line, and subscription_unavailable against 5%, above which a button that goes straight to the Play Store is on the table. The records do not show whether the internal-track real purchase check we added to the release procedure was done before 1.6.025 shipped.

Frequently asked questions

More AX notes

• GA4 said 19 installs yesterday and 8 were real: filtering out test phones and Play pre-launch and review devices
• A 20% staged rollout went out at 100%: a completed release in the Play Publisher API cannot go back to staged
• Pre-gzipping Android assets, measured: zero APK savings, and .pack.gz became .pack with a FileNotFoundException
• A market:// link opens a store chooser on SKT-sold Samsung phones: set the Play package first
• A splash that never moved on: addOnCompleteListener(this) and onStop
• A reverse-construction generator and the gate behind 1,100 arrow-puzzle levels

All notes on Play Store and monetization · All AX notes