GA4 said 19 installs yesterday and 8 were real: filtering out test phones and Play pre-launch and review devices

IN2025 came two per version; our S20 logged 0 seconds and 13 data wipes

The split came out of a day-one drop-off review. We had shipped three Block Turn versions on October 17 (1.1.2, 1.1.3 and 1.1.4), and to count who was actually using them we pulled version, country, first-user medium and device model into one table.

IN2025, with no country, arrived two at a time with every version: Play running each uploaded build on its pre-launch report and review devices. The S20 and S21 are ours. They logged app_clear_data again and again with zero engagement. Only the last two rows, both from ads, were people (two users from the United States with no model, left out of the table, were only flagged as likely review devices).

Recounted per day from October 11 to 17 (from the drop-off review):

On October 11, the day Firebase went in, and on October 17, the day of three releases, test and review devices outnumbered real installs.

The gate meant to keep test phones out of GA4 had a gap

The app had a collection gate from the start. The October 11 code that added Firebase turns collection off in debug builds and test-ad builds, with one exception: it turns collection on when a files/analytics_on file exists. The comment says "so device harnesses don't fill the reports." Still, the app ran on the S20 and S21 with collection on, and every harness run wiped its data with pm clear. The records do not settle which build or path turned collection on: a store update, a release build with real ad unit IDs, or the analytics_on file. The S20 had logged two app_update events, and the release build pulled from the S21 on the night of October 18 was a test-ad build.

The app can do nothing about Play's pre-launch and review devices. They come with every upload, so the more often you ship, the more of these installs you get. Three uploads on October 17 produced six IN2025 first opens.

The report tool written on October 17 grouped by country and medium only, with no device dimension. The October 15 analysis has a data-pollution section listing 3 installs and 10 app_clear_data events from the S21 and 2 installs from emulators (sdk_gphone, which then stayed out of the October 18 filter), and its fifth to-do was to consider turning analytics off for release-build device tests or marking test devices as internal traffic in GA4. Nobody acted on that "consider" for three days, and in the meantime the morning report read 19 as real installs.

The report tool now filters by default; --include-test shows everything

The commit at 17:06 on October 18 (dca02f2) appends this condition to every GA4 request the report tool sends. Rows with no country go, and our own phone models go unless the user came from an ad (cpc), so a real S20 or S21 owner who clicked an ad still counts.

TEST_MODELS = ["SM-G981N", "SM-G991N", "IN2025"]
NOT_TEST = {"notExpression": {"orGroup": {"expressions": [
    {"filter": {"fieldName": "country", "inListFilter": {"values": ["(not set)", ""]}}},
    {"andGroup": {"expressions": [
        {"filter": {"fieldName": "mobileDeviceModel", "inListFilter": {"values": TEST_MODELS}}},
        {"notExpression": {"filter": {"fieldName": "firstUserMedium", "stringFilter": {"value": "cpc"}}}}]}}]}}}

We ran the report from October 17 on with the filter and with --include-test side by side. Six non-ad rows with country (not set) disappeared, Korean non-ad installs went from 4 to 1, and the 5 Korean installs from ads stayed. Because the condition is applied at query time, it works on data GA4 already holds, which is how the October 11 to 17 numbers were recounted.

The commit said "DebugView only"; the official docs recommend a developer traffic filter as well

The same commit added one line before pm clear in seven harnesses (E2E, reward checks, monkey runs):

adb shell setprop debug.firebase.analytics.app <package name>

The commit message says "so test runs go to DebugView only," and the script comments, the review document and the report to the operator all said test data would no longer mix into GA4 reports. The same sentence went into another note on this site, The agent said it was tested. The game crashed on the first tutorial tap: a missing VIBRATE permission and the fresh-install monkey test, and was corrected there on October 26, 2026.

Checked against the official docs while writing this note, that sentence does not hold. Firebase's DebugView page says the setting persists until you disable it with .none., says these events are included in the daily BigQuery export by default, and recommends a developer traffic data filter to keep test traffic out of reports. The GA4 help page on that filter describes it as the thing that lets developers keep using DebugView while "their activity doesn't appear in your reports." Neither the repository nor the session logs show that filter being created, and we did not look at the GA4 property settings directly. We also did not measure whether events from setprop runs actually reached this property's reports; that conclusion rests on the docs.

The October 18 report says the setting was not verified on the real S20 and S21 because the agent's permission check blocked the command that changes device settings; that night it was recorded as working on an Android 7 emulator only, and whether events left the reports was not checked then either. Counting at HEAD on October 26, 12 scripts call pm clear and 5 of them still have no setprop. Four existed since October 4 and were missed when the first seven were edited; one was written on October 22.

Five days later an emulator got through, and a week later arm64 devices

On the morning of October 23, the agent corrected its own report from the day before: the "one Block Turn 1.1.5 user" was the emulator it had used for verification. Emulators report an empty mobileDeviceModel to GA4, so they passed the model list untouched. A commit at 08:14 (5fc744b) dropped four x86 emulator names on the deviceModel dimension. Why that run reached GA4 at all is not recorded. The commit message also lists a debug build, 1.1.5-debug, among the versions GA4 showed, though debug builds should have had collection off.

At noon on October 25, a midpoint check put the first-puzzle (R001) solve rate of 1.1.6 at 6 of 12, or 50%. A commit at 16:14 (53d6386) corrected it. Four Play pre-launch and review devices in the United States (sdk_gphone64_arm64, sdk_gphone16k_arm64) were in the count, and the filter from October 23 only knew the x86 names. With the filter widened to every model starting with sdk_gphone, 8 real users remained and 6 solved it: 75%, close to the previous ad cohort's 77% (23 of 30). The same check found that all 6 GA4 users of 1.1.7 and 1.1.8 were Play pre-launch and review devices.

A GA4 data filter could not have fixed last week's numbers

The October 15 document first proposed turning collection off or marking test devices as internal traffic in GA4; the October 18 report proposed marking debug devices or an internal traffic filter. GA4's help says data filters do not affect historical data. Correcting the October 11 to 17 numbers needed a report-side filter regardless. A report-side filter, in turn, only drops the models we know about and misses the next new test device. Our call is to use both: the report filter for past data and Play's pre-launch and review devices, debug mode plus a developer traffic filter for events from our own phones from now on.

Turning collection off in the build was already in place, and it did not stop the app from running with collection on on the S20 and S21.

Real organic users on an S20, S21 or IN2025 are dropped too

The filter also removes real people who install organically on an S20, S21 or IN2025. A few devices only swing the average while real installs are in single digits per day; at a few hundred installs a day, a handful of test phones would disappear in the noise.

Six checks: Data API dimensions, a harness grep and a developer traffic filter

1. Pull first_open from the GA4 Data API by date, country, firstUserMedium, mobileDeviceModel, deviceModel and appVersion.
2. Flag rows with country (not set), a deviceModel starting with sdk_gphone or "Android SDK built for", your own phone models on non-ad installs, and devices with repeated app_clear_data and zero engagement. Be suspicious of any model that shows up in the same number for every new version on release day.
3. Make exclusion the default in your report tool, with a flag that shows everything, and compare the two outputs side by side once.
4. Find harness scripts that call pm clear without debug.firebase.analytics.app: grep -l "pm.*clear" tools/* | xargs grep -L debug.firebase.analytics.app. Run it again whenever a script is added.
5. In GA4 admin, create a developer traffic data filter in the Testing state, confirm harness events get labeled, then activate it. Data an active filter drops is gone for good, so do not skip the Testing state.
6. After every release, look at that version's users by device model before reading any funnel number.

Whoever you report install counts to, do not call a number real installs until you have split it by device model once. A per-device event timeline built from the same GA4 data is in Puzzle game day-one churn: per-device event timelines, exact board math and four walls in chapter one; when you build the test-device list its steps mention, add the sdk_gphone prefix and the country (not set) condition from step 2.

What's left

The next check is to add the developer traffic filter in the Testing state, run one harness on the S20, and see whether those events are labeled. The five scripts without setprop need the same line. After that comes the question of why the October 22 emulator run had collection on.

FAQ

More AX notes

• A month of 'temporary error': our subscription details screen never waited for products, and 1 of 16 users started checkout
• A 20% staged rollout went out at 100%: a completed release in the Play Publisher API cannot go back to staged
• Pre-gzipping Android assets, measured: zero APK savings, and .pack.gz became .pack with a FileNotFoundException
• A market:// link opens a store chooser on SKT-sold Samsung phones: set the Play package first
• A splash that never moved on: addOnCompleteListener(this) and onStop
• A reverse-construction generator and the gate behind 1,100 arrow-puzzle levels

All notes on Play Store and monetization · All AX notes