I have added macrobenchmark module with StartupTimingMetric and FrameTimingMetric metrics long time ago and my setup was working fine. Since then I have not been running benchmarks due to some reasons. Recently, had time to go back and update the user flows and run the benchmarks again. But my FrameTimingMetric benchmarks started to fail at step of running measureBlock. UiAutomator couldn’t find any object by resource id inside of measureBlock but could see all of them in setupBlock. I’m using Compose and necessary composables have testTag added.

It was working fine a year ago when I have added benchmarks initially, but now they stopped. In the end I managed to fix the problem by removing startupMode = StartupMode.COLD. I have no idea why this flag is affecting the visibility of composables in measureBlock only, while setupBlock works just fine.

Could someone help me understand why this is happening?

// NOT working case

        rule.measureRepeated(
            packageName = InstrumentationRegistry.getArguments().getString("targetAppId")
                ?: throw Exception("targetAppId not passed as instrumentation runner arg"),
            metrics = listOf(FrameTimingMetric()),
            compilationMode = compilationMode,
            startupMode = StartupMode.COLD,
            iterations = 10,
            setupBlock = {
                pressHome()
                startActivityAndWait()
                // other setup methods
                // "MyList" is visible here and list var is NOT null
                // val list = device.findObject(By.res("MyList"))
            },
            measureBlock = {
                // "MyList" is NOT visible here and list var IS null
                val list = device.findObject(By.res("MyList"))
            }
        )

// Working case

        rule.measureRepeated(
            packageName = InstrumentationRegistry.getArguments().getString("targetAppId")
                ?: throw Exception("targetAppId not passed as instrumentation runner arg"),
            metrics = listOf(FrameTimingMetric()),
            compilationMode = compilationMode,
            // startupMode = StartupMode.COLD,      // <- REMOVE this line
            iterations = 10,
            setupBlock = {
                pressHome()
                startActivityAndWait()
                // other setup methods
                // "MyList" is visible here and list var is NOT null
                // val list = device.findObject(By.res("MyList"))
            },
            measureBlock = {
                // "MyList" is visible here and list var is NOT null
                val list = device.findObject(By.res("MyList"))
            }
        )