For #19901: integrate Jetback Benchmark (microbenchmark).
parent
589f166b29
commit
6d609bc651
@ -0,0 +1,66 @@
|
|||||||
|
/* This Source Code Form is subject to the terms of the Mozilla Public
|
||||||
|
* License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||||
|
* file, You can obtain one at https://mozilla.org/MPL/2.0/. */
|
||||||
|
|
||||||
|
// This comment contains the central documentation for how we configured Jetpack Benchmark. Currently:
|
||||||
|
// - microbenchmark: configured differently than recommended (see inline notes below)
|
||||||
|
// - macrobenchmark: not configured
|
||||||
|
//
|
||||||
|
// To run our benchmarks, you need to set the "benchmark" gradle property. You can:
|
||||||
|
// - (preferred) Run via the command line (change the class you run on):
|
||||||
|
// ./gradlew -Pbenchmark app :connectedCheck -Pandroid.testInstrumentationRunnerArguments.class=org.mozilla.fenix.perf.SampleBenchmark
|
||||||
|
// - Use the IDE. Temporarily set the "benchmark" property in app/build.gradle with "ext.benchmark=true"
|
||||||
|
// near the top of the file. DO NOT COMMIT THIS.
|
||||||
|
// - (note: I was unable to get IDE run configurations working)
|
||||||
|
//
|
||||||
|
// To get the results, look at this file (we recommend using the median; results are in nanoseconds):
|
||||||
|
// app/build/outputs/connected_android_test_additional_output/nightlyAndroidTest/connected/<device>/org.mozilla.fenix-benchmarkData.json
|
||||||
|
//
|
||||||
|
// I was unable to get the results to print directly in Android Studio (perhaps it's my device).
|
||||||
|
//
|
||||||
|
// The official documentation suggests configuring microbenchmark in a separate module. This would
|
||||||
|
// require any benchmarked code to be in a library module, not the :app module (see below). To avoid
|
||||||
|
// this requirement, we created the "benchmark" gradle property.
|
||||||
|
//
|
||||||
|
// For the most accurate results, the documentation recommends running tests on rooted devices with
|
||||||
|
// the CPU clock locked.
|
||||||
|
//
|
||||||
|
// See https://developer.android.com/studio/profile/benchmark#what-to-benchmark for when writing a
|
||||||
|
// jetpack microbenchmark is a good fit.
|
||||||
|
|
||||||
|
// I think `android` represents this object:
|
||||||
|
// https://google.github.io/android-gradle-dsl/3.3/com.android.build.gradle.AppExtension.html
|
||||||
|
ext.maybeConfigForJetpackBenchmark = { android ->
|
||||||
|
if (!project.hasProperty("benchmark")) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// The official documentation https://developer.android.com/studio/profile/benchmark#full-setup
|
||||||
|
// recommends setting up the Microbenchmark library in a separate module from your app: AFAICT,
|
||||||
|
// the reason for this is to prevent the benchmarks from being configured against debug
|
||||||
|
// builds. We chose not to do this because it's a lot of work to pull code out into a
|
||||||
|
// separate module just to benchmark it. We were able to replicate the outcome by setting
|
||||||
|
// this testBuildType property.
|
||||||
|
android.testBuildType "nightly"
|
||||||
|
|
||||||
|
// WARNING: our proguard configuration for androidTest is not set up correctly so the tests
|
||||||
|
// fail if we don't disable minification. DISABLING MINIFICATION PRODUCES BENCHMARKS THAT ARE
|
||||||
|
// LESS REPRESENTATIVE TO THE USER EXPERIENCE, however, so we made this tradeoff to reduce
|
||||||
|
// implementation time.
|
||||||
|
project.ext.disableOptimization = true
|
||||||
|
|
||||||
|
android.defaultConfig {
|
||||||
|
// WARNING: the benchmark framework warns you if you're running the test in a configuration
|
||||||
|
// that will compromise the accuracy of the results. Unfortunately, I couldn't get everything
|
||||||
|
// working so I had to suppress some things.
|
||||||
|
//
|
||||||
|
// - ACTIVITY-MISSING: we're supposed to use the test instrumentation runner,
|
||||||
|
// "androidx.benchmark.junit4.AndroidBenchmarkRunner". However, when I do so, I get an error
|
||||||
|
// that we're unable to launch the activity. My understanding is that this runner will use an
|
||||||
|
// "IsolationActivity" to reduce the impact of other work on the device from affecting the benchmark
|
||||||
|
// and to opt into a lower-max CPU frequency on unrooted devices that support it
|
||||||
|
// - UNLOCKED: ./gradlew lockClocks, which locks the CPU frequency, fails on my device. See
|
||||||
|
// https://issuetracker.google.com/issues/176836267 for potential workarounds.
|
||||||
|
testInstrumentationRunnerArgument 'androidx.benchmark.suppressErrors', 'ACTIVITY-MISSING,UNLOCKED'
|
||||||
|
}
|
||||||
|
}
|
@ -0,0 +1,38 @@
|
|||||||
|
/* This Source Code Form is subject to the terms of the Mozilla Public
|
||||||
|
* License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||||
|
* file, You can obtain one at https://mozilla.org/MPL/2.0/. */
|
||||||
|
|
||||||
|
package org.mozilla.fenix.perf
|
||||||
|
|
||||||
|
import androidx.benchmark.junit4.BenchmarkRule
|
||||||
|
import androidx.benchmark.junit4.measureRepeated
|
||||||
|
import androidx.test.ext.junit.runners.AndroidJUnit4
|
||||||
|
import org.junit.Ignore
|
||||||
|
import org.junit.Rule
|
||||||
|
import org.junit.Test
|
||||||
|
import org.junit.runner.RunWith
|
||||||
|
|
||||||
|
/**
|
||||||
|
* To run this benchmark:
|
||||||
|
* - Comment out @Ignore: DO NOT COMMIT THIS!
|
||||||
|
* - See run instructions in app/benchmark.gradle
|
||||||
|
*
|
||||||
|
* See https://developer.android.com/studio/profile/benchmark#write-benchmark for how to write a
|
||||||
|
* real benchmark, including testing UI code. See
|
||||||
|
* https://developer.android.com/studio/profile/benchmark#what-to-benchmark for when jetpack
|
||||||
|
* microbenchmark is a good fit.
|
||||||
|
*/
|
||||||
|
@Ignore("This is a sample: we don't want it to run when we run all the tests")
|
||||||
|
@RunWith(AndroidJUnit4::class)
|
||||||
|
class SampleBenchmark {
|
||||||
|
@get:Rule
|
||||||
|
val benchmarkRule = BenchmarkRule()
|
||||||
|
|
||||||
|
@Test
|
||||||
|
fun additionBenchmark() = benchmarkRule.measureRepeated {
|
||||||
|
var i = 0
|
||||||
|
while (i < 10_000_000) {
|
||||||
|
i++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
Loading…
Reference in New Issue