Merge "Revert "Revert "Add benchmarks that run simple programs"""ndk-r21endk-r21dndk-r21cndk-r21bndk-r21-rc1ndk-r21-beta2ndk-r21-beta1ndk-r21ndk-release-r21

5 files changed, 418 insertions, 34 deletions

diff --git a/benchmarks/README.md b/benchmarks/README.md
index 2616a51f8..fe447d170 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -1,48 +1,44 @@
-Bionic Benchmarks
-=================
+# Bionic Benchmarks
 
-Bionic benchmarks is a command line tool for measuring the runtimes of libc functions. It is built
-on top of [Google benchmarks](https://github.com/google/benchmark) with some additions to organize
-tests into suites.
+[TOC]
+
+## libc benchmarks (bionic-benchmarks)
 
-Running the benchmarks
-----------------------
+`bionic-benchmarks` is a command line tool for measuring the runtimes of libc functions. It is built
+on top of [Google Benchmark](https://github.com/google/benchmark) with some additions to organize
+tests into suites.
 
 ### Device benchmarks
 
-    $ mma
-    $ adb remount
-    $ adb sync
+    $ mmma bionic/benchmarks
+    $ adb root
+    $ adb sync data
     $ adb shell /data/benchmarktest/bionic-benchmarks/bionic-benchmarks
     $ adb shell /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks
 
-When operated without specifying an xml file, the default is to run all
-of the benchmarks in alphabetical order.
-
-You can use `--benchmark_filter=getpid` to just run benchmarks with "getpid"
-in their name.
+By default, `bionic-benchmarks` runs all of the benchmarks in alphabetical order. Pass
+`--benchmark_filter=getpid` to run just the benchmarks with "getpid" in their name.
 
 ### Host benchmarks
 
-See the benchmarks/run-on-host.sh script. The host benchmarks can be run
-with 32 bit or 64 bit bionic, or the host glibc.
+See the `benchmarks/run-on-host.sh` script. The host benchmarks can be run with 32-bit or 64-bit
+Bionic, or the host glibc.
 
-## Suites
+### XML suites
 
 Suites are stored in the `suites/` directory and can be chosen with the command line flag
-'--bionic_xml'.
+`--bionic_xml`.
 
-To choose a specific xml file, use the `--bionic_xml=FILE.XML` option. By default, this
-option searches for the xml file in the `suites/` directory. If it doesn't exist
-in that directory then the file will be found as relative to the current
-directory. If the option specifies the full path to an xml file such as
-`/data/nativetest/suites/example.xml`, it will be used as is.
+To choose a specific XML file, use the `--bionic_xml=FILE.XML` option. By default, this option
+searches for the XML file in the `suites/` directory. If it doesn't exist in that directory, then
+the file will be found as relative to the current directory. If the option specifies the full path
+to an XML file such as `/data/nativetest/suites/example.xml`, it will be used as-is.
 
-If no xml file is specified through the command-line option, the default is to use `suites/full.xml`.
-However, for the host bionic benchmarks (bionic-benchmarks-glibc), the default
-is to use `suites/host.xml`.
+If no XML file is specified through the command-line option, the default is to use `suites/full.xml`.
+However, for the host bionic benchmarks (`bionic-benchmarks-glibc`), the default is to use
+`suites/host.xml`.
 
-### Format
+### XML suite format
 
 The format for a benchmark is:
 
@@ -55,19 +51,19 @@ The format for a benchmark is:
 </fn>
 ```
 
-xml-specified values for iterations and cpu take precedence over those specified via command line
-(via '--bionic_iterations' and '--bionic_cpu', respectively.)
+XML-specified values for iterations and cpu take precedence over those specified via command line
+(via `--bionic_iterations` and `--bionic_cpu`, respectively.)
 
 To make small changes in runs, you can also schedule benchmarks by passing in their name and a
-space-separated list of arguments via the 'bionic_extra' command line flag, e.g.
-'--bionic_extra="BM_string_memcpy AT_COMMON_SIZES"' or '--bionic_extra="BM_string_memcmp 32 8 8"'
+space-separated list of arguments via the `--bionic_extra` command line flag, e.g.
+`--bionic_extra="BM_string_memcpy AT_COMMON_SIZES"` or `--bionic_extra="BM_string_memcmp 32 8 8"`
 
 Note that benchmarks will run normally if extra arguments are passed in, and it will fail
 with a segfault if too few are passed in.
 
 ### Shorthand
 
-For the sake of brevity, multiple runs can be scheduled in one xml element by putting one of the
+For the sake of brevity, multiple runs can be scheduled in one XML element by putting one of the
 following in the args field:
 
     NUM_PROPS
@@ -81,5 +77,112 @@ the suites directory.
 
 ### Unit Tests
 
-Bionic benchmarks also has its own set of unit tests, which can be run from the binary in
+`bionic-benchmarks` also has its own set of unit tests, which can be run from the binary in
 `/data/nativetest[64]/bionic-benchmarks-tests`
+
+## Process startup time (bionic-spawn-benchmarks)
+
+The `spawn/` subdirectory has a few benchmarks measuring the time used to start simple programs
+(e.g. Toybox's `true` and `sh -c true`). Run it on a device like so:
+
+    m bionic-spawn-benchmarks
+    adb root
+    adb sync data
+    adb shell /data/benchmarktest/bionic-spawn-benchmarks/bionic-spawn-benchmarks
+    adb shell /data/benchmarktest64/bionic-spawn-benchmarks/bionic-spawn-benchmarks
+
+Google Benchmark reports both a real-time figure ("Time") and a CPU usage figure. For these
+benchmarks, the CPU measurement only counts time spent in the thread calling `posix_spawn`, not that
+spent in the spawned process. The real-time is probably more useful, and it is the figure used to
+determine the iteration count.
+
+Locking the CPU frequency seems to improve the results of these benchmarks significantly, and it
+reduces variability.
+
+## Google Benchmark notes
+
+### Repetitions
+
+Google Benchmark uses two settings to control how many times to run each benchmark, "iterations" and
+"repetitions". By default, the repetition count is one. Google Benchmark runs the benchmark a few
+times to determine a sufficiently-large iteration count.
+
+Google Benchmark can optionally run a benchmark run repeatedly and report statistics (median, mean,
+standard deviation) for the runs. To do so, pass the `--benchmark_repetitions` option, e.g.:
+
+    # ./bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --benchmark_repetitions=4
+    ...
+    -------------------------------------------------------------------
+    Benchmark                         Time             CPU   Iterations
+    -------------------------------------------------------------------
+    BM_stdlib_strtoll              27.7 ns         27.7 ns     25290525
+    BM_stdlib_strtoll              27.7 ns         27.7 ns     25290525
+    BM_stdlib_strtoll              27.7 ns         27.7 ns     25290525
+    BM_stdlib_strtoll              27.8 ns         27.7 ns     25290525
+    BM_stdlib_strtoll_mean         27.7 ns         27.7 ns            4
+    BM_stdlib_strtoll_median       27.7 ns         27.7 ns            4
+    BM_stdlib_strtoll_stddev      0.023 ns        0.023 ns            4
+
+There are 4 runs, each with 25290525 iterations. Measurements for the individual runs can be
+suppressed if they aren't needed:
+
+    # ./bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --benchmark_repetitions=4 --benchmark_report_aggregates_only
+    ...
+    -------------------------------------------------------------------
+    Benchmark                         Time             CPU   Iterations
+    -------------------------------------------------------------------
+    BM_stdlib_strtoll_mean         27.8 ns         27.7 ns            4
+    BM_stdlib_strtoll_median       27.7 ns         27.7 ns            4
+    BM_stdlib_strtoll_stddev      0.043 ns        0.043 ns            4
+
+### CPU frequencies
+
+To get consistent results between runs, it can sometimes be helpful to restrict a benchmark to
+specific cores, or to lock cores at specific frequencies. Some phones have a big.LITTLE core setup,
+or at least allow some cores to run at higher frequencies than others.
+
+A core can be selected for `bionic-benchmarks` using the `--bionic_cpu` option or using the
+`taskset` utility. e.g. A Pixel 3 device has 4 Kryo 385 Silver cores followed by 4 Gold cores:
+
+    blueline:/ # /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --bionic_cpu=0
+    ...
+    ------------------------------------------------------------
+    Benchmark                  Time             CPU   Iterations
+    ------------------------------------------------------------
+    BM_stdlib_strtoll       64.2 ns         63.6 ns     11017493
+
+    blueline:/ # /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --bionic_cpu=4
+    ...
+    ------------------------------------------------------------
+    Benchmark                  Time             CPU   Iterations
+    ------------------------------------------------------------
+    BM_stdlib_strtoll       21.8 ns         21.7 ns     33167103
+
+A similar result can be achieved using `taskset`. The first parameter is a bitmask of core numbers
+to pass to `sched_setaffinity`:
+
+    blueline:/ # taskset f /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll
+    ...
+    ------------------------------------------------------------
+    Benchmark                  Time             CPU   Iterations
+    ------------------------------------------------------------
+    BM_stdlib_strtoll       64.3 ns         63.6 ns     10998697
+
+    blueline:/ # taskset f0 /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll
+    ...
+    ------------------------------------------------------------
+    Benchmark                  Time             CPU   Iterations
+    ------------------------------------------------------------
+    BM_stdlib_strtoll       21.3 ns         21.2 ns     33094801
+
+To lock the CPU frequency, use the sysfs interface at `/sys/devices/system/cpu/cpu*/cpufreq/`.
+Changing the scaling governor to `performance` suppresses the warning that Google Benchmark
+otherwise prints:
+
+    ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
+
+Some devices have a `perf-setup.sh` script that locks CPU and GPU frequencies. Some TradeFed
+benchmarks appear to be using the script. For more information:
+ * run `get_build_var BOARD_PERFSETUP_SCRIPT`
+ * run `m perf-setup.sh` to install the script into `${OUT}/data/local/tmp/perf-setup.sh`
+ * see: https://android.googlesource.com/platform/platform_testing/+/refs/heads/master/scripts/perf-setup/
diff --git a/benchmarks/spawn/Android.bp b/benchmarks/spawn/Android.bp
new file mode 100644
index 000000000..ecce8e855
--- /dev/null
+++ b/benchmarks/spawn/Android.bp
@@ -0,0 +1,96 @@
+//
+// Copyright (C) 2019 The Android Open Source Project
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+//  * Redistributions of source code must retain the above copyright
+//    notice, this list of conditions and the following disclaimer.
+//  * Redistributions in binary form must reproduce the above copyright
+//    notice, this list of conditions and the following disclaimer in
+//    the documentation and/or other materials provided with the
+//    distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+// FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+// COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+// INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+// BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+// OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+// AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+// OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+
+cc_benchmark {
+    name: "bionic-spawn-benchmarks",
+    srcs: ["spawn_benchmarks.cpp"],
+    static_libs: ["libbase"],
+
+    // Install these binaries in the same directory as the main benchmark binary.
+    data: [
+        ":bench_noop",
+        ":bench_noop_nostl",
+        ":bench_noop_static",
+    ],
+
+    host_supported: true,
+    target: {
+        darwin: { enabled: false },
+        windows: { enabled: false },
+        linux_glibc_x86: { enabled: false },
+    },
+}
+
+cc_defaults {
+    name: "noop_binary_defaults",
+
+    compile_multilib: "both",
+    multilib: {
+        lib32: { suffix: "32" },
+        lib64: { suffix: "64" },
+    },
+
+    host_supported: true,
+    target: {
+        darwin: { enabled: false },
+        windows: { enabled: false },
+        linux_glibc_x86: { enabled: false },
+    },
+}
+
+cc_binary {
+    defaults: ["noop_binary_defaults"],
+    name: "bench_noop",
+    srcs: ["noop.cpp"],
+
+    // When this binary is installed to host/linux-x86/bin, its runpath is ${ORIGIN}/../lib64, which
+    // is fine for finding host/linux-x86/lib64/libc++.so. When it's installed to
+    // host/linux-x86/benchmarktest64/bionic-spawn-benchmarks, the runpath needs an extra "..".
+    target: {
+        linux_glibc_x86_64: {
+            ldflags: [
+                "-Wl,--rpath,${ORIGIN}/../../lib64",
+            ],
+        },
+    }
+}
+
+cc_binary {
+    defaults: ["noop_binary_defaults"],
+    name: "bench_noop_nostl",
+    srcs: ["noop.cpp"],
+    stl: "none",
+}
+
+cc_binary {
+    defaults: ["noop_binary_defaults"],
+    name: "bench_noop_static",
+    srcs: ["noop.cpp"],
+    static_executable: true,
+    stl: "libc++_static",
+}
diff --git a/benchmarks/spawn/AndroidTest.xml b/benchmarks/spawn/AndroidTest.xml
new file mode 100644
index 000000000..93316754d
--- /dev/null
+++ b/benchmarks/spawn/AndroidTest.xml
@@ -0,0 +1,53 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Copyright (C) 2019 The Android Open Source Project
+     All rights reserved.
+
+     Redistribution and use in source and binary forms, with or without
+     modification, are permitted provided that the following conditions
+     are met:
+      * Redistributions of source code must retain the above copyright
+        notice, this list of conditions and the following disclaimer.
+      * Redistributions in binary form must reproduce the above copyright
+        notice, this list of conditions and the following disclaimer in
+        the documentation and/or other materials provided with the
+        distribution.
+
+     THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+     "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+     LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+     FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+     COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+     INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+     BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+     OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+     AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+     OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+     OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+     SUCH DAMAGE.
+-->
+<configuration description="Runs bionic-spawn-benchmarks.">
+    <option name="test-suite-tag" value="apct" />
+
+    <target_preparer class="com.android.tradefed.targetprep.PushFilePreparer">
+        <option name="cleanup" value="true" />
+        <option name="push" value="bionic-spawn-benchmarks->/data/local/tmp/bionic-spawn-benchmarks" />
+    </target_preparer>
+
+    <!-- TODO(b/120549168): This seems necessary for consistent results on a walleye, but it's not working with atest
+    <target_preparer class="com.android.tradefed.targetprep.PushFilePreparer">
+        <option name="push" value="perf-setup.sh->/data/local/tmp/perf-setup.sh" />
+        <option name="post-push" value="chmod 755 /data/local/tmp/perf-setup.sh;/data/local/tmp/perf-setup.sh" />
+    </target_preparer>
+    -->
+
+    <test class="com.android.tradefed.testtype.GoogleBenchmarkTest" >
+        <option name="native-benchmark-device-path" value="/data/local/tmp" />
+        <option name="benchmark-module-name" value="bionic-spawn-benchmarks" />
+
+        <!-- The GoogleBenchmarkTest class ordinarily expects every file in the benchmark's
+             directory (recursively) to be a google-benchmark binary, so we need this setting to
+             avoid failing on the bench_* noop programs, which don't output benchmark results. -->
+        <option name="file-exclusion-filter-regex" value=".*/bench_[^/]*$"  />
+    </test>
+
+</configuration>
diff --git a/benchmarks/spawn/noop.cpp b/benchmarks/spawn/noop.cpp
new file mode 100644
index 000000000..1f1cac037
--- /dev/null
+++ b/benchmarks/spawn/noop.cpp
@@ -0,0 +1,31 @@
+/*
+ * Copyright (C) 2019 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+int main() {
+  return 0;
+}
diff --git a/benchmarks/spawn/spawn_benchmarks.cpp b/benchmarks/spawn/spawn_benchmarks.cpp
new file mode 100644
index 000000000..e8f7c1792
--- /dev/null
+++ b/benchmarks/spawn/spawn_benchmarks.cpp
@@ -0,0 +1,101 @@
+/*
+ * Copyright (C) 2019 The Android Open Source Project
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+ * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <errno.h>
+#include <spawn.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+#include <android-base/file.h>
+#include <android-base/stringprintf.h>
+#include <benchmark/benchmark.h>
+
+static std::string test_program(const char* name) {
+#if defined(__LP64__)
+  return android::base::GetExecutableDirectory() + "/" + name + "64";
+#else
+  return android::base::GetExecutableDirectory() + "/" + name + "32";
+#endif
+}
+
+extern char** environ;
+
+static void BM_spawn_test(benchmark::State& state, const char* const* argv) {
+  for (auto _ : state) {
+    pid_t child = 0;
+    if (int spawn_err = posix_spawn(&child, argv[0], nullptr, nullptr, const_cast<char**>(argv),
+                                    environ)) {
+      state.SkipWithError(android::base::StringPrintf(
+          "posix_spawn of %s failed: %s", argv[0], strerror(spawn_err)).c_str());
+      break;
+    }
+
+    int wstatus = 0;
+    const pid_t wait_result = TEMP_FAILURE_RETRY(waitpid(child, &wstatus, 0));
+    if (wait_result != child) {
+      state.SkipWithError(android::base::StringPrintf(
+          "waitpid on pid %d for %s failed: %s",
+          static_cast<int>(child), argv[0], strerror(errno)).c_str());
+      break;
+    }
+    if (WIFEXITED(wstatus) && WEXITSTATUS(wstatus) == 127) {
+      state.SkipWithError(android::base::StringPrintf("could not exec %s", argv[0]).c_str());
+      break;
+    }
+  }
+}
+
+#define SPAWN_BENCHMARK(name, ...)                                                    \
+    BENCHMARK_CAPTURE(BM_spawn_test, name, (const char*[]) { __VA_ARGS__, nullptr })  \
+        ->UseRealTime()                                                               \
+        ->Unit(benchmark::kMicrosecond)                                               \
+
+SPAWN_BENCHMARK(noop, test_program("bench_noop").c_str());
+SPAWN_BENCHMARK(noop_nostl, test_program("bench_noop_nostl").c_str());
+SPAWN_BENCHMARK(noop_static, test_program("bench_noop_static").c_str());
+
+// Android has a /bin -> /system/bin symlink, but use /system/bin explicitly so we can more easily
+// compare Bionic-vs-glibc on a Linux desktop machine.
+#if defined(__GLIBC__)
+
+SPAWN_BENCHMARK(bin_true, "/bin/true");
+SPAWN_BENCHMARK(sh_true, "/bin/sh", "-c", "true");
+
+#elif defined(__ANDROID__)
+
+SPAWN_BENCHMARK(system_bin_true, "/system/bin/true");
+SPAWN_BENCHMARK(vendor_bin_true, "/vendor/bin/true");
+SPAWN_BENCHMARK(system_sh_true, "/system/bin/sh", "-c", "true");
+SPAWN_BENCHMARK(vendor_sh_true, "/vendor/bin/sh", "-c", "true");
+
+#endif
+
+BENCHMARK_MAIN();