pw_perf_test#

Micro-benchmarks that are easy to write and run

Unstable C++17

  • Simple: Automatically manages boilerplate like iterations and durations.

  • Easy: Uses an intuitive API that resembles GoogleTest.

  • Reusable: Integrates with modules like pw_log that you already use.

Pigweed’s perf test module provides an easy way to measure performance on any test setup!

Getting started#

You can add a simple performance test using the follow steps:

Configure your toolchain#

If necessary, configure your toolchain for performance testing:

Note

Currently, pw_perf_test provides build integration with Bazel and GN. Performance tests can be built in CMake, but must be built as regular executables.

  • pw_perf_test_timer_backend: Sets the backend used to measure durations. Options include:

    • @pigweed//pw_perf_test:chrono_timer: Uses pw_chrono::SystemClock to measure time.

    • @pigweed//pw_perf_test:arm_cortex_timer: Uses cycle count registers available on ARM-Cortex to measure time.

  • Currently, only the logging event handler is supported for Bazel.

  • pw_perf_test_TIMER_INTERFACE_BACKEND: Sets the backend used to measure durations. Options include:

    • "$dir_pw_perf_test:chrono_timer": Uses pw_chrono::SystemClock to measure time.

    • "$dir_pw_perf_test:arm_cortex_timer": Uses cycle count registers available on ARM-Cortex to measure time.

  • pw_perf_test_MAIN_FUNCTION: Indicates the GN target that provides a main function that sets the event handler and runs tests. The default is "$dir_pw_perf_test:logging_main".

Write a test function#

Write a test function that exercises the behavior you wish to benchmark. For this example, we will simulate doing work with:

1void SimulateWork(size_t a, size_t b) {
2  for (volatile size_t i = 0; i < a * b * 100000; i = i + 1) {
3  }
4}

Creating a performance test is as simple as using the PW_PERF_TEST_SIMPLE macro to name the function and optionally provide arguments to it:

1PW_PERF_TEST_SIMPLE(SimpleFunction, SimulateWork, 2, 4);

If you need to do additional setup as part of your test, you can use the PW_PERF_TEST macro, which provides an explicit pw::perf_test::State reference. the behavior to be benchmarked should be put in a loop that checks State::KeepRunning():

1void TestFunction(pw::perf_test::State& state, size_t a, size_t b) {
2  while (state.KeepRunning()) {
3    SimulateWork(a, b);
4  }
5}
6PW_PERF_TEST(FunctionWithArgs, TestFunction, 2, 4);

You can even use lambdas in place of standalone functions:

 1PW_PERF_TEST_SIMPLE(
 2    SimpleLambda, [](size_t a, size_t b) { SimulateWork(a, b); }, 2, 4);
 3
 4PW_PERF_TEST(
 5    LambdaFunction,
 6    [](pw::perf_test::State& state, size_t a, size_t b) {
 7      while (state.KeepRunning()) {
 8        SimulateWork(a, b);
 9      }
10    },
11    2,
12    4);

Build Your Test#

Add your performance test to the build using the pw_cc_perf_test rule from //pw_perf_test:pw_cc_perf_test.bzl.

Arguments

  • All native.cc_binary arguments are supported.

Example

load("//pw_perf_test:pw_cc_perf_test.bzl", "pw_cc_perf_test")

pw_cc_perf_test(
    name = "my_perf_test",
    srcs = ["my_perf_test.cc"],
)

Add your performance test to the build using the pw_perf_test template. This template creates two sub-targets.

  • <target_name>.lib: The test sources without a main function.

  • <target_name>: The test suite binary, linked against pw_perf_test_MAIN_FUNCTION.

Arguments

  • All pw_executable arguments are supported.

  • enable_if: Boolean indicating whether the test should be built. If false, replaces the test with an empty target. Defaults to true.

Example

import("$dir_pw_perf_test/perf_test.gni")

pw_perf_test("my_perf_test") {
  sources = [ "my_perf_test.cc" ]
  enable_if = device_has_1m_flash
}

Run your test#

To run perf tests from GN, locate the associated binaries from the out directory and run/flash them manually.

Use the default Bazel run command: bazel run //path/to:target.

API reference#

Macros#

PW_PERF_TEST(name, function, ...)#

Defines a performance test.

The Framework will create a State and pass it to the provided function. This function should perform whatever behavior is to be measured in a loop as long as State::KeepRunning() returns true.

Example: 

void TestFunction(::pw::perf_test::State& state, args...) {
  // Create any needed variables.
  while (state.KeepRunning()){
    // Run code to be measured here.
  }
}
PW_PERF_TEST(PerformanceTestName, TestFunction, args...);

PW_PERF_TEST_SIMPLE(name, function, ...)#

Defines a simple performance test.

This macro is similar to PW_PERF_TEST, except that the provided function does not take a State parameter. As a result, the function should NOT call State::KeepRunning(). Instead, the macro calls the function within its own internal state loop.

Example: 

void TestFunction(args...) {
  // Run code to be measured here.
}
PW_PERF_SIMPLE_TEST(PerformanceTestName, TestFunction, args...);

EventHandler#

class EventHandler#

Collects and reports test results.

Both the state and the framework classes use these functions to report what happens at each stage.

Public Functions

virtual void RunAllTestsStart(const TestRunInfo &test_run_info) = 0#

A performance test is starting.

virtual void RunAllTestsEnd() = 0#

A performance test has ended.

virtual void TestCaseStart(const TestCase &test_case) = 0#

A performance test case is starting.

virtual void TestCaseIteration(const TestIteration &test_iteration) = 0#

A performance test case has completed an iteration.

virtual void TestCaseMeasure(const TestMeasurement &test_measurement) = 0#

A performance test case has produced a Measurement.

virtual void TestCaseEnd(const TestCase &test_case) = 0#

A performance test case has ended.

Design#

pw_perf_test uses a Framework singleton similar to that of pw_unit_test. This singleton is statically created, and tests declared using macros such as PW_PERF_TEST will automatically register themselves with it.

A provided main function interacts with the Framework by calling pw::perf_test::RunAllTests and providing an EventHandler. For each registered test, the Framework creates a State object and passes it to the test function.

The State object tracks the number of iterations. It expects the test function to include a loop with the condition of State::KeepRunning. This loop should include the behavior being banchmarked, e.g.

while (state.KeepRunning()) {
  // Code to be benchmarked.
}

In particular, State::KeepRunning should be called exactly once before the first iteration, as in a for or while loop. The State object will use the timer facade to measure the elapsed duration between successive calls to State::KeepRunning.

Additionally, the State object receives a reference to the EventHandler from the Framework, and uses this to report both test progress and performance measurements.

Timers#

Currently, Pigweed provides two implementations of the timer interface. Consumers may provide additional implementations and use them as a backend for the timer facade.

Chrono Timer#

This timer depends pw_chrono and will only measure performance in terms of nanoseconds. It is the default for performance tests on host.

Cycle Count Timer#

On ARM Cortex devices, clock cycles may more accurately measure the actual performance of a benchmark.

This implementation is OS-agnostic, as it directly accesses CPU registers. It enables the DWT register through the DEMCR register. While this provides cycle counts directly from the CPU, it notably overflows if the duration of a test exceeding 2^32 clock cycles. At 100 MHz, this is approximately 43 seconds.

Warning

The interface only measures raw clock cycles and does not take into account other possible sources of pollution such as LSUs, Sleeps and other registers. Read more on the DWT methods of counting instructions.

EventHandlers#

Currently, Pigweed provides one implementation of EventHandler. Consumers may provide additional implementations and use them by providing a dedicated main function that passes the handler to pw::perf_test::RunAllTests.

LoggingEventHandler#

The default method of running performance tests uses a LoggingEventHandler. This event handler only logs the test results to the console and nothing more. It was chosen as the default method due to its portability and to cut down on the time it would take to implement other printing log handlers. Make sure to set a pw_log backend.

Roadmap#

On this page