pw_compilation_testing#

The pw_compilation_testing module provides for negative compilation (NC) testing. Negative compilation tests ensure that code that should not compile does not compile. Negative compilation testing is helpful in a variety of scenarios, for example:

  • Testing for compiler errors, such as [[nodiscard]] checks.

  • Testing that a template cannot be instantiated with certain types.

  • Testing that a static_assert statement is triggered as expected.

  • For a constexpr function, testing that a PW_ASSERT is triggered as expected.

Negative compilation tests are only supported in GN currently. Negative compilation tests are not currently supported in GN on Windows due to b/241565082.

Warning

This module is in an early, experimental state. Do not use it unless you have consulted with the Pigweed team.

Negative compilation test example#

#include "pw_unit_test/framework.h"
#include "pw_compilation_testing/negative_compilation.h"

template <int kValue>
struct MyStruct {
  static_assert(kValue % 2 == 0, "wrong number!");

  constexpr int MultiplyOdd(int runtime_value) const {
    PW_ASSERT(runtime_value % 2 == 0);
    return kValue * runtime_value;
  }
};

[[maybe_unused]] MyStruct<16> this_one_works;

// NC tests cannot be compiled, so they are created in preprocessor #if or
// #elif blocks. These NC tests check that a static_assert statement fails if
// the code is compiled.
#if PW_NC_TEST(NegativeOddNumber)
PW_NC_EXPECT("wrong number!");
[[maybe_unused]] MyStruct<-1> illegal;
#elif PW_NC_TEST(PositiveOddNumber)
PW_NC_EXPECT("wrong number!");
[[maybe_unused]] MyStruct<5> this_is_illegal;
#endif  // PW_NC_TEST

struct Foo {
  // Negative compilation tests can go anywhere in a source file.
#if PW_NC_TEST(IllegalValueAsClassMember)
  PW_NC_EXPECT("wrong number!");
  MyStruct<12> also_illegal;
#endif  // PW_NC_TEST
};

TEST(MyStruct, MultiplyOdd) {
  MyStruct<5> five;
  EXPECT_EQ(five.MultiplyOdd(3), 15);

  // This NC test checks that a specific PW_ASSERT() fails when expected.
  // This only works in an NC test if the PW_ASSERT() fails while the compiler
  // is executing constexpr code. The test code is used in a constexpr
  // statement to force compile-time evaluation.
  #if PW_NC_TEST(MyStruct_MultiplyOdd_AssertsOnOddNumber)
  [[maybe_unused]] constexpr auto fail = [] {
    PW_NC_EXPECT("PW_ASSERT\(runtime_value % 2 == 0\);");
    MyStruct<3> my_struct;
    return my_struct.MultiplyOdd(4);  // Even number, PW_ASSERT should fail.
  }();
  #endif  // PW_NC_TEST
}

// PW_NC_TESTs can be conditionally executed using preprocessor conditionals.
#if PW_CXX_STANDARD_IS_SUPPORTED(20)
#if PW_NC_TEST(RequiresSomeCpp20Feature)
[[maybe_unused]] constinit MyStruct<4> constinit_works;
#endif // PW_NC_TEST
#endif // PW_CXX_STANDARD_IS_SUPPORTED(20)

Creating a negative compilation test#

  • Declare a pw_cc_negative_compilation_test() GN target or set negative_compilation_test = true in a pw_test() target.

  • Add the test to the build in a toolchain with negative compilation testing enabled (pw_compilation_testing_NEGATIVE_COMPILATION_ENABLED = true).

  • In the test source files, add #include "pw_compilation_testing/negative_compilation.h".

  • Use the PW_NC_TEST(TestName) macro in a #if statement.

  • Immediately after the PW_NC_TEST(TestName), provide one or more Python-style regular expressions with the PW_NC_EXPECT() macro, one per line.

  • Execute the tests by running the build.

To simplify parsing, all PW_NC_TEST() statements must fit on a single line and cannot have any other code before or after them. PW_NC_EXPECT() statements may span multiple lines, but must contain a single regular expression as a string literal. The string may be comprised of multiple implicitly concatenated string literals. The PW_NC_EXPECT() statement cannot contain anything else except for //-style comments.

Test assertions#

Negative compilation tests must have at least one assertion about the compilation output. The assertion macros must be placed immediately after the line with the PW_NC_TEST() or the test will fail.

PW_NC_EXPECT(regex_string_literal)#

When negative compilation tests are run, checks the compilation output for the provided regular expression. The argument to the PW_NC_EXPECT() statement must be a string literal. The literal is interpreted character-for-character as a Python raw string literal and compiled as a Python re regular expression.

For example, PW_NC_EXPECT("something (went|has gone) wrong!") searches the failed compilation output with the Python regular expression re.compile("something (went|has gone) wrong!").

PW_NC_EXPECT_GCC(regex_string_literal)#

Same as PW_NC_EXPECT, but only applies when compiling with GCC.

PW_NC_EXPECT_CLANG(regex_string_literal)#

Same as PW_NC_EXPECT, but only applies when compiling with Clang.

Test expectation tips

Be as specific as possible, but avoid compiler-specific error text. Try matching against the following:

  • static_assert messages.

  • Contents of specific failing lines of source code: PW_NC_EXPECT("PW_ASSERT\(!empty\(\));").

  • Comments on affected lines: PW_NC_EXPECT("// Cannot construct from nullptr").

  • Function names: PW_NC_EXPECT("SomeFunction\(\).*private").

Do not match against the following:

  • Source file paths.

  • Source line numbers.

  • Compiler-specific wording of error messages, except when necessary.

Design#

The basic flow for negative compilation testing is as follows.

  • The user defines negative compilation tests in preprocessor #if blocks using the PW_NC_TEST() and PW_NC_EXPECT macros.

  • The build invokes the pw_compilation_testing.generator script. The generator script:

    • finds PW_NC_TEST() statements and extracts a list of test cases,

    • finds all associated PW_NC_EXPECT statements, and

    • generates build targets for each negative compilation tests, passing the test information and expectations to the targets.

  • The build compiles the test source file with all tests disabled.

  • The build invokes the negative compilation test targets, which run the pw_compilation_testing.runner script. The test runner script:

    • invokes the compiler, setting a preprocessor macro that enables the #if block for the test.

    • captures the compilation output, and

    • checks the compilation output for the PW_NC_EXPECT expressions.

  • If compilation failed, and the output matches the test case’s PW_NC_EXPECT expressions, the test passes.

  • If compilation succeeded or the PW_NC_EXPECT expressions did not match the output, the test fails.

Existing frameworks#

Pigweed’s negative compilation tests were inspired by Chromium’s no-compile tests tests and a similar framework used internally at Google. Pigweed’s negative compilation testing framework improves on these systems in a few respects:

  • Trivial integration with unit tests. Negative compilation tests can easily be placed alongside other unit tests instead of in separate files.

  • Safer, more natural macro-based API for test declarations. Other systems use #ifdef macro checks to define test cases, which fail silently when there are typos. Pigweed’s framework uses function-like macros, which provide a clean and natural API, catch typos, and ensure the test is integrated with the NC test framework.

  • More readable, flexible test assertions. Other frameworks place assertions in comments after test names, while Pigweed’s framework uses function-like macros. Pigweed also supports compiler-specific assertions.

  • Assertions are required. This helps ensure that compilation fails for the expected reason and not for an accidental typo or unrelated issue.

On this page