Pigweed Eng Blog #2: Feature flags in Bazel builds#

By Ted Pudlik

Published 2024-05-31

Let’s say you’re migrating your build system to Bazel. Your project heavily relies on preprocessor defines to configure its code.

-DBUILD_FEATURE_CPU_PROFILE
-DBUILD_FEATURE_HEAP_PROFILE
-DBUILD_FEATURE_HW_SHA256

In your source files, you use these preprocessor variables to conditionally compile some sections, via #ifdef. When building the same code for different final product configurations, you want to set different defines.

How do you model this in Bazel?

This post discusses three possible approaches:

  1. Easy but limited: bazelrc config with copts

  2. More power with no code changes: Platform-based Skylib flags

  3. Error-preventing approach: Chromium-style build flag pattern

Which one to choose? If you have the freedom to refactor your code to use the Chromium pattern, give it a try! It is not difficult to maintain once implemented, and can prevent real production issues by detecting typos.

Easy but limited: bazelrc config with copts#

Let’s start with the simplest approach: you can put the compiler options into your bazelrc configuration file.

# .bazelrc

common:mydevice_evt1 --copts=-DBUILD_FEATURE_CPU_PROFILE
common:mydevice_evt1 --copts=-DBUILD_FEATURE_HEAP_PROFILE
common:mydevice_evt1 --copts=-DBUILD_FEATURE_HW_SHA256
# and so on

Then, when you build your application, the defines will all be applied:

bazel build --config=mydevice_evt1 //src:application

Configs are expanded recursively, allowing you to group options together and reuse them:

# .bazelrc

common:full_profile --copts=-DBUILD_FEATURE_CPU_PROFILE
common:full_profile --copts=-DBUILD_FEATURE_HEAP_PROFILE

# When building for mydevice_evt1, use full_profile.
common:mydevice_evt1 --config=full_profile

# When building for mydevice_evt2, additionally enable HW_SHA256.
common:mydevice_evt2 --config=full_profile
common:mydevice_evt1 --copts=-DBUILD_FEATURE_HW_SHA256

Downsides#

While it is simple, the config-with-copts approach has a few downsides.

Dangerous typos#

If you misspell BUILDDD_FEATURE_CPU_PROFILE [sic!] in your .bazelrc, the actual BUILD_FEATURE_CPU_PROFILE variable will take the default value of 0. So, although you intended to enable this feature, it will just remain disabled!

This isn’t just a Bazel problem, but a general issue with the simple BUILD_FEATURE macro pattern. If you misspell BUILD_FEATUER_CPU_PROFILE [sic!] in your C++ file, you’ll get it to evaluate to 0 in any build system!

One way to avoid this issue is to use the “Chromium-style” build flag pattern, BUILDFLAG(CPU_PROFILE). If you do, a misspelled or missing define becomes a compiler error. However, the config-with-copts approach is a little too simple to express this pattern, which requires code generation of the build flag headers.

No multi-platform build support#

Bazel allows you to perform multi-platform builds. For example, in a single Bazel invocation you can build a Python flasher program (that will run on your laptop) which embeds as a data dependency the microcontroller firmware to flash (that will run on the microcontroller). We do this in Pigweed’s own examples.

Unfortunately, the config-with-copts pattern doesn’t work nicely with multi-platform build primitives. (Technically, the problem is that Bazel doesn’t support transitioning on –config.) If you want a multi-platform build, you need some more sophisticated idiom.

No build system variables#

This approach doesn’t introduce any variable that can be used within the build system to e.g. conditionally select different source files for a library, choose a different library as a dependency, or remove some targets from the build altogether. We’re really just setting preprocessor defines here.

Limited multirepo support#

The .bazelrc files are not automatically inherited when another repo depends on yours. They can be imported, but it’s an all-or-nothing affair.

More power with no code changes: Platform-based Skylib flags#

Let’s address some shortcomings of the approach above by representing the build features as Skylib flags and grouping them through platform-based flags. (Important note: this feature is still under development! See the Appendix for workarounds for older Bazel versions.)

The platform sets a bunch of flags:

# //platform/BUILD.bazel

# The platform definition
platform(
  name = "mydevice_evt1",
  flags = [
    "--//build/feature:cpu_profile=true",
    "--//build/feature:heap_profile=true",
    "--//build/feature:hw_sha256=true",
  ],
)

The flags have corresponding code-generated C++ libraries:

# //build/feature/BUILD.bazel
load("@bazel_skylib//rules:common_settings.bzl", "bool_flag")
# I'll show one possible implementation of feature_cc_library later.
load("//:feature_cc_library.bzl", "feature_cc_library")

# This is a boolean flag, but there's support for int- and string-valued
# flags, too.
bool_flag(
    name = "cpu_profile",
    build_setting_default = False,
)

# This is a custom rule that generates a cc_library target that exposes
# a header "cpu_profile.h", the contents of which are either,
#
# BUILD_FEATURE_CPU_PROFILE=1
#
# or,
#
# BUILD_FEATURE_CPU_PROFILE=0
#
# depending on the value of the cpu_profile bool_flag. This "code
# generation" is so simple that it can actually be done in pure Starlark;
# see below.
feature_cc_library(
    name = "cpu_profile_cc",
    flag = ":cpu_profile",
)

# Analogous library that exposes the constant in Python.
feature_py_library(
    name = "cpu_profile_py",
    flag = ":cpu_profile",
)

# And in Rust, why not?
feature_rs_library(
    name = "cpu_profile_rs",
    flag = ":cpu_profile",
)

bool_flag(
    name = "heap_profile",
    build_setting_default = False,
)

feature_cc_library(
    name = "heap_profile_cc",
    flag = ":heap_profile",
)

bool_flag(
    name = "hw_sha256",
    build_setting_default = False,
)

feature_cc_library(
    name = "hw_sha256_cc",
    flag = ":hw_sha256",
)

C++ libraries that want to access the variable needs to depend on the cpu_profile_cc (or heap_profile_cc, hw_sha256_cc) library.

Here’s one possible implementation of feature_cc_library:

# feature_cc_library.bzl
load("@bazel_skylib//rules:common_settings.bzl", "BuildSettingInfo")

def feature_cc_library(name, build_setting):
    hdrs_name = name + ".hdr"

    flag_header_file(
        name = hdrs_name,
        build_setting = build_setting,
    )

    native.cc_library(
        name = name,
        hdrs = [":" + hdrs_name],
    )

def _impl(ctx):
    out = ctx.actions.declare_file(ctx.attr.build_setting.label.name + ".h")

    # Convert boolean flags to canonical integer values.
    value = ctx.attr.build_setting[BuildSettingInfo].value
    if type(value) == type(True):
        if value:
            value = 1
        else:
            value = 0

    ctx.actions.write(
        output = out,
        content = r"""
#pragma once
#define {}={}
""".format(ctx.attr.build_setting.label.name.upper(), value),
    )
    return [DefaultInfo(files = depset([out]))]

flag_header_file = rule(
    implementation = _impl,
    attrs = {
        "build_setting": attr.label(
            doc = "Build setting (flag) to construct the header from.",
            mandatory = True,
        ),
    },
)

Advantages#

Composability of platforms#

A neat feature of the simple config-based approach was that configs could be composed through recursive expansion. Fortunately, platforms can be composed, too! There are two mechanisms for doing so:

  1. Use platforms’ support for inheritance. This allows “subplatforms” to override entries from “superplatforms”. But, only single inheritance is supported (each platform has at most one parent).

  2. The other approach is to compose lists of flags directly, through concatenation:

    FEATURES_CORTEX_M7 = [
  "--//build/feature:some_feature",
]

FEATURES_MYDEVICE_EVT1 = FEATURES_CORTEX_M7 + [
  "--//build/feature:some_other_feature",
]

platform(
  name = "mydevice_evt1",
  flags = FEATURES_MYDEVICE_EVT1,
)

    Concatenation doesn’t allow overriding entries, but frees you from the single-parent limitation of inheritance.

    Tip

    This approach can also be used to define custom host platforms: HOST_CONSTRAINTS in @local_config_platform//:constraints.bzl contains the autodetected @platform//os and @platforms//cpu constraints set by Bazel’s default host platform.

Multi-platform build support#

How do you actually associate the platform with a binary you want to build? One approach is to just specify the platform on the command-line when building a cc_binary:

bazel build --platforms=//platform:mydevice_evt1 //src:main

But another approach is to leverage multi-platform build, through platform_data:

# //src/BUILD.bazel
load("@rules_platform//platform_data:defs.bzl", "platform_data")

cc_binary(name = "main")

platform_data(
    name = "main_mydevice_evt1",
    target = ":main",
    platform = "//platform:mydevice_evt1",
)

Then you can keep your command-line simple:

bazel build //src:main_mydevice_evt1

Flags correspond to build variables#

You can make various features of the build conditional on the value of the flag. For example, you can select different dependencies:

# //build/feature/BUILD.bazel
config_setting(
  name = "hw_sha256=true",
  flag_values = {
    ":hw_sha256": "true",
  },
)

# //src/BUILD.bazel
cc_library(
  name = "my_library",
  deps = [
    "//some/unconditional:dep",
  ] + select({
    "//build/feature:hw_sha256=true": ["//extra/dep/for/hw_sha256:only"],
    "//conditions:default": [],
})

Any Bazel rule attribute described as “configurable” can take a value that depends on the flag in this way. Library header lists and source lists are common examples, but the vast majority of attributes in Bazel are configurable.

Downsides#

Typos remain dangerous#

If you used “Chromium-style” build flags you would be immune to dangerous typos when using this Bazel pattern. But until then, you still have this problem, and actually it got worse!

If you forget to #include "build/features/hw_sha256.h" in the C++ file that references the preprocessor variable, the build system or compiler will still not yell at you. Instead, the BUILD_FEATURE_HA_SHA256 variable will take the default value of 0.

This is similar to the typo problem with the config approach, but worse, because it’s easier to miss an #include than to misspell a name, and you’ll need to add these #include statements in many places.

One way to mitigate this problem is to make the individual feature_cc_library targets private, and gather them into one big library that all targets will depend on:

feature_cc_library(
    name = "cpu_profile_cc",
    flag = ":cpu_profile",
    visibility = ["//visibility:private"],
)

feature_cc_library(
    name = "heap_profile_cc",
    flag = ":heap_profile",
    visibility = ["//visibility:private"],
)

feature_cc_library(
    name = "hw_sha256_cc",
    flag = ":hw_sha256",
    visibility = ["//visibility:private"],
)

# Code-generated cc_library that #includes all the individual
# feature_cc_library headers.
all_features_cc_library(
    name = "all_features",
    deps = [
        ":cpu_profile_cc",
        ":heap_profile_cc",
        ":hw_sha256_cc",
        # ... and many more.
    ],
    visibility = ["//visibility:public"],
)

However, a more satisfactory solution is to adopt Chromium-style build flags, which we discuss next.

Build settings have mandatory default values#

The Skylib bool_flag that represents the build flag within Bazel has a build_setting_default attribute. This attribute is mandatory.

This may be a disappointment if you were hoping to provide no default, and have Bazel return errors if no value is explicitly set for a flag (either via a platform, through .bazelrc, or on the command line). The Skylib build flags don’t support this.

The danger here is that the default value may be unsafe, and you forget to override it when adding a new platform (or for some existing platform, when adding a new flag).

There is an alternative pattern that allows you to define default-less build flags: instead of representing build flags as Skylib flags, you can represent them as constraint_setting objects. I won’t spell this pattern out in this blog post, but it comes with its own drawbacks:

  • The custom code-generation rules are more complex, and need to parse the constraint_value names to infer the build flag values.

  • All supported flag values must be explicitly enumerated in the BUILD files, and the code-generation rules need explicit dependencies on them. This leads to substantially more verbose BUILD files.

On the whole, I’d recommend sticking with the Skylib flags!

Error-preventing approach: Chromium-style build flag pattern#

This pattern builds on More power with no code changes: Platform-based Skylib flags by adding a macro helper for retrieving flag values that guards against typos. The BUILD.bazel files look exactly the same as in the previous section, but:

  1. Users of flags access them in C++ files via BUILDFLAG(SOME_NAME).

  2. The code generated by feature_cc_library is a little more elaborate than a plain SOME_NAME=1 or SOME_NAME=0, and it includes a dependency on the Chromium build flag header.

Here’s the feature_cc_library implementation:

load("@bazel_skylib//rules:common_settings.bzl", "BuildSettingInfo")

def feature_cc_library(name, build_setting):
    """Generates a cc_library from a common build setting.

    The generated cc_library exposes a header [build_setting.name].h that
    defines a corresponding build flag.

    Example:

        feature_cc_library(
            name = "evt1_cc",
            build_setting = ":evt1",
        )

    *  This target is a cc_library that exposes a header you can include via
       #include "build/flags/evt1.h".
    *  That header defines a build flag you can access in your code through
       BUILDFLAGS(EVT1).
    *  If you wish to use the build flag from a cc_library, add the target
       evt1_cc to your cc_library's deps.

    Args:
      name: Name for the generated cc_library.
      build_setting: One of the Skylib "common settings": bool_flag, int_flag,
        string_flag, etc. See
        https://github.com/bazelbuild/bazel-skylib/blob/main/docs/common_settings_doc.md
    """
    hdrs_name = name + ".hdr"

    flag_header_file(
        name = hdrs_name,
        build_setting = build_setting,
    )

    native.cc_library(
        name = name,
        hdrs = [":" + hdrs_name],
        # //:buildflag is a cc_library containing the
        # Chromium build flag header.
        deps = ["//:buildflag"],
    )

def _impl(ctx):
    out = ctx.actions.declare_file(ctx.attr.build_setting.label.name + ".h")

    # Convert boolean flags to canonical integer values.
    value = ctx.attr.build_setting[BuildSettingInfo].value
    if type(value) == type(True):
        if value:
            value = 1
        else:
            value = 0

    ctx.actions.write(
        output = out,
        content = r"""
#pragma once

#include "buildflag.h"

#define BUILDFLAG_INTERNAL_{}() ({})
""".format(ctx.attr.build_setting.label.name.upper(), value),
    )
    return [DefaultInfo(files = depset([out]))]

flag_header_file = rule(
    implementation = _impl,
    attrs = {
        "build_setting": attr.label(
            doc = "Build setting (flag) to construct the header from.",
            mandatory = True,
        ),
    },
)

Bottom line#

If you have the freedom to refactor your code to use the Chromium pattern, Bazel provides safe and convenient idioms for expressing configuration through build flags. Give it a try!

Otherwise, you can still use platform-based Skylib flags, but beware typos and missing #include statements!

Appendix#

A couple “deep in the weeds” questions came up while this blog post was being reviewed. I thought they were interesting enough to discuss here, for the interested reader!

Why isn’t the reference code a library?#

If you made it this far you might be wondering, why is the code listing for feature_cc_library even here? Why isn’t it just part of Pigweed, and used in our own codebase?

The short answer is that Pigweed is middleware supporting multiple build systems, so we don’t want to rely on the build system to generate configuration headers.

But the longer answer has to do with how this blog post came about. Some time ago, I was migrating team A’s build from CMake to Bazel. They used Chromium build flags, but in CMake, so to do a build migration they needed Bazel support for this pattern. So I put an implementation together. I wrote a design document, but it had confidential details and was not widely shared.

Then team B comes along and says, “we tried migrating to Bazel but couldn’t figure out how to support build flags” (not the Chromium flags, but the “naive” kind; i.e. their problem statement was exactly the one the blog opens with). So I wrote a less-confidential but still internal doc for them saying “here’s how you could do it”; basically, More power with no code changes: Platform-based Skylib flags.

Then Pigweed’s TL comes along and says “Ted, don’t you feel like spending a day fighting with RST [the markup we use for pigweed.dev]?” Sorry, actually they said something more like, “Why is this doc internal, can’t we share this more widely”? Well we can. So that’s the doc you’re reading now!

But arguably the story shouldn’t end here: Pigweed should probably provide a ready-made implementation of Chromium build flags for downstream projects. See b/342454993 to check out how that’s going!

Do you need to generate actual files?#

If you are a Bazel expert, you may ask: do we need to have Bazel write out the actual header files, and wrap those in a cc_library? If we’re already writing a custom rule for feature_cc_library, can we just set -D defines by providing CcInfo? That is, do something like this:

define = "{}={}"..format(
  ctx.attr.build_setting.label.name.upper(),
  value)
return [CcInfo(
  compilation_context=cc_common.create_compilation_context(
    defines=depset([define])))]

The honest answer is that this didn’t occur to me! But one reason to prefer writing out the header files is that this approach generalizes in an obvious way to other programming languages: if you want to generate Python or Golang constants, you can use the same pattern, just change the contents of the file. Generalizing the CcInfo approach is trickier!

What can I do on older Bazel versions?#

This blog focused on describing approaches that rely on platform-based flags. But this feature is very new: in fact, as of this writing, it is still under development, so it’s not available in any Bazel version! So what can you do?

One approach is to define custom wrapper rules for your cc_binary targets that use a transition to set the flags. You can see examples of such transitions in the Pigweed examples project.

On this page