Project Builder#

pw_build: Integrations for Bazel, GN, and CMake

The pw_build Python module contains a light-weight build command execution library used for projects that require running multiple commands to perform a build. For example: running cmake alongside gn.

Get started

How to write your own build script.

Get Started

Reference

Reference details about the pw_build Python API.

Reference

Get Started#

The quickest way to get started with Project Builder is to create a build script based on existing examples.

Example Build Scripts#

Examples of Project Builder based pw build commands:

Reference#

At a high level:

A single BuildRecipe contains many BuildCommands which are run sequentially.
All BuildRecipe names must be unique and can have dependencies on other recipes.
Multiple BuildRecipes are passed into the ProjectBuilder class which provides options for logging and terminal output.
Calling the run_builds function starts executing commands. This function allows specifying the number of parallel workers (the number of recipes which are executed in parallel).

        flowchart TB
    subgraph BuildRecipeA ["<strong>BuildRecipe</strong>: ninja"]
        buildCommandA1["<strong>BuildCommand</strong><br> gn gen out"]
        buildCommandA2["<strong>BuildCommand</strong><br> ninja -C out default"]
        buildCommandA1-->buildCommandA2
    end

    subgraph BuildRecipeB ["<strong>BuildRecipe</strong>: bazel"]
        buildCommandB1["<strong>BuildCommand</strong><br> bazel build //...:all"]
        buildCommandB2["<strong>BuildCommand</strong><br> bazel test //...:all"]
        buildCommandB1-->buildCommandB2
    end

    ProjectBuilder["builder = <strong>ProjectBuilder</strong>(build_recipes=...)"]
    BuildRecipeA-->ProjectBuilder
    BuildRecipeB-->ProjectBuilder

    run_builds["builder.<strong>run_builds</strong>(workers=1)"]
    ProjectBuilder-->run_builds

BuildCommand#

class pw_build.build_recipe.BuildCommand(build_dir: ~pathlib.Path | None = None, build_system_command: str | None = None, build_system_extra_args: list[str] = <factory>, targets: list[str] = <factory>, command: list[str] = <factory>, run_if: ~typing.Callable[[~pathlib.Path], bool] = <function BuildCommand.<lambda>>, working_dir: ~pathlib.Path | None = None)#

Store details of a single build step.

Example usage:

from pw_build.build_recipe import BuildCommand, BuildRecipe

def should_gen_gn(out: Path):
    return not (out / 'build.ninja').is_file()

cmd1 = BuildCommand(build_dir='out',
                    command=['gn', 'gen', '{build_dir}'],
                    run_if=should_gen_gn)

cmd2 = BuildCommand(build_dir='out',
                    build_system_command='ninja',
                    build_system_extra_args=['-k', '0'],
                    targets=['default']),

Parameters:

build_dir – Output directory for this build command. This can be omitted if the BuildCommand is included in the steps of a BuildRecipe.
build_system_command – This command should end with ninja, make, or bazel.
build_system_extra_args – A list of extra arguments passed to the build_system_command. If running bazel test include test as an extra arg here.
targets – Optional list of targets to build in the build_dir.
command – List of strings to run as a command. These are passed to subprocess.run(). Any instances of the '{build_dir}' string literal will be replaced at run time with the out directory.
run_if – A callable function to run before executing this BuildCommand. The callable takes one Path arg for the build_dir. If the callable returns true this command is executed. All BuildCommands are run by default.
working_dir – Optional working directory to run build command in

BuildCommand Run Filters#

pw_build.build_recipe.should_gn_gen(out: Path) → bool#

Returns True if the gn gen command should be run.

Returns True if build.ninja or args.gn files are missing from the build directory.

pw_build.build_recipe.should_gn_gen_with_args(gn_arg_dict: Mapping[str, bool | str | list | tuple]) → Callable#

Returns a callable which writes an args.gn file prior to checks.

Parameters:: gn_arg_dict – Dictionary of key value pairs to use as gn args.
Returns:: Callable which takes a single Path argument and returns a bool for True if the gn gen command should be run.

The returned function will:

Always re-write the args.gn file.
Return True if build.ninja or args.gn files are missing.

pw_build.build_recipe.should_regenerate_cmake(cmake_generate_command: list[str]) → Callable[[Path], bool]#

Return a callable to determine if cmake should be regenerated.

Parameters:: cmake_generate_command – Full list of args to run cmake.

The returned function will return True signaling CMake should be re-run if:

The provided CMake command does not match an existing args in the cmake_cfg_command.txt file in the build dir.
build.ninja is missing or cmake_cfg_command.txt is missing.

When the function is run it will create the build directory if needed and write the cmake_generate_command args to the cmake_cfg_command.txt file.

BuildRecipe#

class pw_build.build_recipe.BuildRecipe( build_dir: ~pathlib.Path, steps: list[~pw_build.build_recipe.BuildCommand] = <factory>, title: str | None = None, enabled: bool = True, auto_create_build_dir: bool = True, clean_globs: list[str] = <factory>, dependencies: list[str] = <factory>, )#

Dataclass to store a list of BuildCommands.

Example usage:

from pw_build.build_recipe import BuildCommand, BuildRecipe

def should_gen_gn(out: Path) -> bool:
    return not (out / 'build.ninja').is_file()

recipe = BuildRecipe(
    build_dir='out',
    title='Vanilla Ninja Build',
    steps=[
        BuildCommand(command=['gn', 'gen', '{build_dir}'],
                     run_if=should_gen_gn),
        BuildCommand(build_system_command='ninja',
                     build_system_extra_args=['-k', '0'],
                     targets=['default']),
    ],
)

Parameters:

build_dir – Output directory for this BuildRecipe. On init this out dir is set for all included steps.
steps – List of BuildCommands to run.
title – Custom title. The build_dir is used if this is omitted. Each build recipe name must be unique.
auto_create_build_dir – Auto create the build directory and all necessary parent directories before running any build commands.
clean_globs – Glob strings used to match files that should be deleted when removing build outputs.
dependencies – list of build recipe names that this BuildRecipe depends on

ProjectBuilder#

class pw_build.project_builder.ProjectBuilder(build_recipes: ~typing.Sequence[~pw_build.build_recipe.BuildRecipe], jobs: int | None = None, banners: bool = True, keep_going: bool = False, abort_callback: ~typing.Callable = <function _exit>, execute_command: ~typing.Callable[[list, dict, ~pw_build.build_recipe.BuildRecipe, ~pathlib.Path | None, ~logging.Logger, ~typing.Callable | None], bool] = <function execute_command_pure>, charset: ~pw_build.project_builder.ProjectBuilderCharset = ('OK ', 'FAIL', '... '), colors: bool = False, separate_build_file_logging: bool = False, send_recipe_logs_to_root: bool = False, root_logger: ~logging.Logger = <Logger pw_build (WARNING)>, root_logfile: ~pathlib.Path | None = None, log_level: int = 20, allow_progress_bars: bool = True, use_verbatim_error_log_formatting: bool = False, log_build_steps: bool = False, source_path: ~pathlib.Path | None = None, dry_run: bool = False)#

Pigweed Project Builder

Controls how build recipes are executed and logged.

Example usage:

import logging
from pathlib import Path

from pw_build.build_recipe import BuildCommand, BuildRecipe
from pw_build.project_builder import ProjectBuilder

def should_gen_gn(out: Path) -> bool:
    return not (out / 'build.ninja').is_file()

recipe = BuildRecipe(
    build_dir='out',
    title='Vanilla Ninja Build',
    steps=[
        BuildCommand(command=['gn', 'gen', '{build_dir}'],
                     run_if=should_gen_gn),
        BuildCommand(build_system_command='ninja',
                     build_system_extra_args=['-k', '0'],
                     targets=['default']),
    ],
)

project_builder = ProjectBuilder(
    build_recipes=[recipe1, ...]
    banners=True,
    log_level=logging.INFO
    separate_build_file_logging=True,
    root_logger=logging.getLogger(),
    root_logfile=Path('build_log.txt'),
)

Parameters:

build_recipes – List of build recipes. Each build recipe title must be unique.
jobs – The number of jobs bazel, make, and ninja should use by passing -j to each.
banners – Print the project banner at the start of each build.
keep_going – If True keep going flags are passed to bazel and ninja with the -k option.
abort_callback – A callback that is called if a build is aborted.
execute_command – The underlying command to use to execute build steps.
charset – A ProjectBuilderCharset that controls visual elements of the terminal output.
colors – Forcibly enables/disables ANSI colors in stdout and logfiles.
separate_build_file_logging – If True separate logfiles will be created per build recipe. The location of each file depends on if a root_logfile is provided. If a root logfile is used each build recipe logfile will be created in the same location. If no root_logfile is specified the per build log files are placed in each build dir as log.txt
send_recipe_logs_to_root – If True will send all build recipie output to the root logger. This only makes sense to use if the builds are run in serial.
root_logger – The logging.Logger that will be parent to all build recipe logging.
root_logfile – Optional root logfile.
log_level – Optional log_level, defaults to logging.INFO.
allow_progress_bars – If False progress bar output will be disabled.
use_verbatim_error_log_formatting – Use a blank log format when printing errors from sub builds to the root logger.
log_build_steps – If True all build step lines will be logged to the screen and logfiles. Default: False.
source_path – Path to the root of the source files. Defaults to the current working directory. If running under bazel this will be set to the $BUILD_WORKSPACE_DIRECTORY environment variable. Otherwise $PW_PROJECT_ROOT will be used.
dry_run – If True only print shell commands instead of executing them.