Project Builder#

pw_build: Integrations for Bazel, GN, and CMake

The pw_build Python module contains a light-weight build command execution library used for projects that require running multiple commands to perform a build. For example: running cmake alongside gn.

Get started

How to write your own build script.

Reference

Reference details about the pw_build Python API.

pw build CLI

Command line interface usage.

Get Started#

The quickest way to get started with Project Builder is to create a build script based on existing examples.

Example Build Scripts#

Examples of Project Builder based pw build commands:

Upstream Pigweed repo main.py <main:pw_build/py/pw_build/main.py>
Examples repo build_project.py
Kudzu repo build_project.py

Reference#

At a high level:

A single BuildRecipe contains many BuildCommands which are run sequentially.
Multiple BuildRecipes can be created and passed into the ProjectBuilder class which provides logging and terminal output options.
A ProjectBuilder instance is then passed to the run_builds function which allows specifying the number of parallel workers (the number of recipes which are executed in parallel).

flowchart TB subgraph BuildRecipeA ["BuildRecipe: ninja"] buildCommandA1["BuildCommand gn gen out"] buildCommandA2["BuildCommand ninja -C out default"] buildCommandA1-->buildCommandA2 end subgraph BuildRecipeB ["BuildRecipe: bazel"] buildCommandB1["BuildCommand bazel build //...:all"] buildCommandB2["BuildCommand bazel test //...:all"] buildCommandB1-->buildCommandB2 end ProjectBuilder["ProjectBuilder(build_recipes=...)"] BuildRecipeA-->ProjectBuilder BuildRecipeB-->ProjectBuilder run_builds["run_builds(project_builder=..., workers=1)"] ProjectBuilder-->run_builds

BuildCommand#

class pw_build.build_recipe.BuildCommand(build_dir: ~pathlib.Path | None = None, build_system_command: str | None = None, build_system_extra_args: list[str] = <factory>, targets: list[str] = <factory>, command: list[str] = <factory>, run_if: ~typing.Callable[[~pathlib.Path], bool] = <function BuildCommand.<lambda>>, working_dir: ~pathlib.Path | None = None)#

Store details of a single build step.

Example usage:

from pw_build.build_recipe import BuildCommand, BuildRecipe

def should_gen_gn(out: Path):
    return not (out / 'build.ninja').is_file()

cmd1 = BuildCommand(build_dir='out',
                    command=['gn', 'gen', '{build_dir}'],
                    run_if=should_gen_gn)

cmd2 = BuildCommand(build_dir='out',
                    build_system_command='ninja',
                    build_system_extra_args=['-k', '0'],
                    targets=['default']),

Parameters:

build_dir – Output directory for this build command. This can be omitted if the BuildCommand is included in the steps of a BuildRecipe.
build_system_command – This command should end with ninja, make, or bazel.
build_system_extra_args – A list of extra arguments passed to the build_system_command. If running bazel test include test as an extra arg here.
targets – Optional list of targets to build in the build_dir.
command – List of strings to run as a command. These are passed to subprocess.run(). Any instances of the '{build_dir}' string literal will be replaced at run time with the out directory.
run_if – A callable function to run before executing this BuildCommand. The callable takes one Path arg for the build_dir. If the callable returns true this command is executed. All BuildCommands are run by default.
working_dir – Optional working directory to run build command in

BuildCommand Run Filters#

pw_build.build_recipe.should_gn_gen(out: Path) → bool#

Returns True if the gn gen command should be run.

Returns True if build.ninja or args.gn files are missing from the build directory.

pw_build.build_recipe.should_gn_gen_with_args(gn_arg_dict: Mapping[str, bool | str | list | tuple]) → Callable#

Returns a callable which writes an args.gn file prior to checks.

Parameters:: gn_arg_dict – Dictionary of key value pairs to use as gn args.
Returns:: Callable which takes a single Path argument and returns a bool for True if the gn gen command should be run.

The returned function will:

Always re-write the args.gn file.
Return True if build.ninja or args.gn files are missing.

pw_build.build_recipe.should_regenerate_cmake(cmake_generate_command: list[str]) → Callable[[Path], bool]#

Return a callable to determine if cmake should be regenerated.

Parameters:: cmake_generate_command – Full list of args to run cmake.

The returned function will return True signaling CMake should be re-run if:

The provided CMake command does not match an existing args in the cmake_cfg_command.txt file in the build dir.
build.ninja is missing or cmake_cfg_command.txt is missing.

When the function is run it will create the build directory if needed and write the cmake_generate_command args to the cmake_cfg_command.txt file.

BuildRecipe#

class pw_build.build_recipe.BuildRecipe( build_dir: ~pathlib.Path, steps: list[~pw_build.build_recipe.BuildCommand] = <factory>, title: str | None = None, enabled: bool = True, auto_create_build_dir: bool = True, clean_globs: list[str] = <factory>, )#

Dataclass to store a list of BuildCommands.

Example usage:

from pw_build.build_recipe import BuildCommand, BuildRecipe

def should_gen_gn(out: Path) -> bool:
    return not (out / 'build.ninja').is_file()

recipe = BuildRecipe(
    build_dir='out',
    title='Vanilla Ninja Build',
    steps=[
        BuildCommand(command=['gn', 'gen', '{build_dir}'],
                     run_if=should_gen_gn),
        BuildCommand(build_system_command='ninja',
                     build_system_extra_args=['-k', '0'],
                     targets=['default']),
    ],
)

Parameters:

build_dir – Output directory for this BuildRecipe. On init this out dir is set for all included steps.
steps – List of BuildCommands to run.
title – Custom title. The build_dir is used if this is ommited.
auto_create_build_dir – Auto create the build directory and all necessary parent directories before running any build commands.

ProjectBuilder#

class pw_build.project_builder.ProjectBuilder(build_recipes: ~typing.Sequence[~pw_build.build_recipe.BuildRecipe], jobs: int | None = None, banners: bool = True, keep_going: bool = False, abort_callback: ~typing.Callable = <function _exit>, execute_command: ~typing.Callable[[list, dict, ~pw_build.build_recipe.BuildRecipe, ~pathlib.Path | None, ~logging.Logger, ~typing.Callable | None], bool] = <function execute_command_no_logging>, charset: ~pw_build.project_builder.ProjectBuilderCharset = ('OK ', 'FAIL', '... '), colors: bool = False, separate_build_file_logging: bool = False, send_recipe_logs_to_root: bool = False, root_logger: ~logging.Logger = <Logger pw_build (WARNING)>, root_logfile: ~pathlib.Path | None = None, log_level: int = 20, allow_progress_bars: bool = True, use_verbatim_error_log_formatting: bool = False, log_build_steps: bool = False, source_path: ~pathlib.Path | None = None)#

Pigweed Project Builder

Controls how build recipes are executed and logged.

Example usage:

import logging
from pathlib import Path

from pw_build.build_recipe import BuildCommand, BuildRecipe
from pw_build.project_builder import ProjectBuilder

def should_gen_gn(out: Path) -> bool:
    return not (out / 'build.ninja').is_file()

recipe = BuildRecipe(
    build_dir='out',
    title='Vanilla Ninja Build',
    steps=[
        BuildCommand(command=['gn', 'gen', '{build_dir}'],
                     run_if=should_gen_gn),
        BuildCommand(build_system_command='ninja',
                     build_system_extra_args=['-k', '0'],
                     targets=['default']),
    ],
)

project_builder = ProjectBuilder(
    build_recipes=[recipe1, ...]
    banners=True,
    log_level=logging.INFO
    separate_build_file_logging=True,
    root_logger=logging.getLogger(),
    root_logfile=Path('build_log.txt'),
)

Parameters:

build_recipes – List of build recipes.
jobs – The number of jobs bazel, make, and ninja should use by passing -j to each.
banners – Print the project banner at the start of each build.
keep_going – If True keep going flags are passed to bazel and ninja with the -k option.
abort_callback – A callback that is called if a build is aborted.
execute_command – The underlying command to use to execute build steps.
charset – A ProjectBuilderCharset that controls visual elements of the terminal output.
colors – Forcibly enables/disables ANSI colors in stdout and logfiles.
separate_build_file_logging – If True separate logfiles will be created per build recipe. The location of each file depends on if a root_logfile is provided. If a root logfile is used each build recipe logfile will be created in the same location. If no root_logfile is specified the per build log files are placed in each build dir as log.txt
send_recipe_logs_to_root – If True will send all build recipie output to the root logger. This only makes sense to use if the builds are run in serial.
root_logger – The logging.Logger that will be parent to all build recipe logging.
root_logfile – Optional root logfile.
log_level – Optional log_level, defaults to logging.INFO.
allow_progress_bars – If False progress bar output will be disabled.
use_verbatim_error_log_formatting – Use a blank log format when printing errors from sub builds to the root logger.
log_build_steps – If True all build step lines will be logged to the screen and logfiles. Default: False.
source_path – Path to the root of the source files. Defaults to the current working directory. If running under bazel this will be set to the $BUILD_WORKSPACE_DIRECTORY environment variable. Otherwise $PW_PROJECT_ROOT will be used.

pw_build.project_builder.run_builds(project_builder: ProjectBuilder, workers: int = 1) → int#

Upstream `pw build` Command-Line Interface Usage#

This is the command line interface provided by the pw build command (main.py) in upstream Pigweed.

pw_build.project_builder_presubmit_runner

usage: pw build [-h] [-C directory [target ...]]
                [--build-system-command directory command]
                [--run-command RUN_COMMAND] [-S SOURCE_PATH]
                [--default-build-system {ninja,bazel}] [-j JOBS] [-k]
                [--parallel] [--parallel-workers PARALLEL_WORKERS]
                [--logfile LOGFILE] [--separate-logfiles] [--debug-logging]
                [--banners | --no-banners] [--colors | --no-colors]
                [--patterns PATTERNS]
                [--ignore-patterns IGNORE_PATTERNS_STRING]
                [--exclude-list EXCLUDE_LIST [EXCLUDE_LIST ...]]
                [--restart | --no-restart] [--serve-docs]
                [--serve-docs-port SERVE_DOCS_PORT]
                [--serve-docs-path SERVE_DOCS_PATH] [-f] [-r RECIPE] [-s STEP]
                [-l] [--all] [--progress-bars | --no-progress-bars]
                [--log-build-steps | --no-log-build-steps] [-w] [-b BASE]
                [--tab-complete-option [TAB_COMPLETE_OPTION]]
                [--tab-complete-format {bash,fish,zsh}]
                [--tab-complete-recipe [TAB_COMPLETE_RECIPE]]
                [--tab-complete-presubmit-step [TAB_COMPLETE_PRESUBMIT_STEP]]
                [target ...]

Named Arguments#

-r, --recipe

Run a build recipe. Include an asterix to match more than one name. For example: –recipe ‘gn_*’

Default: []

-s, --step

Run presubmit step. Include an asterix to match more than one step name. For example: –step ‘*_format’

Default: []

-l, --list

List all known build recipes and presubmit steps.

Default: False

--all

Run all known build recipes.

Default: False

--progress-bars, --no-progress-bars

Show progress bars in the terminal.

Default: True

--log-build-steps, --no-log-build-steps

Show ninja build step log lines in output.

-w, --watch

Use pw_watch to monitor changes.

Default: False

-b, --base

Git revision to diff for changed files. This is used for presubmit steps.

--tab-complete-option

Print tab completions for the supplied option text.

--tab-complete-format

Possible choices: bash, fish, zsh

Output format for tab completion results.

Default: “bash”

--tab-complete-recipe

Print tab completions for the supplied recipe name.

--tab-complete-presubmit-step

Print tab completions for the supplied presubmit name.

Build Directory and Command Options#

-C, --build-directory

Specify a build directory and optionally targets to build. pw watch -C out target1 target2 is equivalent to ‘ninja -C out taret1 target2’. The ‘out’ directory will be used if no others are provided.

Default: []

target

Default build targets. For example if the build directory is ‘out’ then, ‘ninja -C out taret1 target2’ will be run. To specify one or more directories, use the -C / --build-directory option.

Default: []

--build-system-command

Build system command for . Default: ninja

Default: []

--run-command

Additional commands to run. These are run before any -C arguments and may be repeated. For example: –run-command ‘bazel build //pw_cli/…’ –run-command ‘bazel test //pw_cli/…’ -C out python.lint python.test

Default: []

-S, --source-path

Path to the root of the source files. Defaults to the current working directory. If running under bazel this will be set to the $BUILD_WORKSPACE_DIRECTORY environment variable.

--default-build-system

Possible choices: ninja, bazel

Build system to use when no build directories or build targets are specified on the command line.

Build Execution Options#

-j, --jobs

Specify the number of cores to use for each build system. This is passed to ninja, bazel and make as “-j”

-k, --keep-going

Keep building past the first failure. This is equivalent to running “ninja -k 0” or “bazel build -k”.

Default: False

--parallel

Run all builds in parallel.

Default: False

--parallel-workers

How many builds may run at the same time when –parallel is enabled. Default: 0 meaning run all in parallel.

Default: 0

Log File Options#

--logfile

Global build output log file.

--separate-logfiles

Create separate log files per build directory.

Default: False

--debug-logging

Enable Python build execution tool debug logging.

Default: False

Display Output Options#

--banners, --no-banners

Show pass/fail banners.

Default: True

--colors, --no-colors

Force color output from ninja.

Default: True

Watch Options#

--patterns

Comma delimited list of globs to watch to trigger recompile.

Default: “.bazel,.bzl,*.bloaty,*.c,*.cc,*.css,*.cpp,*.cmake,CMakeLists.txt,*.dts,*.dtsi,*.emb,*.gn,*.gni,*.go,*.h,*.hpp,*.html,*.java,*.js,*.ld,*.md,*.options,*.proto,*.py,*.rs,*.rst,*.s,*.S,*.toml,*.ts”

--ignore-patterns

Comma delimited list of globs to ignore events from.

--exclude-list

Directories to ignore during pw watch. This option may be repeated. Directories are passed as separate arguments.

Default: []

--restart, --no-restart

Whether to restart ongoing builds if files change.

Default: True

--serve-docs

Start a webserver for docs on localhost. The port for this webserver can be set with the –serve-docs-port option. Defaults to http://127.0.0.1:8000

Default: False

--serve-docs-port

Set the port for the docs webserver. Default: 8000.

Default: 8000

--serve-docs-path

Set the path for the docs to serve. Default: docs/gen/docs in the build directory.

Default: docs/gen/docs

-f, --fullscreen

Use a fullscreen interface.

Default: False

Project Builder#

Get Started#

Example Build Scripts#

Reference#

BuildCommand#

BuildCommand Run Filters#

BuildRecipe#

ProjectBuilder#

Upstream pw build Command-Line Interface Usage#

Named Arguments#

Build Directory and Command Options#

Build Execution Options#

Log File Options#

Display Output Options#

Watch Options#

Upstream `pw build` Command-Line Interface Usage#