The presubmit module provides Python tools for running presubmit checks and
checking and fixing code format. It also includes the presubmit check script for
the Pigweed repository,
Presubmit checks are essential tools, but they take work to set up, and
projects don’t always get around to it. The
pw_presubmit module provides
tools for setting up high quality presubmit checks for any project. We use this
framework to run Pigweed’s presubmit on our workstations and in our automated
pw_presubmit module also includes
pw format, a tool that provides a
unified interface for automatically formatting code in a variety of languages.
pw format, you can format Bazel, C, C++, Python, GN, and Go code
according to configurations defined by your project.
pw format leverages
existing tools like
clang-format, and it’s simple to add support for new
languages. (Note: Bazel formatting requires
buildifier to be present on your
system. If it’s not Bazel formatting passes without checking.)
pw_presubmit package includes presubmit checks that can be used with any
project. These checks include:
Check code format of several languages including C, C++, and Python
Initialize a Python environment
Run all Python tests
Ensure source files are included in the GN and Bazel builds
Build and run all tests with GN
Build and run all tests with Bazel
Ensure all header files contain
Creating a presubmit check for your project¶
Creating a presubmit check for a project using
pw_presubmit is simple, but
requires some customization. Projects must define their own presubmit check
Python script that uses the
A project’s presubmit script can be registered as a
pw_cli plugin, so that it can be run as
Setting up the command-line interface¶
pw_presubmit.cli module sets up the command-line interface for a
presubmit script. This defines a standard set of arguments for invoking
presubmit checks. Its use is optional, but recommended.
Argument parsing code for presubmit checks.
- pw_presubmit.cli.add_arguments(parser: argparse.ArgumentParser, programs: Optional[pw_presubmit.presubmit.Programs] = None, default: str = '') None ¶
Adds common presubmit check options to an argument parser.
- pw_presubmit.cli.run(program: Sequence[Callable], output_directory: Optional[pathlib.Path], package_root: pathlib.Path, clear: bool, root: Optional[pathlib.Path] = None, repositories: Collection[pathlib.Path] = (), only_list_steps=False, **other_args) int ¶
Processes arguments from add_arguments and runs the presubmit.
program – from the –program option
output_directory – from –output-directory option
package_root – from –package-root option
clear – from the –clear option
root – base path from which to run presubmit checks; defaults to the root of the current directory’s repository
repositories – roots of Git repositories on which to run presubmit checks; defaults to the root of the current directory’s repository
only_list_steps – list the steps that would be executed, one per line, instead of executing them
**other_args – remaining arguments defined by by add_arguments
exit code for sys.exit; 0 if succesful, 1 if an error occurred
A presubmit check is defined as a function or other callable. The function must
accept one argument: a
PresubmitContext, which provides the paths on which
to run. Presubmit checks communicate failure by raising an exception.
Presubmit checks may use the
filter_paths decorator to automatically filter
the paths list for file types they care about.
Either of these functions could be used as presubmit checks:
@pw_presubmit.filter_paths(endswith='.py') def file_contains_ni(ctx: PresubmitContext): for path in ctx.paths: with open(path) as file: contents = file.read() if 'ni' not in contents and 'nee' not in contents: raise PresumitFailure('Files must say "ni"!', path=path) def run_the_build(_): subprocess.run(['make', 'release'], check=True)
Presubmit checks functions are grouped into “programs” – a named series of
checks. Projects may find it helpful to have programs for different purposes,
such as a quick program for local use and a full program for automated use. The
example script uses
pw_presubmit.Programs to define
Existing Presubmit Checks¶
A small number of presubmit checks are made available through
Formatting checks for a variety of languages are available from
pw_presubmit.format_code. These include C/C++, Java, Go, Python, GN, and
others. All of these checks can be included by adding
pw_presubmit.format_code.presubmit_checks() to a presubmit program. These
all use language-specific formatters like clang-format or yapf.
These will suggest fixes using
pw format --fix.
pragma_once check that confirms the first non-comment line of
C/C++ headers is
#pragma once. This is enabled by adding
pw_presubmit.pragma_once to a presubmit program.
There are two checks in the
gn_python_check. They assume there’s a top-level
python GN target.
gn_pylint runs Pylint and Mypy checks and
gn_python_check runs Pylint,
Mypy, and all Python tests.
The inclusive language check looks for words that are typical of non-inclusive code, like using “master” and “slave” in place of “primary” and “secondary” or “sanity check” in place of “consistency check”.
These checks can be disabled for individual lines with “inclusive-language: ignore” on the line in question or the line above it, or for entire blocks by using “inclusive-language: disable” before the block and “inclusive-language: enable” after the block.
The pw_presubmit package provides tools for running presubmit checks.
- exception pw_presubmit.PresubmitFailure(description: str = '', path=None)¶
Optional exception to use for presubmit failures.
- __init__(description: str = '', path=None)¶
- class pw_presubmit.Programs(**programs: Sequence)¶
A mapping of presubmit check programs.
Use is optional. Helpful when managing multiple presubmit check programs.
- __init__(**programs: Sequence)¶
Initializes a name: program mapping from the provided keyword args.
A program is a sequence of presubmit check functions. The sequence may contain nested sequences, which are flattened.
- pw_presubmit.call(*args, **kwargs) None ¶
Optional subprocess wrapper that causes a PresubmitFailure on errors.
- pw_presubmit.filter_paths(endswith: Iterable[str] = '', exclude: Iterable[Union[Pattern[str], str]] = (), always_run: bool = False) Callable[[Callable], pw_presubmit.presubmit.Check] ¶
Decorator for filtering the paths list for a presubmit check function.
Path filters only apply when the function is used as a presubmit check. Filters are ignored when the functions are called directly. This makes it possible to reuse functions wrapped in @filter_paths in other presubmit checks, potentially with different path filtering rules.
endswith – str or iterable of path endings to include
exclude – regular expressions of paths to exclude
a wrapped version of the presubmit function
A simple example presubmit check script follows. This can be copied-and-pasted to serve as a starting point for a project’s presubmit check script.
pigweed_presubmit.py for a more complex presubmit check script example.
"""Example presubmit check script.""" import argparse import logging import os from pathlib import Path import re import sys from typing import List, Pattern try: import pw_cli.log except ImportError: print('ERROR: Activate the environment before running presubmits!', file=sys.stderr) sys.exit(2) import pw_presubmit from pw_presubmit import ( build, cli, cpp_checks, environment, format_code, git_repo, inclusive_language, filter_paths, python_checks, PresubmitContext, ) from pw_presubmit.install_hook import install_hook # Set up variables for key project paths. PROJECT_ROOT = Path(os.environ['MY_PROJECT_ROOT']) PIGWEED_ROOT = PROJECT_ROOT / 'pigweed' # Rerun the build if files with these extensions change. _BUILD_EXTENSIONS = frozenset( ['.rst', '.gn', '.gni', *format_code.C_FORMAT.extensions]) # # Presubmit checks # def release_build(ctx: PresubmitContext): build.gn_gen(PROJECT_ROOT, ctx.output_dir, build_type='release') build.ninja(ctx.output_dir) def host_tests(ctx: PresubmitContext): build.gn_gen(PROJECT_ROOT, ctx.output_dir, run_host_tests='true') build.ninja(ctx.output_dir) # Avoid running some checks on certain paths. PATH_EXCLUSIONS = ( re.compile(r'^external/'), re.compile(r'^vendor/'), ) # Use the upstream pragma_once check, but apply a different set of path # filters with @filter_paths. @filter_paths(endswith='.h', exclude=PATH_EXCLUSIONS) def pragma_once(ctx: PresubmitContext): cpp_checks.pragma_once(ctx) # # Presubmit check programs # QUICK = ( # List some presubmit checks to run pragma_once, host_tests, # Use the upstream formatting checks, with custom path filters applied. format_code.presubmit_checks(exclude=PATH_EXCLUSIONS), # Include the upstream inclusive language check. inclusive_language.inclusive_language, # Include just the lint-related Python checks. python_checks.gn_pylint.with_filter(exclude=PATH_EXCLUSIONS), ) FULL = ( QUICK, # Add all checks from the 'quick' program release_build, # Use the upstream Python checks, with custom path filters applied. # Checks listed multiple times are only run once. python_checks.gn_python_check.with_filter(exclude=PATH_EXCLUSIONS), ) PROGRAMS = pw_presubmit.Programs(quick=QUICK, full=FULL) def run(install: bool, **presubmit_args) -> int: """Process the --install argument then invoke pw_presubmit.""" # Install the presubmit Git pre-push hook, if requested. if install: install_hook(__file__, 'pre-push', ['--base', 'HEAD~'], git_repo.root()) return 0 return cli.run(root=PROJECT_ROOT, **presubmit_args) def main() -> int: """Run the presubmit checks for this repository.""" parser = argparse.ArgumentParser(description=__doc__) cli.add_arguments(parser, PROGRAMS, 'quick') # Define an option for installing a Git pre-push hook for this script. parser.add_argument( '--install', action='store_true', help='Install the presubmit as a Git pre-push hook and exit.') return run(**vars(parser.parse_args())) if __name__ == '__main__': pw_cli.log.install(logging.INFO) sys.exit(main())
Code formatting tools¶
pw_presubmit.format_code module formats supported source files using
external code format tools. The file
format_code.py can be invoked directly
from the command line or from