Pigweed’s GN Python Build#

generated_requirements.txt#

To ensure the requirements of in-tree pw_python_package targets are installed pw_python_venv introduces the source_packages argument. This is a list of in-tree pw_python_package GN targets expected to be used within the venv. When the venv is created each pw_python_package’s setup.cfg file is read to pull the install_requires section for all third party dependencies. The full list of all in-tree packages and any in-tree transitive dependencies is then written to the out directory in a single generated_requirements.txt.

Take the //pw_build/py/gn_tests:downstream_tools_build_venv example below, its source package is a single pw_python_distribution package which bundles the pw_env_setup and pw_console pw_python_package``s. Those two packages each depend on a few other ``pw_python_package targets. The output generated_requirements.txt below merges all these package deps and adds -c lines for constraint files.

pw_python_distribution("downstream_project_tools") {
  packages = [
    "$dir_pw_env_setup/py",
    "$dir_pw_console/py",
  ]
  generate_setup_cfg = {
    name = "downstream_project_tools"
    version = "0.0.1"
    include_default_pyproject_file = true
  }
}

pw_python_venv("downstream_tools_build_venv") {
  path = "$root_build_dir/python-venv-downstream-tools-test"
  requirements = []
  constraints =
      [ "$dir_pw_env_setup/py/pw_env_setup/virtualenv_setup/constraint.list" ]
  source_packages = [ ":downstream_project_tools" ]
}

# Auto-generated requirements.txt from the following packages:
#
# //pw_arduino_build/py:py
# //pw_build/py/gn_tests:downstream_project_tools
# //pw_build/py:py
# //pw_cli/py:py
# //pw_console/py:py
# //pw_env_setup/py:py
# //pw_log_tokenized/py:py
# //pw_package/py:py
# //pw_presubmit/py:py
# //pw_stm32cube_build/py:py

# Constraint files:
-c ../../../../../../../pw_env_setup/py/pw_env_setup/virtualenv_setup/constraint.list

black>=23.1.0
build>=0.8.0
coloredlogs
coverage
ipython
jinja2
mypy>=0.971
parameterized
pip-tools>=6.12.3
prompt-toolkit>=3.0.26
psutil
ptpython>=3.0.20
pygments
pylint>=2.9.3
pyperclip
pyserial>=3.5,<4.0
pyyaml
setuptools
six
toml
types-pygments
types-pyserial>=3.5,<4.0
types-pyyaml
types-setuptools
types-six
websockets
wheel
yapf>=0.31.0

compiled_requirements.txt#

The above generated_requirements.txt file is then fed into the pip-compile command from the pip-tools package to fully expand and pin each package with hashes. The resulting compiled_requirements.txt can then be used as the single Python requirements file for replicating this pw_python_venv elsewhere. Each pw_python_venv will get this single file containing the exact versions of each required Python package.

Below is a snippet of the compiled_requirements.txt for this pw_python_venv target: //pw_build/py/gn_tests:downstream_tools_build_venv

#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --allow-unsafe --generate-hashes
#      --output-file=python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/compiled_requirements.txt
#      --resolver=backtracking
#      python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/generated_requirements.txt
#
appdirs==1.4.4 \
    --hash=sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41 \
    --hash=sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128
    # via
    #   -c python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/../../../../../../../pw_env_setup/py/pw_env_setup/virtualenv_setup/constraint.list
    #   ptpython
astroid==2.14.2 \
    --hash=sha256:0e0e3709d64fbffd3037e4ff403580550f14471fd3eaae9fa11cc9a5c7901153 \
    --hash=sha256:a3cf9f02c53dd259144a7e8f3ccd75d67c9a8c716ef183e0c1f291bc5d7bb3cf
    # via
    #   -c python/gen/pw_build/py/gn_tests/downstream_tools_build_venv/../../../../../../../pw_env_setup/py/pw_env_setup/virtualenv_setup/constraint.list
    #   pylint
...

The presence of hashes in the above example can be controlled via the pip_generate_hashes arg to the pw_python_venv template.

Downloading Packages#

The pw_python_venv target adds an optional sub target that will download all Python packages from remote servers into a local directory. The remote server is typically pypi.org.

Taking the //pw_build/py/gn_tests:downstream_tools_build_venv target as an example again let’s build a local cache. To run the download target append .vendor_wheels to the end of the pw_python_venv target name. In this example it would be //pw_build/py/gn_tests:downstream_tools_build_venv.vendor_wheels

To build that one gn target with ninja, pass the output name from gn as a target name for ninja:

gn gen out
ninja -C out \
  $(gn ls out --as=output \
    '//pw_build/py/gn_tests:downstream_tools_build_venv.vendor_wheels')

This creates a wheels folder with all downloaded packages and a pip_download_log.txt with verbose logs from running pip download.

out/python/gen/pw_build/py/gn_tests/downstream_tools_build_venv.vendor_wheels/
├── pip_download_log.txt
└── wheels
    ├── appdirs-1.4.4-py2.py3-none-any.whl
    ├── astroid-2.14.2-py3-none-any.whl
    ├── backcall-0.2.0-py2.py3-none-any.whl
    ├── black-23.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    ├ ...
    ├── websockets-10.4-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    ├── wheel-0.40.0-py3-none-any.whl
    ├── wrapt-1.14.1.tar.gz
    └── yapf-0.31.0-py2.py3-none-any.whl

Note the above output has both Python wheel .whl and source distribution .tar.gz files. The .whl may contain Python packages with precompiled C extensions. This is denoted by this part of the filename: cp311-cp311-manylinux_2_17_x86_64.whl. These binary packages are selected by the pip download command based on the host machine python version, OS, and CPU architecture.

.vendor_wheels can attempt to download binary packages for multiple platforms all at once by setting a GN arg:

pw_build_PYTHON_PIP_DOWNLOAD_ALL_PLATFORMS = true

This will invoke pip download for each combination of platform, architecture and Python version. This can take a significant amount of time to complete. The current set of combinations is shown below:

    # These platform args are derived from the cffi pypi package:
    #   https://pypi.org/project/cffi/#files
    # See also these pages on Python wheel filename format:
    #   https://peps.python.org/pep-0491/#file-name-convention
    # and Platform compatibility tags:
    #   https://packaging.python.org/en/latest/specifications/
    #      platform-compatibility-tags/
    platform_args = [
        '--platform=any',
        '--platform=macosx_10_9_universal2',
        '--platform=macosx_10_9_x86_64',
        '--platform=macosx_11_0_arm64',
        '--platform=manylinux2010_x86_64',
        '--platform=manylinux2014_aarch64',
        '--platform=manylinux2014_x86_64',
        '--platform=manylinux_2_17_aarch64',
        '--platform=manylinux_2_17_x86_64',
        '--platform=musllinux_1_1_x86_64',
        '--platform=win_amd64',
        # Note: These 32bit platforms are omitted
        # '--platform=manylinux2010_i686',
        # '--platform=manylinux2014_i686',
        # '--platform=manylinux_2_12_i686'
        # '--platform=musllinux_1_1_i686',
        # '--platform=win32',
    ]

    # Pigweed supports Python 3.8 and up.
    python_version_args = [
        [
            '--implementation=py3',
            '--abi=none',
        ],
        [
            '--implementation=cp',
            '--python-version=3.8',
            '--abi=cp38',
        ],
        [
            '--implementation=cp',
            '--python-version=3.9',
            '--abi=cp39',
        ],
        [
            '--implementation=cp',
            '--python-version=3.10',
            '--abi=cp310',
        ],
        [
            '--implementation=cp',
            '--python-version=3.11',
            '--abi=cp311',
        ],
    ]

appnope; sys_platform == "darwin"
colorama; sys_platform == "win32"

Installing Offline#

Once the vendor wheel output is saved to a directory in your project you can use this as the default pip install location in two different ways.

# Adds --no-index forcing pip to not reach out to the internet (pypi.org) to
# download packages. Using this option requires setting
# pw_build_PYTHON_PIP_INSTALL_FIND_LINKS as well.
pw_build_PYTHON_PIP_INSTALL_OFFLINE = true

# List of paths to folders containing Python wheels (*.whl) or source tar
# files (*.tar.gz). Pip will check each of these directories when looking for
# potential install candidates.
pw_build_PYTHON_PIP_INSTALL_FIND_LINKS = [
  "//environment/cipd/packages/python_packages/universal",
  "//environment/cipd/packages/python_packages/linux/cp311",
]

# Optional: Adds '--no-cache-dir' forcing pip to ignore any previously cached
# Python packages. On most systems this is located in ~/.cache/pip/
pw_build_PYTHON_PIP_INSTALL_DISABLE_CACHE = false

Testing#

Tests for a Python package are listed in its pw_python_package target. Adding a new test is simple: write the test file and list it in its accompanying Python package. The build will run it when the test, the package, or one of its dependencies is updated.

Static analysis#

pw_python_package targets are preconfigured to run Pylint, Mypy and Ruff on their source and test files. Users may specify which pylintrc, mypy_ini and ruff_toml files to use on a per-package basis. The configuration files may also be provided in the directory structure; the tools will locate them using their standard means. Like tests, static analysis is only run when files or their dependencies change.

Packages may opt out of static analysis as necessary by setting static_analysis on the pw_python_package target.

The default set of analysis tools to run can be set globally via a GN arg pw_build_PYTHON_STATIC_ANALYSIS_TOOLS. By default this is set to include the below tools:

  # Default set of Python static alaysis tools to run for pw_python_package targets.
  pw_build_PYTHON_STATIC_ANALYSIS_TOOLS = [
    "pylint",
    "mypy",
  ]

In addition to user specified mypy_ini files some arguments are always passed to mypy by default. They can be seen in this excerpt of //pw_build/python.gni below:

    args = [
      "--pretty",
      "--show-error-codes",

      # Use a mypy cache dir for this target only to avoid cache conflicts in
      # parallel mypy invocations.
      "--cache-dir",
      rebase_path("$target_gen_dir/$target_name/.mypy_cache", root_build_dir),
    ]

    # Use this environment variable to force mypy to colorize output.
    # See https://github.com/python/mypy/issues/7771
    environment = [ "MYPY_FORCE_COLOR=1" ]

Building Python wheels#

Wheels are the standard format for distributing Python packages. The Pigweed Python build supports creating wheels for individual packages and groups of packages. Building the .wheel subtarget creates a .whl file for the package using the PyPA’s build tool.

The .wheel subtarget of any pw_python_package or pw_python_distribution records the location of the generated wheel with GN metadata. Wheels for a Python package and its transitive dependencies can be collected from the pw_python_package_wheels key. See Python Distributable Templates.

Protocol buffers#

The Pigweed GN build supports protocol buffers with the pw_proto_library target (see pw_protobuf_compiler). Python protobuf modules are generated as standalone Python packages by default. Protocol buffers may also be nested within existing Python packages. In this case, the Python package in the source tree is incomplete; the final Python package, including protobufs, is generated in the output directory.

Generating setup.cfg#

The pw_python_package target in the BUILD.gn duplicates much of the information in the setup.cfg file. In many cases, it would be possible to generate a setup.cfg file rather than including it in the source tree. However, removing the setup.cfg would preclude using a direct, editable installation from the source tree.

Pigweed packages containing protobufs are generated in full or in part. These packages may use generated setup files, since they are always packaged or installed from the build output directory.

Dependencies#

Pigweed is organized into distinct modules. In Python, each module is a separate package, potentially with dependencies on other local or PyPI packages.

The basic Python packaging tools lack dependency tracking for local packages. For example, a package’s setup.cfg lists all of its dependencies, but pip is not aware of local packages until they are installed. Packages must be installed with their dependencies taken into account, in topological sorted order.

To work around this, one could set up a private PyPI server instance, but this is too cumbersome for daily development and incompatible with editable package installation.

Testing#

Tests are crucial to having a healthy, maintainable codebase. While they take some initial work to write, the time investment pays for itself many times over by contributing to the long-term resilience of a codebase. Despite their benefit, developers don’t always take the time to write tests. Any barriers to writing and running tests result in fewer tests and consequently more fragile, bug-prone codebases.

There are lots of great Python libraries for testing, such as unittest and pytest. These tools make it easy to write and execute individual Python tests, but are not well suited for managing suites of interdependent tests in a large project. Writing a test with these utilities does not automatically run them or keep running them as the codebase changes.

Static analysis#

Various static analysis tools exist for Python. Two widely used, powerful tools are Pylint and Mypy. Using these tools improves code quality, as they catch bugs, encourage good design practices, and enforce a consistent coding style. As with testing, barriers to running static analysis tools cause many developers to skip them. Some developers may not even be aware of these tools.

Deploying static analysis tools to a codebase like Pigweed has some challenges. Mypy and Pylint are simple to run, but they are extremely slow. Ideally, these tools would be run constantly during development, but only on files that change. These tools do not have built-in support for incremental runs or dependency tracking.

Another challenge is configuration. Mypy and Pylint support using configuration files to select which checks to run and how to apply them. Both tools only support using a single configuration file for an entire run, which poses a challenge to modular middleware systems where different parts of a project may require different configurations.

Protocol buffers#

Protocol buffers are an efficient system for serializing structured data. They are widely used by Google and other companies.

The protobuf compiler protoc generates Python modules from .proto files. protoc strictly generates protobuf modules according to their directory structure. This works well in a monorepo, but poses a challenge to a middleware system like Pigweed. Generating protobufs by path also makes integrating protobufs with existing packages awkward.