Workflows Launcher#

pw_build: Integrations for Bazel, GN, and CMake

The workflows launcher is a command-line tool for running tools, builds, and other workflows defined in a workflows.json file. It provides a centralized and discoverable way to execute common development tasks.

Note

b/425973227: This tool is currently under active development, and will change significantly as the experience is refined.

Getting Started#

bazelisk run //pw_build/py:workflows_launcher

python -m pw_build.workflows.launcher

The launcher will automatically find and load a workflows.json file in the current directory or any parent directory.

Commands#

The Workflows launcher provides a set of built-in commands, and will later support commands generated from the workflows.json file.

Built-in Commands#

describe#

The describe command prints a human-readable description of a Tool, Build, BuildConfig, or TaskGroup from the workflows.json file. This is useful for inspecting the configuration of a specific workflow, and is particularly helpful for debugging programmatically-generated configurations.

Example:

bazelisk run //pw_build/py:workflows_launcher -- describe format

python -m pw_build.workflows.launcher describe format

This will print the configuration for the format tool:

▒█████▄   █▓  ▄███▒  ▒█    ▒█ ░▓████▒ ░▓████▒ ▒▓████▄
 ▒█░  █░ ░█▒ ██▒ ▀█▒ ▒█░ █ ▒█  ▒█   ▀  ▒█   ▀  ▒█  ▀█▌
 ▒█▄▄▄█░ ░█▒ █▓░ ▄▄░ ▒█░ █ ▒█  ▒███    ▒███    ░█   █▌
 ▒█▀     ░█░ ▓█   █▓ ░█░ █ ▒█  ▒█   ▄  ▒█   ▄  ░█  ▄█▌
 ▒█      ░█░ ░▓███▀   ▒█▓▀▓█░ ░▓████▒ ░▓████▒ ▒▓████▀

name: "format"
description: "Find and fix code formatting issues"
use_config: "bazel_default"
target: "@pigweed//:format"
analyzer_friendly_args: "--check"

build#

The build command launches the build for the requested build name.

Example:

bazelisk run //pw_build/py:workflows_launcher -- build all_host

python -m pw_build.workflows.launcher build all_host

Note

b/425973227: This doesn’t offer any configurability or additional argument handling today.

Generated Commands#

The workflows launcher generates commands from the tools and groups defined in the workflows.json configuration file.

Tools#

For each tool in the tools list, a command is created with the same name. Running this command will execute the tool’s specified command.

From the example workflows.json, the following command is created:

bazelisk run //pw_build/py:workflows_launcher -- format

python -m pw_build.workflows.launcher format

This will launch a bazel run invocation of Pigweed’s code formatter tool.

Groups#

For each group in the groups list, a command is created with the same name. Running this command will execute all the workflows in the group in sequence.

From the example workflows.json, the following command is created:

bazelisk run //pw_build/py:workflows_launcher -- presubmit

python -m pw_build.workflows.launcher presubmit

This will launch a series of builds followed by code health check tooling as enumerated by the group named presubmit.

Configuration#

The Workflows launcher is configured via ProtoJSON. The schema lives in workflows.proto:

Configuration schema
package pw.build.proto;

import "google/protobuf/any.proto";

// A distinct build configuration.
//
// Build configurations are expected to be hermetic, meaning that if the
// same build configuration is reused across multiple builds it should be safe
// to reuse the same build directory.
message BuildConfig {
  // Descriptive name for this build configuration.
  // This is used as an ID and must be unique across a workflow configuration.
  string name = 1;

  // A help string that describes this configuration.
  string description = 2;

  // The name of the driver that should be used for this build configuration.
  string build_type = 3;

  // Arguments that should be passed to underlying build system.
  repeated string args = 4;

  // Environment variables unique to this build configuration.
  map<string, string> env = 5;
}

// A discrete build.
message Build {
  // A human-readable name for this build.
  // This is the name that users will write out to trigger this build.
  string name = 1;

  // A help string that describes this build.
  string description = 2;

  // The configuration that should be used when building the targets specified
  // by this build.
  oneof config {
    // The ID/name of a shared BuildConfig.
    string use_config = 3;
    // A unique one-off config.
    BuildConfig build_config = 4;
  }

  // Target patterns that should be built/tested by this configuration.
  repeated string targets = 5;
}

// A runnable tool.
message Tool {
  // The name that is used to launch this tool.
  string name = 1;

  // A descriptive help string for this tool.
  string description = 2;

  // The configuration that this tool will be built under.
  oneof config {
    // The ID/name of a shared BuildConfig.
    string use_config = 3;
    // A unique one-off config.
    BuildConfig build_config = 4;
  }

  // Target to build and launch.
  string target = 5;

  enum Type {
    // A tool that may produce output, mutate the world, require realtime user
    // interaction, or have side-effects that affect hermeticity of subsequent
    // actions.
    // By default, all tools are assumed to be of this kind.
    GENERAL = 0;

    // A runnable tool that exists *purely* to surface information.
    // e.g. linters, code style checks, etc.
    // Theses tools may NOT:
    //   - Modify files.
    //   - Upload/download arbitrary files.
    //   - Modify local system state.
    //   - Interact with attached devices.
    ANALYZER = 1;
  }

  // The type of tool.
  Type type = 6;

  // Arguments to inject that make a Tool safe to run as an ANALYZER.
  // These are not injected when this tool is normally launched, only when
  // the tool is launched in a way that requires behavior that doesn't affect
  // hermeticity.
  repeated string analyzer_friendly_args = 7;
}

// A series of builds and analyzers that are launchable as a group to
// accelerate workflows.
//
// The builds and analyzers listed in this workflow *may* be ran in order,
// but no guarantee is provided. Builds and analyzers listed here should
// behave predictably and deterministically when run out-of-order.
// Having order-based dependencies in a workflow is considered an antipattern.
message TaskGroup {
  // The name used to launch this workflow.
  string name = 1;

  // A descriptive help string for this tool.
  string description = 2;

  // The name of builds that should be performed as part of this workflow.
  repeated string builds = 3;

  // The name of analyzers that should be run as part of this workflow.
  //
  // Note: Only ANALYZER tools, or tools that have analyzer_friendly_args
  // may be listed here.
  repeated string analyzers = 4;
}

// These options are used when loading ProtoJSON files since JSON doesn't
// have any inherent mechanism for sharing/reusing snippets from other files.
message ConfigLoaderOptions {
  message ImportedConfigFragment {
    // The file to import entries from.
    // This path is the result of strip/import_prefix of any dependent
    // configurations.
    string config_path = 1;

    // The `name`s that should be imported into the current config.
    // Note that all transitive requirements are also imported. i.e.
    // importing a LaunchableWorkflow will import all referenced builds,
    // analyzers, build configs, etc.
    repeated string imported_ids = 2;
  }

  // Config fragments that should be imported from another externally-provided
  // file.
  repeated ImportedConfigFragment imports = 1;
}

// The launcher config.
message WorkflowSuite {
  ConfigLoaderOptions loader_options = 1;
  repeated BuildConfig configs = 2;
  repeated Build builds = 3;
  repeated Tool tools = 4;
  repeated TaskGroup groups = 5;
}

The workflows launcher searches for a workflows.json file from the current working directory, traversing through parent directories as needed. When a workflows.json file is found, it is loaded as a WorkflowSuite Protobuf message. This configuration is then used to drive the rest of the launcher’s behavior.

Example workflows.json
{
  "configs": [
    {
      "name": "bazel_default",
      "description": "Default bazel build configuration (host)",
      "build_type": "bazel",
      "args": [],
      "env": {}
    }
  ],
  "builds": [
    {
      "name": "all_host",
      "use_config": "bazel_default",
      "targets": [
        "//..."
      ]
    },
    {
      "name": "all_host_cpp20",
      "build_config": {
        "name": "bazel_default_cpp20",
        "description": "Host C++20 build",
        "build_type": "bazel",
        "args": [
          "--//pw_toolchain/cc:cxx_standard=20"
        ]
      },
      "targets": [
        "//..."
      ]
    },
    {
      "name": "docs",
      "use_config": "bazel_default",
      "targets": [
        "//docs"
      ]
    },
    {
      "name": "build_rp2040_tests",
      "build_config":{
        "name": "pico_rp2040",
        "description": "Default Pi Pico build (rp2040)",
        "build_type": "bazel",
        "args": [
          "--config=rp2040",
          "--build_tests_only"
        ]
      },
      "targets": [
        "//..."
      ]
    },
    {
      "name": "build_rp2350_tests",
      "build_config": {
        "name": "pico_rp2350",
        "description": "Default Pi Pico build (rp2350)",
        "build_type": "bazel",
        "args": [
          "--config=rp2350",
          "--build_tests_only"
        ]
      },
      "targets": [
        "//..."
      ]
    }
  ],
  "tools": [
    {
      "name": "format",
      "description": "Find and fix code formatting issues",
      "use_config": "bazel_default",
      "analyzer_friendly_args": ["--check"],
      "target": "@pigweed//:format"
    },
    {
      "name": "owners_lint",
      "description": "Identify OWNERS file issues",
      "use_config": "bazel_default",
      "target": "@pigweed//pw_presubmit/py:owners_lint",
      "type": "ANALYZER"
    }
  ],
  "groups": [
    {
      "name": "presubmit",
      "builds": [
        "all_host",
        "docs",
        "rp2040",
        "rp2350"
      ],
      "analyzers": [
        "format",
        "owners_lint"
      ]
    }
  ]
}

Build programming#

The definition of how build configurations become a series of actions is defined by Workflows build drivers.

On this page