Workflows Launcher#

pw_build: Integrations for Bazel, GN, and CMake

The workflows launcher is a command-line tool for running tools, builds, and other workflows defined in a workflows.json file. It provides a centralized and discoverable way to execute common development tasks.

Note

b/425973227: This tool is currently under active development, and will change significantly as the experience is refined.

Get Started#

New project setup#

Add a native_binary entry point to your root BUILD.bazel file:

load("@bazel_skylib//rules:native_binary.bzl", "native_binary")

native_binary(
    name = "pw",
    src = "@pigweed//pw_build/py:workflows_launcher",
)

Add a workflows.json file to the root of your project.
```
$ echo {} >> workflows.json
```
Copy or symlink the helper pw entry point to the root of your project.

Now you’re ready to start exploring the Workflows tool.

General usage (in Pigweed)#

To see available commands, launch the pw entry point at the root of the Pigweed repository.

$ ./pw

The launcher will automatically find and load a workflows.json file in the current directory or any parent directory.

Note

The Workflows tool does not yet offer any GN or CMake build integration. Please use pw_env_setup and pw_presubmit.

Commands#

The Workflows launcher provides a set of built-in commands, and will later support commands generated from the workflows.json file.

Built-in Commands#

`describe`#

The describe command prints a human-readable description of a Tool, Build, BuildConfig, or TaskGroup from the workflows.json file. This is useful for inspecting the configuration of a specific workflow, and is particularly helpful for debugging programmatically-generated configurations.

Example:

$ ./pw describe format

This will print the configuration for the format tool:

▒█████▄   █▓  ▄███▒  ▒█    ▒█ ░▓████▒ ░▓████▒ ▒▓████▄
 ▒█░  █░ ░█▒ ██▒ ▀█▒ ▒█░ █ ▒█  ▒█   ▀  ▒█   ▀  ▒█  ▀█▌
 ▒█▄▄▄█░ ░█▒ █▓░ ▄▄░ ▒█░ █ ▒█  ▒███    ▒███    ░█   █▌
 ▒█▀     ░█░ ▓█   █▓ ░█░ █ ▒█  ▒█   ▄  ▒█   ▄  ░█  ▄█▌
 ▒█      ░█░ ░▓███▀   ▒█▓▀▓█░ ░▓████▒ ░▓████▒ ▒▓████▀

name: "format"
description: "Find and fix code formatting issues"
use_config: "bazel_default"
target: "@pigweed//:format"
analyzer_friendly_args: "--check"

`build`#

The build command launches the build for the requested build name.

Example:

$ ./pw build all_host

Note

b/425973227: This doesn’t offer any configurability or additional argument handling today.

Generated Commands#

The workflows launcher generates commands from the tools and groups defined in the workflows.json configuration file.

Tools#

For each tool in the tools list, a command is created with the same name. Running this command will execute the tool’s specified command.

From the example workflows.json, the following command is created:

$ ./pw format

This will launch a bazel run invocation of Pigweed’s code formatter tool.

Groups#

For each group in the groups list, a command is created with the same name. Running this command will execute all the workflows in the group in sequence.

From the example workflows.json, the following command is created:

$ ./pw presubmit

This will launch a series of builds followed by code health check tooling as enumerated by the group named presubmit.

Configuration#

The Workflows launcher is configured via ProtoJSON. The schema lives in workflows.proto:

Configuration schema

package pw.build.proto;

import "google/protobuf/any.proto";

// Describes a group of outputs of a build action.
message OutputGroupSpec {
  // The name by which this output group can be referenced.
  string name = 1;

  // File glob patterns that are included in this output group.
  // These follow starlark/python file globbing semantics.
  repeated string glob_patterns = 2;

  // The prefix that should be stripped from these paths when they are
  // packaged or exported.
  //
  // If any files in the glob patterns aren't beneath this prefix, an error
  // will be raised.
  string strip_prefix = 3;

  // If a build succeeds and no files are captured by this output group then
  // an error will be raised. Setting this field to True suppresses this error.
  bool allow_empty = 4;
}

message OutputGroup {
  // Files that match the given globs.
  repeated string matching_files = 1;

  // The prefix that should be stripped from these paths when they are
  // packaged or exported.
  string strip_prefix = 2;
}

message BuildArtifacts {
  // The root of the output directory. All OutputGroupSpec paths are
  // relative to this root.
  string output_root = 1;

  // Output groups related to the requested build.
  repeated OutputGroup output_groups = 2;
}

// A distinct build configuration.
//
// Build configurations are expected to be hermetic, meaning that if the
// same build configuration is reused across multiple builds it should be safe
// to reuse the same build directory.
message BuildConfig {
  // Descriptive name for this build configuration.
  // This is used as an ID and must be unique across a workflow configuration.
  string name = 1;

  // A help string that describes this configuration.
  string description = 2;

  // The name of the driver that should be used for this build configuration.
  string build_type = 3;

  // Arguments that should be passed to underlying build system.
  repeated string args = 4;

  // Environment variables unique to this build configuration.
  map<string, string> env = 5;

  // Driver-specific options.
  // This is forwarded to the driver for the respective build_type, and is used
  // to configure options that are more complex then simply passing additional
  // flags.
  google.protobuf.Any driver_options = 6;
}

// A discrete build.
message Build {
  // A human-readable name for this build.
  // This is the name that users will write out to trigger this build.
  string name = 1;

  // A help string that describes this build.
  string description = 2;

  // The configuration that should be used when building the targets specified
  // by this build.
  oneof config {
    // The ID/name of a shared BuildConfig.
    string use_config = 3;
    // A unique one-off config.
    BuildConfig build_config = 4;
  }

  // Target patterns that should be built/tested by this configuration.
  repeated string targets = 5;

  // A shortcut subcommand that launches this build. For example:
  //
  //   # Given rerun_shortcut=build_foo, the following subcommand will be
  //   # available:
  //   $ ./pw build_foo
  //
  // If this is not set, the assumed way to run this build is `build [name]`.
  string rerun_shortcut = 6;

  // The ID/name of shared OutputGroupSpecs.
  repeated string use_output = 7;

  // Unique one-off specs.
  repeated OutputGroupSpec output_spec = 8;
}

// A runnable tool.
message Tool {
  // The name that is used to launch this tool.
  string name = 1;

  // A descriptive help string for this tool.
  string description = 2;

  // The configuration that this tool will be built under.
  oneof config {
    // The ID/name of a shared BuildConfig.
    string use_config = 3;
    // A unique one-off config.
    BuildConfig build_config = 4;
  }

  // Target to build and launch.
  string target = 5;

  enum Type {
    // A tool that may produce output, mutate the world, require realtime user
    // interaction, or have side-effects that affect hermeticity of subsequent
    // actions.
    // By default, all tools are assumed to be of this kind.
    GENERAL = 0;

    // A runnable tool that exists *purely* to surface information.
    // e.g. linters, code style checks, etc.
    // Theses tools may NOT:
    //   - Modify files.
    //   - Upload/download arbitrary files.
    //   - Modify local system state.
    //   - Interact with attached devices.
    ANALYZER = 1;
  }

  // The type of tool.
  Type type = 6;

  // Arguments to inject that make a Tool safe to run as an ANALYZER.
  // These are not injected when this tool is normally launched, only when
  // the tool is launched in a way that requires behavior that doesn't affect
  // hermeticity.
  repeated string analyzer_friendly_args = 7;

  // A shortcut subcommand that launches this tool. For example:
  //
  //   # Given rerun_shortcut=check_formatting, the following subcommand will be
  //   # available:
  //   $ ./pw check_formatting
  //
  // Note that this always launches the analyzer version of this tool, and is
  // unsupported for other tool types.
  // If this is not set, the assumed way to run this analyzer is `check [name]`.
  string rerun_shortcut = 8;

  // The ID/name of shared OutputGroupSpecs.
  repeated string use_output = 9;

  // Unique one-off specs.
  repeated OutputGroupSpec output_spec = 10;
}

// A series of builds and analyzers that are launchable as a group to
// accelerate workflows.
//
// The builds and analyzers listed in this workflow *may* be ran in order,
// but no guarantee is provided. Builds and analyzers listed here should
// behave predictably and deterministically when run out-of-order.
// Having order-based dependencies in a workflow is considered an antipattern.
message TaskGroup {
  // The name used to launch this workflow.
  string name = 1;

  // A descriptive help string for this tool.
  string description = 2;

  // The name of builds that should be performed as part of this workflow.
  repeated string builds = 3;

  // The name of analyzers that should be run as part of this workflow.
  //
  // Note: Only ANALYZER tools, or tools that have analyzer_friendly_args
  // may be listed here.
  repeated string analyzers = 4;
}

// These options are used when loading ProtoJSON files since JSON doesn't
// have any inherent mechanism for sharing/reusing snippets from other files.
message ConfigLoaderOptions {
  message ImportedConfigFragment {
    // The file to import entries from.
    // This path is the result of strip/import_prefix of any dependent
    // configurations.
    string config_path = 1;

    // The `name`s that should be imported into the current config.
    // Note that all transitive requirements are also imported. i.e.
    // importing a LaunchableWorkflow will import all referenced builds,
    // analyzers, build configs, etc.
    repeated string imported_ids = 2;
  }

  // Config fragments that should be imported from another externally-provided
  // file.
  repeated ImportedConfigFragment imports = 1;
}

// The launcher config.
message WorkflowSuite {
  ConfigLoaderOptions loader_options = 1;
  repeated BuildConfig configs = 2;
  repeated Build builds = 3;
  repeated Tool tools = 4;
  repeated TaskGroup groups = 5;
  repeated OutputGroupSpec output_specs = 6;
}

The workflows launcher searches for a workflows.json file from the current working directory, traversing through parent directories as needed. When a workflows.json file is found, it is loaded as a WorkflowSuite Protobuf message. This configuration is then used to drive the rest of the launcher’s behavior.

Build programming#

The definition of how build configurations become a series of actions is defined by Workflows build drivers.

Workflows Launcher#

Get Started#

New project setup#

General usage (in Pigweed)#

Commands#

Built-in Commands#

describe#

build#

Generated Commands#

Tools#

Groups#

Configuration#

Build programming#

`describe`#

`build`#