CI & tryjobs (gh run)#

pw_ghish: GitHub CLI-like interface for Gerrit code reviews and CI checks

./gh run and ./gh pr checks map GitHub Actions CLI workflows to LUCI Buildbucket and Android Busytown / TreeHugger (ci.android.com). You can check build statuses, inspect step execution trees, view step failure logs, and rerun failed tryjob or presubmit builders from the terminal.

Quick reference#

# Check status and duration of LUCI tryjob or Busytown builders on the active CL:
$ ./gh pr checks

# Watch checks until completion, exiting with failure logs if a check fails:
$ ./gh pr checks --watch --fail-fast

# View a summary of all tryjobs on the active CL:
$ ./gh run view

# Inspect the step execution tree and failure summaries for a builder:
$ ./gh run view -j pigweed-lintformat

# Print failure summaries and step log snippets for all failed checks:
$ ./gh run view --log-failed

# Rerun all failed builders on the active CL:
$ ./gh run rerun --failed

Overview: pr checks and gh run#

Matching the GitHub CLI, pw_ghish divides CI commands into two groups:

  1. Change-level check status (./gh pr checks): Displays a status table for a Gerrit CL, supports polling until completion (--watch), and returns an exit code suitable for shell pipelines and merge gates.

  2. Run and builder inspection (./gh run): Inspects individual LUCI or Busytown builders and step execution trees (run view -j <builder>), fetches failure logs (--log-failed and --log), and triggers builder retries (run rerun).

Both pr checks and run commands resolve the active Gerrit change from your current Git branch or HEAD commit when no target argument is supplied. You can also pass an explicit change number (e.g. 472267), patchset suffix (e.g. 472267/3), Gerrit URL/shortlink (see Targeting changes), a Buildbucket build ID (e.g. 8671182706745774001), or a Busytown build ID and target (e.g. P99164207/target_name or P99164207 --target target_name).

Checking and watching CL status: pr checks#

./gh pr checks queries the active CI provider (LUCI Buildbucket and/or Android Busytown) to display the status, duration, and build URLs of checks on a change:

# Check active change on current branch:
$ ./gh pr checks

# Check a specific change by number:
$ ./gh pr checks 413992
Checks for Change 413992 (Patchset 1)
  ✓  static-checks-pigweed           16s       https://ci.chromium.org/b/8680709829694997521

# Check a historical patchset:
$ ./gh pr checks 413992/1

By default, non-blocking experimental builders (cq_experimental) are omitted from the display. Pass --experimental (or -e) to include them. When ci.hide_tag_filters is configured in .ghish.toml or git config (for example, hide-in-gerrit:subbuild on orchestrator builds), matching child subbuilds are hidden by default and can be shown with --all:

$ ./gh pr checks --experimental
$ ./gh pr checks 413992 -e
$ ./gh pr checks --all

Gerrit gates, Busytown / TreeHugger, and dual-stack CI#

In addition to LUCI Buildbucket tryjobs, ./gh pr checks and ./gh run support Android Busytown / TreeHugger (ci.android.com) and Gerrit SubmitRequirements / automated verification labels (such as Presubmit-Verified, Android-Build-Verified, Lint-Verified, Tree-Approval, and Kokoro):

  • Android Busytown / TreeHugger (busytown): Automatically enabled on Android Gerrit hosts (*android-review.googlesource.com, ag/..., aosp/...), on changes with Presubmit-Verified / Presubmit-Ready labels or TreeHugger messages, or when ci.providers includes "busytown" in .ghish.toml. Queries Gerrit’s treetop~presubmittasks endpoint for structured per-target statuses and falls back to parsing TreeHugger Gerrit messages and Presubmit-Verified votes when treetop~ is unavailable.

  • Dual-stack repositories (ci.providers = ["buildbucket", "busytown"]): Queries both LUCI Buildbucket and Busytown in parallel, merges their check items into a single table and JSON response, and routes run view and run rerun to the appropriate provider per build.

  • Gerrit Submit Requirements and gates: Unsatisfied or non-Buildbucket Gerrit gates are rendered in a Gerrit Gates / Submit Requirements: section and included in the JSON output and exit code evaluation. On repositories that use only Gerrit labels (ci.providers = ["gerrit"]), ./gh pr checks evaluates Gerrit verification labels and submit requirements directly.

Watching checks: --watch and --fail-fast#

Use --watch (-w) to poll until all blocking checks finish. Combine with --fail-fast to exit as soon as any blocking check fails. By default, failing checks print their step failure summary and log snippet upon exit:

# Watch until all blocking checks complete (default 15s interval):
$ ./gh pr checks --watch

# Exit on first check failure and print step failure logs:
$ ./gh pr checks --watch --fail-fast

# Custom poll interval (e.g. every 30s):
$ ./gh pr checks --watch --fail-fast --interval 30s

# Open checks overview in web browser:
$ ./gh pr checks --web

# Output JSON:
$ ./gh pr checks --json

Process exit code contract#

./gh pr checks reports the state of the change through its process exit code, matching the GitHub CLI:

Exit code

Meaning

0

At least one blocking check ran and all of them passed.

8

Nothing has failed, but at least one blocking check is still running.

1

At least one blocking check failed or was canceled, no checks were reported at all, or the command itself errored.

The contract holds on every invocation, with or without --watch, --json, or --template, so it is safe to use as a merge gate:

$ ./gh pr checks --watch && ./gh pr merge --cq

A change with no reported checks exits 1, because CI that was never scheduled is not CI that passed. Experimental (non-blocking) builders never affect the exit code, even with --experimental, since the Commit Queue does not gate on them; --experimental only controls what is displayed.

Canceled checks on superseded patchsets#

Canceled checks exit 1 as well, since a canceled build did not pass, but they are reported as canceled rather than failed. Querying an older patchset usually produces them, because uploading a new patchset cancels the runs still in flight on the previous one:

$ ./gh pr checks 472267/43
...
Error: 29 of 75 checks were canceled on Change 472267 (Patchset 43): ... and 24 more.

Checks are usually canceled because a newer patchset superseded this one.
To query the current patchset instead:
  gh pr checks 472267

Reused builds from equivalent earlier patchsets#

When a new patchset is uploaded without changing code—such as a trivial rebase (TRIVIAL_REBASE or TRIVIAL_REBASE_WITH_MESSAGE_UPDATE), a commit-message edit (NO_CODE_CHANGE), NO_CHANGE, or MERGE_FIRST_PARENT_UPDATE—LUCI CV keeps tryjob results from the earlier patchset instead of rerunning every builder.

pr checks, pr view, pr status, and gh run walk backwards across these code-equivalent patchsets (stopping at the most recent REWORK) and include their builds:

  • Builds reused from an earlier patchset are annotated with (from patchset <N>) in pr checks, run list, run view, and failure reports, and carry a patchset field in --json output (alongside equivalentPatchsets on pr checks --json and run view --json).

  • When multiple equivalent patchsets have builds for the same builder, the newest build is shown—except when the newest build was CANCELED and an earlier equivalent build succeeded, in which case the passing build is kept.

Inspecting and rerunning builds: gh run#

Listing runs: run list#

Lists all checks and tryjobs for a change with duration, Buildbucket build ID, status, and URL (pass --all to include subbuilds hidden by ci.hide_tag_filters):

# List runs for active change:
$ ./gh run list

# List runs for a specific change:
$ ./gh run list 472267

# Include non-blocking experimental checks or hidden subbuilds:
$ ./gh run list --experimental
$ ./gh run list --all

# Output JSON:
$ ./gh run list --json

Viewing runs, steps, and logs: run view#

./gh run view supports three modes of inspection:

1. Change-level run overview#

Running ./gh run view renders a summary of all builders on the change:

$ ./gh run view
Showing 10 checks for Change 467905 (Patchset 22) • docs: Integrate navs
Gerrit CL: https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/467905

JOBS
✓ static-checks-pigweed                    (22s)
✓ pigweed-mac-arm-bazel-python             (4m53s)
✓ pigweed-linux-gn-platform                (10m48s)
✓ pigweed-linux-gn-host                    (8m55s)
✓ pigweed-linux-gn-compatibility-platform  (10m4s)
✓ pigweed-linux-gn-compatibility-main      (15m11s)
✓ pigweed-linux-bazel-python               (1m24s)
✓ docs-builder                             (2m36s)
✓ docs-builder-newpatchset                 (3m15s)
X pigweed-lintformat                       (5m32s)

To view step execution details for a specific check, run:
  gh run view -j <builder-name>
To inspect failure summaries and logs for failed checks, run:
  gh run view --log-failed

2. Builder step execution tree (-j <builder>)#

Inspect the execution steps and timing of a specific builder on a change.

pw_ghish collapses passing internal recipe substeps into top-level phases, highlights failing child steps, and extracts 1-line failure diagnostics (such as formatting diffs or compiler errors):

$ ./gh run view -j pigweed-lintformat
Steps for pigweed-lintformat (Build 8671182706745774001)
Status: FAILURE ✗ | URL: https://ci.chromium.org/b/8671182706745774001

  ✓  setup_build                   running recipe: "pw_presubmit" with Python 3.11.9
  ✓  checkout pigweed
  ✓  environment
  ✓  get steps from programs
  ✓  bazel_format
  ✓  css_format
  !  javascript_format
     └── ✗  failure summary: formatting diff in docs/common/header.js
  ✓  markdown_format
  !  python_format
     └── ✗  failure summary: formatting diff in 3 files: header.py, nav.py, ...
  ✓  restructuredtext_format

Pass -v or --verbose to inspect all unfiltered recipe steps. When a project schedules builders across multiple LUCI buckets, -j accepts either the short builder name (-j <builder>), bucket-qualified selector (-j <bucket>/<builder>), or full Buildbucket path (-j <project>/<bucket>/<builder>).

3. Step failure logs (--log-failed and --log)#

Fetch failure summaries and raw step log snippets in the terminal:

# Inspect all failed checks on current change:
$ ./gh run view --log-failed

# Inspect a specific failed builder:
$ ./gh run view -j pigweed-lintformat --log-failed

# Inspect a Busytown build directly by build ID and target (--target is ghish-only):
$ ./gh run view P99164207/firmware_host_tests --log-failed
$ ./gh run view P99164207 --target firmware_host_tests --log-failed

# Print the full step log instead of the tail snippet:
$ ./gh run view -j pigweed-lintformat --log

# Output JSON report:
$ ./gh run view --log-failed --json

# Open build in web browser:
$ ./gh run view -j pigweed-lintformat --web

On LUCI Buildbucket builds, run view --log-failed prioritizes curated failure summary (or failure_summary) streams before falling back to stdout and stderr (configurable via ci.preferred_logs in .ghish.toml). For orchestrator builds where compilation or test failures occur in a child subbuild, run view --log-failed also includes the top-level build SummaryMarkdown (including reproduction hints such as fx repro) while deduplicating truncated summary blocks against the full failure summary log snippet.

On Android Busytown / TreeHugger builds, run view --log-failed uses a two-tier retrieval strategy:

  1. Local workstation accelerators (Tier A): Invokes /google/data/ro/projects/android/fetch_artifact (to stream build.log) and /google/data/ro/projects/android/ants_cli (to query failed AnTS test invocations) when present on the machine.

  2. Direct HTTP fallback (Tier B): Queries the Android Build Internal v3 REST API (androidbuildinternal.googleapis.com) and ci.android.com using OAuth2 credentials, and extracts the failure snippet around FAILED:, : error:, : fatal error:, or ninja: build stopped markers.

Rerunning CI checks: run rerun#

Rerun specific or all failed builders on a change:

  • LUCI Buildbucket: Constructs and executes the bb add invocation using each build’s recorded Buildbucket <project>/<bucket>/<builder> tuple. Child subbuilds tagged with skip-retry-in-gerrit:subbuild (or matching ci.skip_retry_tag_filters in .ghish.toml) are automatically skipped during --failed reruns so only top-level parent builders are scheduled.

  • Android Busytown / TreeHugger: Triggers a presubmit rerun via Gerrit’s POST /changes/<id>/revisions/<ps>/treetop~runaction?service-id=presubmit&action-id=run endpoint, automatically falling back to voting Presubmit-Ready+1 if the treetop~ plugin is not installed on the Gerrit host.

  • Dual-stack: When a change has failed builds on both Buildbucket and Busytown, run rerun --failed triggers both providers.

# Rerun all failed builders on the current change:
$ ./gh run rerun --failed

# Rerun a single builder on the current change:
$ ./gh run rerun -j pigweed-lintformat

# Rerun a builder on a specific change:
$ ./gh run rerun 472267 -j pigweed-mac-arm-vscode

# Preview the rerun command without executing:
$ ./gh run rerun --failed --dry-run

Watching runs: run watch#

Poll checks until all blocking builds complete:

$ ./gh run watch
$ ./gh run watch 472267 --interval 30s

Comparison with GitHub CLI (gh run & gh pr checks)#

Gerrit projects run LUCI Buildbucket and/or Android Busytown rather than GitHub Actions. ./gh run and ./gh pr checks map these concepts as follows:

LUCI / Busytown vs. GitHub Actions concept map#

GitHub Actions Concept

LUCI / Busytown Equivalent in pw_ghish

Behavioral Notes

Workflow Run

Gerrit Patchset Tryjob Set (or Build ID)

Identified by Gerrit change/patchset (472267/3), a 64-bit Buildbucket ID (8671182706745774001), or a Busytown build ID / workplan ID (P99164207/target or L87654321).

Workflow Job (-j <job>)

LUCI Builder or Busytown Target

Targeted by builder/target name via -j <builder> or direct build ID.

Job Steps

LUCI Recipe Steps / Busytown Target

LUCI recipes emit nested substeps; run view -j collapses passing subtrees and surfaces 1-line failure summaries.

Action Logs

Buildbucket Step Logs / Busytown ``build.log`` & AnTS

--log-failed downloads the step failure summary and stdout/stderr (LUCI) or build.log and failed AnTS tests (Busytown).

Rerun Workflow (gh run rerun)

Buildbucket Retry (bb add) or TreeHugger Rerun (treetop~runaction / Presubmit-Ready+1)

Schedules tryjobs in Buildbucket or re-triggers TreeHugger presubmit on the patchset.

Key behavioral and flag differences#

  • Change-scoped by default: In upstream gh, running gh run list or gh run view with no arguments queries recent workflow runs across the entire repository. Because a Gerrit repository has many concurrent tryjobs across unrelated changes, pw_ghish scopes run list, run view, run watch, and run rerun by default to the active Gerrit CL on your current branch.

  • Fast failure logs on watch (pr checks --watch --fail-fast): Exits --watch as soon as any blocking builder fails and automatically fetches the failing step logs.

  • Experimental builders (-e, --experimental): LUCI distinguishes blocking Commit-Queue builders from cq_experimental builders. Experimental builders are hidden by default and do not cause a non-zero exit code in pr checks.

  • Equivalent patchset reuse: Tryjobs from earlier patchsets with no code changes (e.g. TRIVIAL_REBASE, NO_CODE_CHANGE) are automatically included and annotated with (from patchset <N>), matching LUCI CV’s build reuse rules.

  • --target <target> flag (ghish-only): On run view, --target specifies the Android Busytown target name when inspecting a direct build ID (e.g. gh run view P99164207 --target firmware_host_tests --log-failed). It is long-form only because upstream gh reserves -t for --template.

  • --json flag: On run list and run view, --json is a boolean flag that emits the structured run or failure report (unlike pr view --json <fields> and issue view --json <fields>, which take a comma-separated field list).

For the flag compatibility policy across all subcommands, see Compat policy.