CI/CQ Intro#

Essentials#

Submitting Changes#

The Gerrit Submit button is hidden, replaced by the Submit to CQ button. The Submit button is still accessible behind the ... button in the top right corner of the Gerrit UI, but in some cases requires elevated permissions.

Triggering presubmits#

Presubmits are not automatically run when a patch set is uploaded. Click CQ Dry Run to trigger them. (You can also use git push origin +HEAD:refs/for/main%l=Commit-Queue).

Presubmit validity duration#

If you don’t have recent passing results from a CQ Dry Run (within 24 hours) then Submit to CQ will run presubmits. After any newly run presubmits pass this will submit the change.

User interface#

If a presubmit fails you’ll get a Gerrit comment with a link to the failing build. The status of tryjobs (pending, running, failed, passed, etc.) is shown directly in the Gerrit UI (see Tryjobs).

Auto-Submit#

If you want your change to be automatically submitted when all requirements are met (Code-Review +2, OWNERS-approval, all comments resolved, etc.) set the Auto-Submit label to +1. If submission fails it will be retried a couple times with backoff and then the auto submit job will give up.

Further Details#

Applying changes in testing#

Changes are always rebased on the most recent commit when tested. If they fail to rebase the build fails. CQ Dry Run is the same as voting Commit-Queue +1 label and Submit to CQ is the same as voting Commit-Queue +2. If you vote both Code-Review +2 and Commit-Queue +2 on somebody’s change you are submitting it for them.

Post-Submit builders#

Jobs are run post-submission too and can be seen at https://ci.chromium.org/p/pigweed (for public projects) and https://ci.chromium.org/p/pigweed-internal (for internal projects). Builders can also be viewed from Pigweed’s builder visualization.

Non-`main` branches#

CQ is enabled for all branches. If you upload to an individual repository branch X and the manifest or superproject also has a branch X, that branch of the manifest will be used.

Rollers#

Just because a change has been submitted doesn’t mean it’s live in the project. Submodules and Android Repo Tool projects often need to be rolled before they’re in the most recent checkout of the project.

Presubmit result#

The Presubmit-Verified label is set at the completion of CQ runs. It does not block submission, but can be used by Copybara to see if CQ has passed. If it looks incorrect, do another CQ run and it will be updated.

Builder Visualization#

Pigweed’s builder visualization simplifies the process of browsing, viewing, and triggering builders. The source-of-truth for it is Google-internal, but there’s a public version without the Google-internal bits.

Builder Viz link for external contributors

Builder Viz link for Googlers

Tryjobs#

The colors of tryjobs in the Gerrit UI indicate the status of the tryjob: gray is pending/running, green is passed, red is failed or cancelled, and purple is an infra failure.

Some tryjobs are not yet stable and are run as “experimental”. These can be viewed with Show Experimental Tryjobs. Experimental tryjobs run with CQ but do not block it.

Individual tryjobs can be run additional times using the Choose Tryjobs dialog. This can also be used to run tryjobs that would not normally run on the change. Tryjobs ran this way can be used to satisfy CQ requirements, but don’t block CQ.

Prod vs Dev#

Most builders have “prod” and “dev” versions. The “prod” versions block changes in CQ and may cause emails to be sent out if they fail in CI. The “dev” builders test new VM images before they go to “prod”, so if a “dev” builder is failing when a “prod” builder is not failing, then the “dev” builder is failing because of an upcoming VM change, and teams should take time to get the “dev” builder to pass. For most projects, “dev” builders show up on the far right of console views in the LUCI UI.

Tryjobs CLI#

`bb` command#

The bb command is available in a bootstrapped Pigweed environment and the environments of many downstream projects. It is also available from Chromium’s depot tools <https://chromium.googlesource.com/chromium/tools/depot_tools.git>_.

Querying tryjobs#

In addition to viewing tryjobs in the Gerrit UI, you can use the bb command to query the tryjobs that ran on a change. The command to use is bb ls -cl $URL, but $URL has two non-obvious requirements:

It needs to be a “.googlesource.com” URL and not a Google-internal version of that URL.
It needs to include the patchset number.

$ bb ls -cl https://pigweed-review.googlesource.com/c/pigweed/sample_project/+/53684/1 | egrep -v '^(Tag|By):'
http://ci.chromium.org/b/8841234941714219488 SUCCESS   'pigweed/try/sample-project-xref-generator'
Created on 2021-07-19 at 16:45:32, waited 14.8s, started at 16:45:47, ran for 2m43s, ended at 16:48:30
CL: https://pigweed-review.googlesource.com/c/pigweed/sample_project/+/53684/1

http://ci.chromium.org/b/8841234941714219504 INFRA_FAILURE 'pigweed/try/sample-project-renode-test'
Summary: Infra Failure: Step('run pigweed/pw_unit_test/renode/test.sh') (retcode: None)
Created on 2021-07-19 at 16:45:32, waited 51.6s, started at 16:46:24, ran for 2m40s, ended at 16:49:04
CL: https://pigweed-review.googlesource.com/c/pigweed/sample_project/+/53684/1

To exclude non-experimental tryjobs, add -t cq_experimental:false to the command.

Manually launching tryjobs#

In most cases, individual tryjobs can be launched using Choose Additional Tryjobs. If any relevant tryjobs are not listed here please file a bug.

The bb command can also be used to launch tryjobs, which can useful for tracking down race conditions by launching many copies of a tryjob. Please be careful using this, especially during working hours in California.

URL="https://pigweed-review.googlesource.com/c/pigweed/sample_project/+/53684/1"
TRYJOB="pigweed/sample_project.try/sample-project-xref-generator"
for i in $(seq 1 25); do
  bb add -cl "$URL" "$TRYJOB"
done

Why didn’t lintformat catch this formatting change?#

Rolls of tools like clang can update the preferred format of clang-format. There are two possibilities for addressing this. First, the tool roll could be blocked until formatting passes. This could require coordinating several changes across many repositories. This is further complicated if the new formatting preferred by clang-format is not accepted by the previous version. Second, lintformat can be configured to only run on changed files. This means downstream project lintformat tryjobs would not be run on Pigweed changes, nor on rolls of Pigweed into these projects.

The second choice was selected. This means when tools roll lintformat jobs may start failing in CI, but they only fail in CQ on changes that touch files currently failing in CI. Teams should watch their build alert email list and proactively fix lintformat failures when they come.

Dependent changes#

Creating#

To pull in other changes when testing a change add a patches.json file to the root of the repository. An example is below.

[
  {
    "gerrit_name": "pigweed",
    "number": 123456
  },
  {
    "gerrit_name": "pigweed",
    "number": 654321
  }
]

Patches can be uni- or bidirectional and are transitive. The tryjob will parse this file, and then parse any patches.json files found in the referenced changes. If requirements are truly one-way, don’t list them as two-way. Only the Gerrit instance name (the part before “-review”) is permitted. The repository name is not included.

Note

patches.json cannot be used for changes to the same repo on the same Gerrit host (b/230610752). Just stack these changes instead.

Submitting#

Pigweed’s infrastructure does not support submitting multiple changes together. The best option is to find a way to have changes not depend on each other and submit them separately, or to have a one-way requirement instead of codependent changes, and submit the changes in dependency order, waiting for any necessary rolls before submitting the next change.

Pigweed-related Gerrit hosts are configured to reject submission of all changes containing patches.json files. If the dependency is one-way, then submit the change without dependencies, wait for it to roll (if necessary), remove patches.json from the dependent change, and vote Commit-Queue +2.

If the changes are truly codependent—both (or all) changes need each other—then follow the instructions below.

First, get both changes passing CQ with patches.json files.

Second, if one of the codependent changes is a submodule and another is the parent project, update the submodule change to no longer include the patches.json file. Then directly submit the change that lives in the child submodule, bypassing CQ. This will break the roller, but not the source tree, so others on your team are unaffected.

Note

For the main Pigweed repository, only core Pigweed team members can force submit, and they must first request a temporary ACL to do so. This process requires an associated bug, so have one on hand before reaching out with a force submission request.

Finally, once the change has merged into the child project, update the submodule pointer in the parent project:

Note

Some projects have limitations on submission outside of CQ. Reach out to a core Pigweed team member to bypass CQ for Pigweed itself.

Update your submodule pin to the submitted commit hash (in most cases git submodule update --remote path/to/submodule should be sufficient, but see the git submodule documentation for full details)
Add that change to the parent project change (git add path/to/submodule)
Remove the patches.json file from the change (git rm patches.json)
Commit and push to Gerrit
Click Submit to CQ

After this change is submitted the roller will start working again.

If all changes are to submodules, remove the patches.json files from both changes and directly submit, bypassing CQ. Then create a manual roll change that updates the submodules in question (git submodule update --remote submodule1 submodule2 should be sufficient), upload it, and Submit to CQ.

Details#

Sometimes codependent changes must be made in multiple repositories within an Android Repo Tool workspace or across multiple submodules. This can be done with the patches.json files. Given a situation where pigweed change would break the sample_project, the patches.json files must each refer to the other change.

Pigweed patches.json: [{"gerrit_name": "pigweed", "number": B}]
Sample Project patches.json: [{"gerrit_name": "pigweed", "number": A}]

When running tryjobs for change A, builders will attempt to patch in change B as well. For pure Pigweed tryjobs this fails but the build continues. For the tryjobs that are there to ensure Pigweed doesn’t break the Sample Project, both change A and change B will be applied to the checkout.

There is some validation of the format of the patches.json file, but there’s no error checking on the resolution of the required changes. The assumption is that changes that actually require other changes to pass CQ will fail if those changes aren’t patched into the workspace.

Requirements are transitive. If A requires B and B requires C then tryjobs for A will attempt to patch in A, B, and C. Requirements can also be one-way. If a change has been submitted it’s assumed to already be in the checkout and is not patched in, nor are any transitive requirements processed. Likewise, abandoned changes are ignored.

Banned codewords#

Sometimes the name of an internal Gerrit instance is a codeword we don’t allow on the Pigweed Gerrit instance. For example, you may wish to do the following.

Pigweed change A patches.json: [{"gerrit_name": "secret-project", "number": B}]
Secret-Project change B patches.json: [{"gerrit_name": "pigweed", "number": A}]

This will be rejected by the Pigweed Gerrit instance because using “secret-project” is banned on that Gerrit instance and you won’t be able to push. Instead, do the following, using the requires-helper repository on the Pigweed-Internal Gerrit instance.

Pigweed change A patches.json: [{"gerrit_name": "pigweed-internal", "number": C}]
Secret-Project change B patches.json: [{"gerrit_name": "pigweed", "number": A}]
Pigweed-Internal change C patches.json: [{"gerrit_name": "secret-project", "number": B}]

The pw requires command simplifies creation of the Pigweed-Internal change. In this case the command would be pw requires secret-project:B. Run this inside the Pigweed repository after committing change A and it will create change C and add [{"gerrit_name": "pigweed-internal", "number": C}] to change A. Multiple changes can be handled by passing multiple arguments to pw requires.

Public builders won’t have access to the Pigweed-Internal Gerrit instance so they won’t even be able to see the secret-project reference. Internal builders for other internal projects will see the secret-project reference but won’t be able to resolve it. Builders having access to secret-project will see all three changes and attempt to patch all three in. Pigweed-Internal change C is not included in any workspaces so it will never be patched in, but it transitively applies requirements to public changes.

CI/CQ Intro#

Essentials#

Submitting Changes#

Triggering presubmits#

Presubmit validity duration#

User interface#

Auto-Submit#

Further Details#

Applying changes in testing#

Post-Submit builders#

Non-main branches#

Rollers#

Presubmit result#

Builder Visualization#

Tryjobs#

Prod vs Dev#

Tryjobs CLI#

bb command#

Querying tryjobs#

Manually launching tryjobs#

Why didn’t lintformat catch this formatting change?#

Dependent changes#

Creating#

Submitting#

Details#

Banned codewords#

Non-`main` branches#

`bb` command#