pw_ide CLI#

pw_ide: Code editor and IDE support for Pigweed projects

The pw_ide command-line interface (CLI) provides an editor-independent mechanism for managing code intelligence for Pigweed projects.

Note

Currently, the CLI only supports bootstrap projects, For Bazel projects, we recommend using the Visual Studio Code integration.

Setting up C++ code intelligence#

clangd is a language server that provides C/C++ code intelligence features to any editor that supports the language server protocol (LSP). It uses a compilation database, a JSON file containing the compile commands for the project. Projects that have multiple targets and/or use multiple toolchains need separate compilation databases for each target toolchain. pw_ide provides tools for managing those databases.

Assuming you have one or more compilation databases that have been generated by your build system, start with:

pw ide sync

This command will:

Find every compilation database in your build directory
Analyze each database
- If a database is internally consistent (i.e., it only contains valid compile commands for a single target), it will use that database as-is for the target toolchain that database pertains to. This is the typical case for CMake builds.
- Otherwise, if a database contains commands for multiple target toolchains and/or contains invalid compile commands, the database will be processed, yielding one new compilation database for each target toolchain. Those databases will be used instead of the original.
Link each target to its respective compilation database

Now, you can list the available target toolchains with:

pw ide cpp --list

Then set the target toolchain that clangd should use with:

pw ide cpp --set <selected target name>

clangd will now work as designed since it is configured to use a compilation database that is consistent to just a single target toolchain.

clangd must be run with arguments that provide the Pigweed environment paths to the correct toolchains and sysroots. One way to do this is to launch your editor from the terminal in an activated environment (e.g. running vim from the terminal), in which case nothing special needs to be done as long as your toolchains are in the Pigweed environment or $PATH. But if you launch your editor outside of the activated environment (e.g. launching Visual Studio Code from your GUI shell’s launcher), you can generate the command that invokes clangd with the right arguments with:

pw ide cpp --clangd-command

Setting up Python code intelligence#

Any Python language server should work well with Pigweed projects as long as it’s configured to use the Pigweed virtual environment. You can output the path to the virtual environment on your system with:

pw ide python --venv

Setting up docs code intelligence#

The esbonio language server will provide code intelligence for reStructuredText and Sphinx. It works well with Pigweed projects as long as it is pointed to Pigweed’s Python virtual environment. For Visual Studio Code, simply install the esbonio extension, which will be recommended to you after setting up pw_ide. Once it’s installed, a prompt will ask if you want to automatically install esbonio in your Pigweed Python environment. After that, give esbonio some time to index, then you’re done!

Command-line interface reference#

CLI tools for pw_ide.

usage: pw ide [-h] [-o {stdout,log}] {sync,setup,cpp,python,vscode} ...

Named Arguments#

-o, --output

Possible choices: stdout, log

where program output should go

Default: “pretty”

Sub-commands#

sync#

Setup or sync your Pigweed project IDE features.

pw ide sync [-h]

This will automatically set up your development environment with all the features that Pigweed IDE supports, with sensible defaults.

At minimum, this command will create the .pw_ide working directory and create settings files for all supported editors. Projects can define additional setup steps in .pw_ide.yaml.

When new IDE features are introduced in the future (either by Pigweed or your downstream project), you can re-run this command to set up the new features. It will not overwrite or break any of your existing configuration.

All commands will be run from the PW_PROJECT_ROOT directory

setup#

Deprecated! Please use pw ide sync.

pw ide setup [-h]

cpp#

Configure C/C++ code intelligence support.

pw ide cpp [-h] [-l] [-g] [-s TARGET] [--set-default] [-p]
           [--process-files PROCESS_FILES [PROCESS_FILES ...]]
           [--clangd-command] [--clangd-command-for SYSTEM]

Named Arguments#

-l, --list

list the target toolchains available for C/C++ language analysis

Default: False

-g, --get

print the current target toolchain used for C/C++ language analysis

Default: False

-s, --set

set the target toolchain to use for C/C++ language server analysis

--set-default

set the C/C++ analysis target toolchain to the default defined in pw_ide settings

Default: False

-p, --process

process compilation databases found in directories defined in the settings file

Default: False

--process-files, -P

process the compilation database(s) at the provided path(s)

--clangd-command

print the command for your system that runs clangd in the activated Pigweed environment

Default: False

--clangd-command-for

print the command for the specified system that runs clangd in the activated Pigweed environment

Code intelligence can be provided by clangd or other language servers that use the clangd compilation database format, defined at: https://clang.llvm.org/docs/JSONCompilationDatabase.html

Pigweed projects define their build configuration(s) via a build system, usually GN, Bazel, or CMake. Based on build configurations, the build system generates commands to compile each translation unit in the project. clangd uses those commands to parse the build graph and provide rich code intelligence.

Pigweed projects often target multiple devices & architectures, and use multiple compiler toolchains. As a result, there may be more than one way to compile each translation unit. Your build system ensures that it only invokes a single compiler command for each translation unit which is consistent with the toolchain and target appropriate to that build, which we refer to as a “target toolchain”.

We need to do the same thing with the compilation database that clangd uses. We handle this by:

Processing the compilation database produced the build system into multiple internally-consistent compilation databases, one for each target toolchain.
Providing commands to select which target toolchain you want to use for code analysis.

Refer to the Pigweed documentation or your build system’s documentation to learn how to produce a clangd compilation database. Once you have one, run this command to process it:

pw ide cpp --process-files {path to compile_commands.json}

When using GN and CMake, pw_ide can search your build directory (or other directories specified in settings) for compilation databases with:

pw ide cpp --process

You can now examine the target toolchains that are available to you:

pw ide cpp --list

… and select the target toolchain you want to use:

pw ide cpp --set host_clang

As long as your editor or language server plugin is properly configured, you will now get code intelligence features relevant to that particular target toolchain.

You can see what target toolchain is selected by running:

pw ide cpp

Whenever you switch to a target toolchain you haven’t used before, clangd will index the build, which may take several minutes. This process is not blocking, so you can take advantage of code analysis immediately even while the indexing is in progress. These indexes are cached, so you can switch between targets without re-indexing each time.

If your build configuration changes significantly (e.g. you add a new file to the project), you will need to re-process the compilation database for that change to be recognized and manifested in the target toolchain. Your target toolchain selection will not change, and your index will only need to be incrementally updated.

You can generate the clangd command your editor needs to run with:

pw ide cpp --clangd-command

If your editor uses JSON for configuration, you can export the same command in that format:

pw ide cpp --clangd-command-for json

python#

Configure Python code intelligence support.

pw ide python [-h] [--venv] [--install-editable MODULE]

Named Arguments#

--venv

print the path to the Pigweed Python virtual environment

Default: False

--install-editable

install a Pigweed Python module in editable mode

You can generate the path to the Python virtual environment interpreter that your editor/language server should use with:

pw ide python --venv

When working on Pigweed’s Python modules, it can be convenient to install them in editable mode to instantly realize code changes. You can do this by running:

pw ide python --install-editable pw_{module name}

Just note that running bootstrap or building will override this.

vscode#

Configure support for Visual Studio Code.

pw ide vscode [-h] [--include SETTINGS_TYPE [SETTINGS_TYPE ...]]
              [--exclude SETTINGS_TYPE [SETTINGS_TYPE ...]]
              [--build-extension]

Named Arguments#

--include

update only these settings types

--exclude

do not update these settings types

--build-extension

build the extension from source

Default: False

This will replace your current Visual Studio Code (VSC) settings for this project (in .vscode/settings.json, etc.) with the following sets of settings, in order:

The Pigweed default settings
Your project’s settings, if any (in .vscode/pw_project_settings.json)
Your personal settings, if any (in .vscode/pw_user_settings.json)

In other words, settings files lower on the list can override settings defined by those higher on the list. Settings defined in the sources above are not active in VSC until they are merged and output to the current settings file by running:

pw ide vscode

Refer to the Visual Studio Code documentation for more information about these settings: https://code.visualstudio.com/docs/getstarted/settings

This command also manages VSC tasks (.vscode/tasks.json) and extensions (.vscode/extensions.json). You can explicitly control which of these settings types (“settings”, “tasks”, and “extensions”) is modified by this command by using the --include or --exclude options.

Your current VSC settings will never change unless you run pw ide commands. Since the current VSC settings are an artifact built from the three settings files described above, you should avoid manually editing that file; it will be replaced the next time you run pw ide vscode. A backup of your previous settings file will be made, and you can diff it against the new file to see what changed.

These commands will never modify your VSC user settings, which are stored outside of the project repository and apply globally to all VSC instances.

The settings files are system-specific and shouldn’t be checked into the repository, except for the project settings (those with pw_project_), which can be used to define consistent settings for everyone working on the project.

Note that support for VSC can be disabled at the project level or the user level by adding the following to .pw_ide.yaml or .pw_ide.user.yaml respectively:

editors:
  vscode: false

Likewise, it can be enabled by setting that value to true. It is enabled by default.

Configuration#

pw_ide has a built-in default configuration, so you don’t need to create a configuration file to get started. You can create a configuration file if you need to override those defaults.

A project configuration can be defined in .pw_ide.yaml in the project root. This configuration will be checked into source control and apply to all developers of the project. Each user can also create a .pw_ide.user.yaml file that overrides both the default and project settings, is not checked into source control, and applies only to that checkout of the project. All of these files have the same schema, in which these options can be configured:

property PigweedIdeSettings.working_dir: Path#

Path to the pw_ide working directory.

The working directory holds C++ compilation databases and caches, and other supporting files. This should not be a directory that’s regularly deleted or manipulated by other processes (e.g. the GN out directory) nor should it be committed to source control.

Default: .pw_ide

property PigweedIdeSettings.compdb_gen_cmd: str | None#

The command that should be run to generate a compilation database.

Defining this allows pw_ide to automatically generate a compilation database if it suspects one has not been generated yet before a sync.

Default: None

property PigweedIdeSettings.compdb_search_paths: list[tuple[Path, str]]#

Paths to directories to search for compilation databases.

If you’re using a build system to generate compilation databases, this may simply be your build output directory. However, you can add additional directories to accommodate compilation databases from other sources.

Entries can be just directories, in which case the default target inference pattern will be used. Or entries can be tuples of a directory and a target inference pattern. See the documentation for target_inference for more information.

Finally, the directories can be concrete paths, or they can be globs that expand to multiple paths.

Note that relative directory paths will be resolved relative to the workspace root.

Default: ['out']

property PigweedIdeSettings.target_inference: str#

A glob-like string for extracting a target name from an output path.

Build systems and projects have varying ways of organizing their build directory structure. For a given compilation unit, we need to know how to extract the build’s target name from the build artifact path. A simple example:

clang++ hello.cc -o host/obj/hello.cc.o

The top-level directory host is the target name we want. The same compilation unit might be used with another build target:

gcc-arm-none-eabi hello.cc -o arm_dev_board/obj/hello.cc.o

In this case, this compile command is associated with the arm_dev_board target.

When importing and processing a compilation database, we assume by default that for each compile command, the corresponding target name is the name of the top level directory within the build directory root that contains the build artifact. This is the default behavior for most build systems. However, if your project is structured differently, you can provide a glob-like string that indicates how to extract the target name from build artifact path.

A * indicates any directory, and ? indicates the directory that has the name of the target. The path is resolved from the build directory root, and anything deeper than the target directory is ignored. For example, a glob indicating that the directory two levels down from the build directory root has the target name would be expressed with */*/?.

Note that the build artifact path is relative to the compilation database search path that found the file. For example, for a compilation database search path of {project dir}/out, for the purposes of target inference, the build artifact path is relative to the {project dir}/out directory. Target inference patterns can be defined for each compilation database search path. See the documentation for compdb_search_paths for more information.

Default: ?

property PigweedIdeSettings.targets_include: list[str]#

The list of targets that should be enabled for code analysis.

In this case, “target” is analogous to a GN target, i.e., a particular build configuration. By default, all available targets are enabled. By adding targets to this list, you can constrain the targets that are enabled for code analysis to a subset of those that are available, which may be useful if your project has many similar targets that are redundant from a code analysis perspective.

Target names need to match the name of the directory that holds the build system artifacts for the target. For example, GN outputs build artifacts for the pw_strict_host_clang_debug target in a directory with that name in its output directory. So that becomes the canonical name for the target.

Default: []

property PigweedIdeSettings.targets_exclude: list[str]#

The list of targets that should not be enabled for code analysis.

In this case, “target” is analogous to a GN target, i.e., a particular build configuration. By default, all available targets are enabled. By adding targets to this list, you can disable/hide targets that should not be available for code analysis.

Default: []

property PigweedIdeSettings.default_target: str | None#

The default target to use when calling --set-default.

This target will be selected when pw ide cpp --set-default is called. You can define an explicit default target here. If that command is invoked without a default target definition, pw_ide will try to infer the best choice of default target. Currently, it selects the target with the broadest compilation unit coverage.

Default: None

property PigweedIdeSettings.cascade_targets: bool#

Mix compile commands for multiple targets to maximize code coverage.

By default (with this set to False), the compilation database for each target is consistent in the sense that it only contains compile commands for one build target, so the code intelligence that database provides is related to a single, known compilation artifact. However, this means that code intelligence may not be provided for every source file in a project, because some source files may be relevant to targets other than the one you have currently set. Those source files won’t have compile commands for the current target, and no code intelligence will appear in your editor.

If this is set to True, compilation databases will still be separated by target, but compile commands for all other targets will be appended to the list of compile commands for that target. This will maximize code coverage, ensuring that you have code intelligence for every file that is built for any target, at the cost of consistency—the code intelligence for some files may show information that is incorrect or irrelevant to the currently selected build target.

The currently set target’s compile commands will take priority at the top of the combined file, then all other targets’ commands will come after in order of the number of commands they have (i.e. in the order of their code coverage). This relies on the fact that clangd parses the compilation database from the top down, using the first compile command it encounters for each compilation unit.

Default: False

property PigweedIdeSettings.sync: list[str]#

A sequence of commands to automate IDE features setup.

pw ide sync should do everything necessary to get the project from a fresh checkout to a working default IDE experience. This defines the list of commands that makes that happen, which will be executed sequentially in subprocesses. These commands should be idempotent, so that the user can run them at any time to update their IDE features configuration without the risk of putting those features in a bad or unexpected state.

Default: ['pw --no-banner ide cpp --process']

property PigweedIdeSettings.clangd_additional_query_drivers: list[str] | None#

Additional query driver paths that clangd should use.

By default, pw_ide supplies driver paths for the toolchains included in Pigweed. If you are using toolchains that are not supplied by Pigweed, you should include path globs to your toolchains here. These paths will be given higher priority than the Pigweed toolchain paths.

If you want to omit the query drivers argument altogether, set this to null.

Default: []

property PigweedIdeSettings.workspace_root: Path#

The root directory of the IDE workspace.

In most cases, the IDE workspace directory is also your Pigweed project root; that’s the default, and if that’s the case, this you can omit this configuration.

If your project has a structure where the directory you open in your IDE is not the Pigweed project root directory, you can specify the workspace root directory here to ensure that IDE support files are put in the right place.

For example, given this directory structure:

my_project
|
├- third_party
├- docs
├- src
|  ├- pigweed.json
|  └- ... all other project source files
|
└- pigweed
    └ ... upstream Pigweed source, e.g. a submodule

In this case `my_project/src/` is the Pigweed project root, but `my_project/` is the workspace root directory you will open in your IDE.

A relative path will be resolved relative to the directory in which the .pw_ide.yaml config file is located.

When to provide additional configuration to support your use cases#

The default configuration for clangd in pw_ide should work without additional configuration as long as you’re using only toolchains provided by Pigweed and your native host toolchain. If you’re using other toolchains, keep reading.

clangd needs two pieces of information to use a toolchain:

A path to the compiler, which will be taken from the compile command.
The same compiler to be reflected in the query driver argument provided when running clangd.

When using pw_ide with external toolchains, you only need to add a path to the compiler to clangd_additional_query_drivers in your project’s pw_ide.yaml file. When processing a compilation database, pw_ide will use the query driver globs to find your compiler and configure clangd to use it.

Using compiler wrappers#

If you’re using ccache or any other wrapper command that is configured using ccache’s’ KEY=VALUE pattern, it will work out of the box.