Item 32: Set up a continuous integration (CI) system

A continuous integration (CI) system is a mechanism for automatically running tools over your codebase, which is triggered whenever there's a change to the codebase – or a proposed change to the codebase.

The recommendation to set up a continuous integration system is not at all Rust-specific, so this Item is a melange of general advice mixed with Rust-specific tool suggestions.

CI Steps

Starting with the specific, what kinds of steps should be included in your CI system? The obvious initial candidates are to:

  • Build the code.
  • Run the tests for the code.

In each case, a CI step should run cleanly, quickly, deterministically and with a zero false positive rate; more on this in the next section.

Throughout this book, various Items have suggested tools and techniques that can help improve your codebase; wherever possible, encode these as CI steps:

  • Item 30 described the various different styles of test; run all test types in CI.
    • Some test types are automatically included in cargo test: unit tests, integration tests and doc tests.
    • Other test types (e.g. example programs) may need to be explicitly triggered.
  • Item 29 waxed lyrical about the advantages of running Clippy over your code; run Clippy in CI.
  • Item 27 suggested documenting your public API; use the cargo doc tool to check that the documentation generates correctly and that any hyperlinks in it resolve correctly.
  • Item 21 included a discussion around declaring a minimum supported Rust version (MSRV) for your code. If you have this, check your MSRV in CI by including a step that tests with that specific Rust version.
  • Item 21 also described the -Z minimal-versions flag for Cargo's dependency resolution; include a CI step that checks semver lower bounds are accurate.
  • Item 26 described the use of features to conditionally include different chunks of code. If your crate has features, build every valid combination of features in CI (and realize that this may involve 2N different variants – hence the advice to avoid feature creep).
  • Item 25 mentioned tools such as cargo-udeps and cargo-deny that can help manage your dependency graph; running these as a CI step prevents regressions.
  • Item 31 discussed the Rust tool ecosystem; consider which of these tools are worth regularly running over your codebase. For example, running rustfmt / cargo fmt in CI will detect code that doesn't comply with your project's style guidelines.
  • Item 33 suggests that you consider making library code no_std compatible where possible. You can only be confident that your code genuinely is no_std compatible if you test no_std compatibility in CI; one option is to make use of the Rust's compiler's cross-compilation abilities, and build for an explicitly no_std target (e.g. thumbv6m-none-eabi).

You can also include CI steps that measure particular aspects of your code:

  • Generate code coverage statistics (e.g. with cargo-tarpaulin), to show what proportion of your codebase is exercised by your tests.
  • Run benchmarks (e.g. with cargo-bench, Item 30), to measure the performance of your code on key scenarios.

These suggestions are a bit more complicated to set up, because the output of the CI step is more useful when it's compared to previous results – in an ideal world, the CI system will detect when a code change is not fully tested, or has an adverse affect on performance, and this typically involves integration with some external tracking system.

Some other suggestions for CI steps that may or may not be relevant for your codebase include:

  • If your project is a library, recall (from Item 25) that any checked-in Cargo.lock file will be ignored by the users of your library. In theory, the semantic version constraints (Item 21) in Cargo.toml should mean that everything works correctly anyway; in practice, consider including:
    • A CI step that builds without any local Cargo.lock, to detect whether the current versions of dependencies still work correctly.
    • A CI step that uses the -Z minimal-versions option to detect whether the minimum matching versions of dependencies work correctly.
  • If your project includes any kind of machine-generated resources that are version-controlled (for example, code generated from protocol buffer messages by prost), then include a CI step that re-generates the resources and checks that there are no differences compared to the checked-in version.
  • If your codebase includes platform-specific (e.g. #[cfg(target_arch = "arm")]) code, run CI steps that confirm that the code builds and works on that platform.
  • If your project manipulates secret values such as access tokens or cryptographic keys, consider including a CI step that searches the codebase for secrets that have been inadvertantly checked in. This is particularly important if your project is public (in which case it may be worth moving the check from CI to a version control presubmit check).

Continuous integration checks don't always need to be integrated with Cargo and the Rust toolchains; sometimes a simple shell script can give more bang for the buck, particularly when a codebase has a local convention that's not universally followed. For example, a codebase might include a convention that any panic-inducing method invocation (Item 18) has a special marker comment or that every TODO: comment has an owner (a person or a tracking ID); a shell script is ideal for checking this.

Finally, consider examining the CI systems of public Rust projects to get ideas for additional CI steps that might be useful for your project. For example, Rust itself has an extensive CI system that includes dozens of steps; most of these steps are overkill for a smaller project, but some may be useful.

CI Principles

Moving from the specific to the general, there are some general principles that should guide the details of your continuous integration system.

The most fundamental principle is: don't waste the time of humans. If a CI system unnecessarily wastes people's time, they will start looking for ways to avoid it.

The most annoying waste of engineer's time is tests that are flaky: sometimes they pass, sometimes they fail, even when the setup and codebase is identical. Whenever possible, be ruthless with flaky tests: hunt them down and put in the time up-front to investigate and fix the cause of the flakiness – it will pay for itself in the long run.

Another common waste of engineering time is a continuous integration system that takes a long time to run, and which only runs after a request for a code review has been triggered. In this sitation, there's the potential to waste two people's time: both the author and also the code reviewer, who may spend time spotting and pointing out issues with the code that the CI bots could have flagged.

To help with this, try to make it easy to run the CI checks manually, independent from the automated system. This allows engineers to get into the habit of triggered them regularly, so that code reviewers never even see problems that the CI would have flagged.

This may also require splitting the checks up ("Testing, Fast and Slow") if there are time-consuming tests that rarely find problems, but which are there as a back-stop to prevent obscure scenarios breaking.

However, it's important that the CI system be integrated with whatever code review system is used for your project, so that a code review can clearly see a green set of checks and be confident that their code review can focus on the important meaning of the code, not on trivial details.

This need for a green build also means that there can be no exceptions to whatever checks your CI system has put in place. This is worthwhile even if you have to work around an occasional false positive from a tool; once your CI system has an accepted failure ("oh, everyone knows that test never passes") then it's vastly harder to spot new regressions.

Item 30 included the common advice of adding a test to reproduce a bug, before fixing the bug. The same principle applies to your CI system, applied to problems with your processes and codebase as a whole, rather than specific lines of code. If you discover a process-like problem, then think about how the continuous integration could have caught the problem, and add that extra step before fixing the problem (and seeing the build turn green).

Public CI Systems

If your codebase is open-source and visible to the public, there are a few extra things to think about with your continuous integration system.

First is the good news: there are lots of free, reliable options for building a continuous integration system for open-source code. At the time of writing, GitHub Actions are probably the best choice, but it's far from the only choice, and more systems appear all the time.

Secondly, for open-source code it's worth bearing in mind that your CI system can act as a guide for how to set up any prerequisites needed for the codebase. This isn't a concern for pure Rust crates, but if your codebase requires additional dependencies – databases, alternative toolchains for FFI code, configuration, etc. – then your CI scripts will be an existence proof of how to get all of that working on a fresh system. Encoding these setup steps in re-usable scripts allows both the humans and the bots to get a working system in a straightforward way.

Finally, there's bad news for publicly visible crates: the possibility of abuse and attacks. This can range from attempts to perform cryptocurrency mining in your CI system, to theft of codebase access tokens, supply chain attacks and worse. To mitigate these risks, consider:

  • Restricting access so that continuous integration scripts only run automatically for known collaborators, and have to be triggered manually for new contributors.
  • Pinning the versions of any external scripts to particular versions, or (better yet) specific known hashes.
  • Closely monitoring any integration steps that need more than just read access to the codebase.