Continuous integration

We have a variety of tooling on Buildkite and GitHub Actions for continuous integration.

Buildkite pipelines

Tests are automatically run in our various Buildkite pipelines. Pipeline steps are generated using the pipeline generator.

To see what checks will get run against your current branch, use sg:

sg ci preview

Development

The source code of the pipeline generator is in /enterprise/dev/ci.

To test the rendering of the entire pipeline, you can run env BUILDKITE_BRANCH=TESTBRANCH go run ./enterprise/dev/ci/gen-pipeline.go and inspect the YAML output. To change the behaviour set the relevant BUILDKITE_ environment variables.

Pipeline operations

Pipeline steps are defined as Operations that apply changes to the given pipeline, such as adding steps and components.

(:[_] *bk.Pipeline) patternType:structural repo:^github\.com/sourcegraph/sourcegraph$ file:^enterprise/dev/ci/internal/ci/operations\.go

Within an Operation you will typically create one or more steps on a pipeline with AddStep, which can be configured with options of type SteptOpt.

(:[_]) StepOpt patternType:structural repo:^github\.com/sourcegraph/sourcegraph$ file:^enterprise/dev/ci/internal/buildkite/buildkite\.go

Operations are then added to a pipeline from GeneratePipeline.

For most basic PR checks, see Creating PR checks for how to create your own steps!

For more advanced usage for specific run types, see Run types.

Creating PR checks

To create a new check that can run on pull requests on relevant files, check the changed.Files type to see if a relevant affectsXyz check already exists.

Affects type:symbol select:symbol.function repo:^github\.com/sourcegraph/sourcegraph$ file:^enterprise/dev/ci/internal/ci/changed

If not, you can define a new one on the changed.Files type.

Then, you can add a new check to CoreTestOperations. Make sure to follow the best practices outlined in docstring.

For more advanced pipelines, see Run types.

Run types

There are a variety of run types available based on branch prefixes. These generate special-purpose pipelines. For example, the main-dry-run/ prefix is used to generate a pipeline similar to the default main branch. See RunType for the various run types available, and examples for how to add more.

GeneratePipeline can leverage RunType when generating pipelines, for example:

RunType.Is(:[_]) OR case :[_]: patternType:structural repo:^github\.com/sourcegraph/sourcegraph$ file:^enterprise/dev/ci/internal/ci/pipeline\.go

For simple PR checks, see Creating PR checks.

Flakes

A flake is generally characterized as one-off or rare issues that can be resolved by retrying the failed job or task. In other words: something that sometimes fails, but if you retry it enough times, it passes, eventually.

Tests are not the only thing that are flaky - flakes can also encompass sporadic infrastructure issues and other unreliable steps.

Flaky tests

Learn more about our flaky test policy in Testing principles: Flaky tests.

Use language specific functionality to skip a test. Create an issue and ping an owner about the skipping (normally on the PR skipping it).

If the language or framework allows for a skip reason, include a link to the issue track re-enabling the test, or leave a docstring with a link.

Flaky steps

If a step is flaky we need to get the build back to reliable as soon as possible. If there is not already a discussion in #buildkite-main create one and link what step you take. Here are the recommended approaches in order:

  1. Revert the PR if a recent change introduced the instability. Ping author.
  2. Use Skip StepOpt when creating the step. Include reason and a link to context. This will still show the step on builds so we don’t forget about it.

An example use of Skip:

--- a/enterprise/dev/ci/internal/ci/operations.go
+++ b/enterprise/dev/ci/internal/ci/operations.go
@@ -260,7 +260,9 @@ func addGoBuild(pipeline *bk.Pipeline) {
 func addDockerfileLint(pipeline *bk.Pipeline) {
        pipeline.AddStep(":docker: Lint",
                bk.Cmd("./dev/ci/docker-lint.sh"),
+               bk.Skip("2021-09-29 example message https://github.com/sourcegraph/sourcegraph/issues/123"),
        )
 }

Buildkite infrastructure

Pipeline setup

To set up Buildkite to use the rendered pipeline, add the following step in the pipeline settings:

go run ./enterprise/dev/ci/gen-pipeline.go | buildkite-agent pipeline upload

Managing secrets

The term secret refers to authentication credentials like passwords, API keys, tokens, etc. which are used to access a particular service. Our CI pipeline must never leak secrets:

  • to add a secret, use the Secret Manager on Google Cloud and then inject it at deployment time as an environment variable in the CI agents, which will make it available to every step.
  • use an environment variable name with one of the following suffixes to ensure it gets redacted in the logs: *_PASSWORD, *_SECRET, *_TOKEN, *_ACCESS_KEY, *_SECRET_KEY, *_CREDENTIALS
  • while environment variables can be assigned when declaring steps, they should never be used for secrets, because they won’t get redacted, even if they match one of the above patterns.

Vulnerability Scanning

Our CI pipeline scans uses Trivy to scan our Docker images for security vulnerabilities.

Trivy will perform scans upon commits to the following branches:

  1. main
  2. branches prefixed by main-dry-run/
  3. branches prefixed by docker-images-patch/$IMAGE (where only a single image is built)

If there are any HIGH or CRITICAL severities in a Docker image that have a known fix:

  1. The CI pipeline will create an annotation that contains links to reports that describe the vulnerabilities
  2. The Trivy scanning step will soft fail. Note that soft failures do not fail builds or block deployments. They simply highlight the failing step for further analysis.

GitHub Actions

Third-Party Licenses

We use the license_finder tool to check third-party dependencies for their licenses. It runs as a GitHub Action on pull requests, which will fail if one of the following occur:

  • If the license for a dependency cannot be inferred. To resolve:
    • Use license_finder licenses add <dep> <license> to set the license manually
  • If the license for a new or updated dependency is not on the list of approved licenses. To resolve, either:
    • Remove the dependency
    • Use license_finder ignored_dependencies add <dep> --why="Some reason" to ignore it
    • Use license_finder permitted_licenses add <license> --why="Some reason" to allow the offending license

The license_finder tool can be installed using gem install license_finder. You can run the script locally using:

# updates ThirdPartyLicenses.csv
./dev/licenses.sh

# runs the same check as the one used in CI, returning status 1
# if there are any unapproved dependencies ('action items')
LICENSE_CHECK=true ./dev/licenses.sh

The ./dev/licenses.sh script will also output some license_finder configuration for debugging purposes - this configuration is based on the doc/dependency_decisions.yml file, which tracks decisions made about licenses and dependencies.

For more details, refer to the license_finder documentation.