This document aims to be a comprehensive guide to using and developing the Continuation Integration (CI) system employed by NimSkull.

Troubleshooting failures

Failing tests

While "Run tester" log will give you the full detail on the test run, it can be very hard to find the failing test in the jungle of passing tests, so there are several integrations built into CI to make this easier:

• Github Annotations are generated for test failures:
  • If you are adding new tests in your PR, failure in any of them will be annotated and you can find them directly in the "Files changed" view of the PR
• All test failures are isolated and printed in "Print all test errors" step. By default Github Actions collapse this step so you'll have to expand it.

"There are differences when building from source archive compared to building from git"

There are several reasons for why this happens:

• The compiler is using information only available when building directly from git, and the source archive generation routine is not updated to include this information in the archive and/or the compiler build is not coded to use the stored information.
• A cosmic ray hit the CI test system and made it generate a slightly different binary from the same source code. A re-run should clear the error in this case.

To get more details about the error, open the failing workflow run and go to "Summary". Scroll down to the "Artifacts" section and look for one named "differences between binary from source archive and git". This contains an HTML report generated by diffoscope about the differences.

If you need more information, the binaries that were compared can be found in the following artifacts:

• "binaries from source archive"
• "release binaries"

Reproducibility test failures

PRs to be merged are tested to make sure that the compiler build remains deterministic with regards to the environment. To learn more about why reproducible builds matter and the concepts, see the Reproducible builds project.

When tests of this category fail, the culprit usually are:

• Your changes introduces nondeterministic variables like the current date/current time.
• If you are writing packaging code, it should be aware that file system ordering is nondeterministic and files must be ordered in memory before they are added to archives.

Text diffs are generated, but there is no highlighting and they can be hard to read when the differences are large. To help with this, companion HTML reports are also generated and can be downloaded from Github Artifacts.

To access the HTML report, open the failing test run and go to "Summary". The reports are saved in artifacts with suffix "reproducibility test diffoscope output".

Lifecycle of an approved PR

The only way to get code into NimSkull's development branch is via PRs, and CI was developed to leverage this.

1. PR is staged for merging.
2. The PR enters the merge queue and is merged to a staging branch with gh-readonly-queue/ prefix. Multiple PRs might be staged in this period to increase efficiency.
3. The testing pipelines test the code, generate binaries and render documentation.
4. Assuming that the staging branch passes all tests, devel is fast-forwarded to the staging branch head commit.
5. The "Publish" pipeline deploys the binaries generated by the testing pipeline earlier in the staging branch.
6. The "Deploy documentation" pipeline deploys the latest documentation to GitHub Pages.

Assumptions in CI design

This lifecycle has a number of useful properties which we use in our approach to CI as follows:

• Code entering devel must have passed through the staging branch, and that devel will be fast-forwarded to this branch:
  • This allows us to run crucial evaluation tasks on the staging branch only, and when that passes and devel is fast-forwarded, we can access prior runs and obtain build results.
• Only "light" tests have to be run on PRs to help developers debug their code before it is reviewed by a human. More time-consuming tests that are not likely to fail can be run later when the PR enters the merge queue.

Pipelines

This section describes the design of each pipeline and the dependencies between them.

"CI"

Goals

• Verify that the compiler, tools, and standard library test suites pass
• Verify that a source archive can be built.
  • Compiler and tools built from the source archive work.
  • Building from source archive is identical to building from a git clone.
• Render HTML documentation for upload to Github Pages.
• Build release binaries.

API

This pipeline handles all the building work, but it does not handle the task of publishing it's generated data. That part is handled by the "Publish" pipeline.

To support publishing, this pipeline exports the build result as Github Artifacts, which are used by other pipelines, these artifacts are:

• "release manifest tool": contains a built copy of tools/release_manifest.nim for Linux.
• "release binaries": built via koch unixrelease and also contains related JSON metadata. These binaries are built straight from the git clone.
• "source archive": generated from the git clone.

Changes to how these artifacts are packaged must be reviewed carefully to ensure that dependent pipelines will still function.

This pipeline also makes use of multiple jobs, which can make tracking its success status hard. To that end, an "All check passed" check is provided that will reflect the final status of the pipeline. Currently this is used for integration with other tools that rely on Github Checks status.

Job dependency graph and internal API

In order to maximize throughput, steps are split into multiple dependent jobs. This allows for:

• Unrelated steps to execute in parallel on multiple VMs.
• Quick turn-around time for test runs without waiting for packaging to finish.

Github Actions UI provides an overall look into the dependency graph which can be seen from "Summary" page of any "CI" workflow run.

In addition to the public API, there are several internal APIs used for sharing data between jobs in this pipeline:

• "binaries from source archive": Contains binaries built from the source archive generated from the git clone. This is generated by package-source job and will be compared to a similar artifact generated by the package-git job.

  This comparison is done to make sure that we can generate the same binaries regardless of whether it is compiled via cloning a Git repo or via our source archive.

• pre_run outputs: The pre_run job provides data that dictates how CI will be run:
  • skipped: Whether a run should be skipped.
  • matrix: The target matrix shared between jobs.

"Build and test"

This workflow builds and test the compiler against the full test suite on a given runner. This workflow is reusable, which meant it can be called by other workflows with varying customizations. Please refer to the workflow file at .github/workflows/build-and-test.yml for more information.

This workflow is used by "CI" workflow to perform testing across all supported platforms.

Internally, this pipeline provides the following API used to share data between jobs:

• "Download compiler" and "Upload compiler" actions: These can be found in .github/actions/download-compiler and .github/actions/upload-compiler respectively. These actions allow a job to upload its workspace, all files changed/generated compared to the git source code, and allow that to be downloaded by dependent jobs.

  This is used to replicate the "workspace with binaries" across parallel jobs so they can run independently.

"Build release package"

This workflow builds the compiler release package (including docs) on a given runner. It can build either from Git or from a source archive. Refer to the workflow file at .github/workflows/package.yml for information on how to employ these.

This workflow is used by "CI" workflow to build binaries for all supported platforms, and to generate binaries from source and Git for comparison.

"Test compiler build reproducibility"

This pipeline verifies that the compiler can be deterministically built. See the Reproducible builds project for more information on the testing criteria.

This pipeline has a rather long runtime, and rarely fails, so it is only run once a PR enters the merge queue.

As a workaround for GitHub's lack of support for different criteria for merge versus entering the queue, this pipeline will be skipped and always report success for PRs.

This pipeline is developed with a rather standard matrix and does not perform any data sharing. Working with this pipeline only requires understanding on the reprotest tool.

"Publish"

Publishes build results from prior run of the "CI" pipeline on the staging branch. Changes to the public API of "CI" must be tested against "Publish".

This pipeline will perform pushes to the repository, so a lot of care must be put into any changes to it.

Only one instance of this pipeline might be run at any given time, due to its mutating nature. Currently Github Action's concurrency feature is used to block multiple runs.

"Deploy documentation"

As the name suggests, this pipeline constructs and deploys the documentation website. This pipeline is dependent on the Release manifest tool.

Unlike other pipelines, the code run will always be that of the latest commit in devel branch. This means that the code should always be aware of differences between versions of NimSkull being deployed.

Currently, only one instance of this pipeline might be run at any given time to prevent races. GitHub Actions's concurrency feature is used to serialize the runs.

Development guidelines

General

• Each CI VM is weak, so prioritize parallelization by dividing the work into multiple independent jobs that can be executed by multiple VMs.
• Avoid repeated runs by extracting as many build results from other pipelines/jobs as possible: CI concurrency is limited, so running less is preferable.
• Steps and errors should be as descriptive as possible: Developers must understand how testing is done in order to debug their code.
• Make jobs that provide highly relevant statuses run as early as possible, for example, tests should be run as soon as it can be done. If tests don't pass reproducibility is a moot point.
• Pipelines that generate release artifacts should use pinned dependencies and OS. This avoid breakage due to random differences in software version.

Github Actions

• Prefer using a version tag. This makes sure that errorneous updates to an action will not cause our CI to fail.

Tools

This section documents some of the tools created for use by CI.

Github Actions Problem Matcher

A problem matcher for NimSkull code is provided at .github/nim-problem-matcher.json.

This can be loaded into a job with

echo "::add-matcher::.github/nim-problem-matcher.json"

and will convert all messages generated by the NimSkull compiler into annotations.

Print all failed testament results

tools/ci_testresults.nim can be used to print only test failures from a testament run.

Use this after a failed testament run to provide developers with a dedicated view of all test failures.

It is also integrated with Github Actions to generate file annotations for failed tests.

Release manifest tool

Located at tools/release_manifest.nim, this tool can be used to compose a release manifest or to query it. This tool can be run on developer machine and has rather comprehensive built-in documentation, please refer to that for more details.