Migrating Your Racket Project from Travis to GitHub Actions

As a Racket developer, if you’ve been running tests and code coverage on your GitHub repos using Travis (for instance, via Greg Hendershott’s travis-racket), you will eventually discover (as I did a few days ago) that Travis is no longer free for open source projects, as it once was. An unfortunate turn of events. In response, the trend among open source developers seems to be to migrate to Github Actions, a free, disturbingly convenient, and admittedly good option, from what I can tell.

Well, blinders on! Let’s get down to it: how can you transition your Racket project from Travis to GitHub? If you don’t have CI set up at all, no worries, this post doesn’t assume a prior Travis setup — it just uses longstanding Travis workflows as guidelines for best practices with GitHub.

Setting Up a Racket Environment

Thankfully, Bogdan Popa has written a successor to travis-racket in setup-racket. This blog post is short and to the point, a great place to start, and you’re already almost done.

There are two aspects to the workflow encouraged in travis-racket that remain to be addressed:

The first is that Travis has a notion of “allowed failures” — tests that you want to run, but which, if they fail, shouldn’t mark the entire run as failed. For Racket developers this feature is particularly used to run against the latest daily snapshot build of Racket (and formerly, you might have used this to run against the Chez Scheme branch when it was still in development). The second aspect is that Travis has a notion of running something “after success” of your test run. For instance, you may run coverage here, so that it isn’t part of your test matrix but, rather, executed and uploaded to your coverage service provider at the end. Read on for the equivalent workflows using GitHub Actions.

Allowed Failures

For “allowed failures,” the equivalent GitHub Actions (henceforth, GA) notion is continue-on-error. If you followed Bogdan’s example in the linked blog post, you may have noticed that you parametrized your test “job” over Racket versions and variants. Essentially, you now just introduce an additional parameter for your tests called “experimental” (or whatever you like) which determines the value of continue-on-error for a particular test. This recipe from the GA docs explains how to set this up. Note that the include directive used there adds this as a one-off step so that it won’t be part of your overall matrix.

After Success

Re: running something “after success,” the equivalent GA recipe is to define a separate “job” in the same “workflow,” and indicate that this new job is contingent on the other one. To illustrate, if your existing job is called “test” and its purpose is to run tests, and the purpose of the new job is to run coverage, you would define this as a new job called “coverage” which includes a flag needs: test. This new job also must include all of the other setup from the test job, since each job is run in an independent virtual machine instance.

jobs:
  test:
    ...
  coverage:
    needs: test
    ...

Before we see a complete example yml file for all this, there’s a problem we will need to contend with if we are interested in uploading coverage data to a third party service.

Getting Coverage to Work

Travis is supported out of the box by services like Coveralls and CodeCov — for instance, you don’t need any kind of coveralls-specific configuration in your repo in order for your travis builds to upload coverage data to coveralls. Not so with GA, where your coverage report will fail to upload, without some additional measures. One desirable option here would be to use a dedicated GA “action” provided by the coverage service provider themselves. I haven’t tried the CodeCov action, so if it works for you, then that’s great (if not, read on). For Coveralls, the action expects coverage data to be generated in “LCOV” format. Racket’s cover library doesn’t support this format yet as far as I know. So, until such a time, an alternative is to add your Coveralls (or CodeCov) repo token to your GA test environment so that coverage data can be uploaded the same way it was formerly when run by Travis. I’ll use the term “Coveralls” going forward, but this approach should work for both (and possibly other) providers.

The main obstacle with this second solution is that your GA test environment needs to have access to your Coveralls token, but the token needs to be kept secret since it allows anyone to upload coverage data for your repo. Can you imagine if one of those deviant internet trolls lurking in the Racket forums chooses to misrepresent your hard-earned 100% test coverage? Why, your prized coverage shield would betray you when you need it most, testifying in your hour of reckoning that indeed, despite the numerous exhortations of common wisdom and good sense, you could not be bothered to increase your test coverage past a paltry 46%. “I ask you, good sir or madam,” your coverage shield may as well say, “would you use software written by this developer? I wouldn’t.” So, as I’m sure you’ll agree, you can’t just check the token into your repo. Here’s how you can do it instead:

1. First, create an environment (call it “test-env“) for your repo by following the instructions here.
2. Then add your Coveralls token as a “secret” in this environment, call it COVERALLS_REPO_TOKEN. Note that you will need to add this separately for each repo for which you track coverage, since each repo has a different token.
3. Add an “env” directive in your workflow yaml file to make the token available as an environment variable while your job is running. See the examples here.

Displaying the Status Badge

You can generate the status badge under the Actions tab for your repo, by clicking into the specific workflow, and you can then use this in your README as usual. You would find, however, that the badge image would show the name of the workflow yml file, which probably isn’t what you want. To remedy this, give your workflow a name — add something like name: build near the top of the file. This name will now be used in the badge instead of the filename.

And with all that, your final test.yml file (or whatever you chose to call it, in .github/workflows/) would look something like this:

name: build

on:
  - push

jobs:
  test:
    runs-on: ubuntu-latest
    continue-on-error: ${{ matrix.experimental }}
    strategy:
      matrix:
        racket-variant: ['BC', 'CS']
        racket-version: ['7.8', 'stable']
        experimental: [false]
        include:
          - racket-version: 'current'
            racket-variant: 'CS'
            experimental: true
    name: Test on Racket ${{ matrix.racket-variant }} ${{ matrix.racket-version }}
    steps:
      - name: Checkout
        uses: actions/checkout@master
      - name: Install Racket
        uses: Bogdanp/setup-racket@v1.8.1
        with:
          architecture: 'x64'
          distribution: 'full'
          variant: ${{ matrix.racket-variant }}
          version: ${{ matrix.racket-version }}
      - name: Install Package and its Dependencies
        run: raco pkg install --auto --batch
      - name: Run Tests
        run: raco test main.rkt
  coverage:
    needs: test
    runs-on: ubuntu-latest
    name: Report Coverage
    environment: test-env
    env:
      COVERALLS_REPO_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }}
    steps:
      - name: Checkout
        uses: actions/checkout@master
      - name: Install Racket
        uses: Bogdanp/setup-racket@v1.8.1
        with:
          architecture: 'x64'
          distribution: 'full'
          variant: 'CS'
          version: 'stable'
      - name: Install package and its dependencies
        run: raco pkg install --auto --batch
      - name: Install coverage tools
        run: raco pkg install --auto --batch cover cover-coveralls
      - name: Report coverage
        run: raco cover -b -f coveralls .

Hosting Your Own Documentation

[Added in June 2022, following a tip from Sam Phillips]

Racket’s package server automatically builds docs for every listed Racket package on a daily basis. It’s great to have high-quality, searchable, crosslinked documentation that your library is a part of. This convenience comes at a cost, however — a build failure in your package would mean that your docs would become unavailable. As the package refresh cycle is 24 hours, this means that even if you fixed the issue immediately, your docs could still be down for a day during which time your users would be left out in the cold. Added to this, the build failure could be because of something you did, or even because of a change in a dependency of your library, so that the availability of your own docs is not entirely in your hands.

But, it could be!

GitHub includes a site hosting service called GitHub Pages which can be integrated into your CI workflows, so that you can build a standalone version of your docs as part of your CI workflows, and have those docs be deployed to the GitHub Pages site for your repo. This will ensure that the documentation for your library will always have a backup that you control. Here’s how you can set that up:

(This follows the recipe in the racket-keyring repo.)

First, create a GitHub Pages site for your repo, for which you could follow steps 1-4 in this blog post. Then, add a GA workflow to your repo which (1) builds a standalone version of your docs, (2) pushes those docs to the branch in your repo from which the GitHub Pages site is built (typically the gh-actions branch), but (3) only when you’ve made changes to your docs or to the docs workflow itself, and (4) only when these changes have been made on the main branch.

That ends up looking something like this:

name: docs

on:
  push:
    branches:
      - main
    paths:
      - 'repo-relative/path/to/scribblings/**'
      - '.github/workflows/docs.yml'

defaults:
  run:
    shell: bash

jobs:
  deploy-docs:
    runs-on: ubuntu-latest
    name: Build and deploy backup docs
    steps:
      - name: Checkout
        uses: actions/checkout@master
      - name: Install Racket
        uses: Bogdanp/setup-racket@v1.8.1
        with:
          architecture: 'x64'
          distribution: 'full'
          variant: 'CS'
          version: 'stable'
      - name: Install Package and its Dependencies
        run: raco pkg install --auto --batch
      - name: Build docs for hosting
        run: scribble +m --redirect-main http://pkg-build.racket-lang.org/doc/ --htmls --dest ./docs repo-relative/path/to/scribblings/my-project-name.scrbl
      - name: Push to GitHub Pages
        uses: JamesIves/github-pages-deploy-action@v4.3.3
        with:
          folder: docs/my-project-name
          branch: gh-pages

Put that in .github/workflows/docs.yml and, assuming you’ve already set up your GitHub Pages site, you should find your docs there in a few minutes.