As a Racket developer, if you’ve been running tests and code coverage on your GitHub repos using Travis (for instance, via Greg Hendershott’s travis-racket), you will eventually discover (as I did a few days ago) that Travis is no longer free for open source projects, as it once was. An unfortunate turn of events. In response, the trend among open source developers seems to be to migrate to Github Actions, a free, disturbingly convenient, and admittedly good option, from what I can tell.
Well, blinders on! Let’s get down to it: how can you transition your Racket project from Travis to GitHub? If you don’t have CI set up at all, no worries, this post doesn’t assume a prior Travis setup — it just uses longstanding Travis workflows as guidelines for best practices with GitHub.
Setting Up a Racket Environment
Thankfully, Bogdan Popa has written a successor to travis-racket in setup-racket. This blog post is short and to the point, a great place to start, and you’re already almost done.
There are two aspects to the workflow encouraged in travis-racket that remain to be addressed:
The first is that Travis has a notion of “allowed failures” — tests that you want to run, but which, if they fail, shouldn’t mark the entire run as failed. For Racket developers this feature is particularly used to run against the latest daily snapshot build of Racket (and formerly, you might have used this to run against the Chez Scheme branch when it was still in development). The second aspect is that Travis has a notion of running something “after success” of your test run. For instance, you may run coverage here, so that it isn’t part of your test matrix but, rather, executed and uploaded to your coverage service provider at the end. Read on for the equivalent workflows using GitHub Actions.
Allowed Failures
For “allowed failures,” the equivalent GitHub Actions (henceforth, GA) notion is continue-on-error. If you followed Bogdan’s example in the linked blog post, you may have noticed that you parametrized your test “job” over Racket versions and variants. Essentially, you now just introduce an additional parameter for your tests called “experimental” (or whatever you like) which determines the value of continue-on-error for a particular test. This recipe from the GA docs explains how to set this up. Note that the include directive used there adds this as a one-off step so that it won’t be part of your overall matrix.
After Success
Re: running something “after success,” the equivalent GA recipe is to define a separate “job” in the same “workflow,” and indicate that this new job is contingent on the other one. To illustrate, if your existing job is called “test” and its purpose is to run tests, and the purpose of the new job is to run coverage, you would define this as a new job called “coverage” which includes a flag needs: test. This new job also must include all of the other setup from the test job, since each job is run in an independent virtual machine instance.
jobs:
test:
...
coverage:
needs: test
...
Before we see a complete example yml file for all this, there’s a problem we will need to contend with if we are interested in uploading coverage data to a third party service.
Getting Coverage to Work
Travis is supported out of the box by services like Coveralls and CodeCov — for instance, you don’t need any kind of coveralls-specific configuration in your repo in order for your travis builds to upload coverage data to coveralls. Not so with GA, where your coverage report will fail to upload, without some additional measures. One desirable option here would be to use a dedicated GA “action” provided by the coverage service provider themselves. I haven’t tried the CodeCov action, so if it works for you, then that’s great (if not, read on). For Coveralls, the action expects coverage data to be generated in “LCOV” format. Racket’s cover library doesn’t support this format yet as far as I know. So, until such a time, an alternative is to add your Coveralls (or CodeCov) repo token to your GA test environment so that coverage data can be uploaded the same way it was formerly when run by Travis. I’ll use the term “Coveralls” going forward, but this approach should work for both (and possibly other) providers.
The main obstacle with this second solution is that your GA test environment needs to have access to your Coveralls token, but the token needs to be kept secret since it allows anyone to upload coverage data for your repo. Can you imagine if one of those deviant internet trolls lurking in the Racket forums chooses to misrepresent your hard-earned 100% test coverage? Why, your prized coverage shield would betray you when you need it most, testifying in your hour of reckoning that indeed, despite the numerous exhortations of common wisdom and good sense, you could not be bothered to increase your test coverage past a paltry 46%. “I ask you, good sir or madam,” your coverage shield may as well say, “would you use software written by this developer? I wouldn’t.” So, as I’m sure you’ll agree, you can’t just check the token into your repo. Here’s how you can do it instead:
- First, create an environment (call it “
test-env“) for your repo by following the instructions here. - Then add your Coveralls token as a “secret” in this environment, call it
COVERALLS_REPO_TOKEN. Note that you will need to add this separately for each repo for which you track coverage, since each repo has a different token. - Add an “env” directive in your workflow yaml file to make the token available as an environment variable while your job is running. See the examples here.
Displaying the Status Badge
You can generate the status badge under the Actions tab for your repo, by clicking into the specific workflow, and you can then use this in your README as usual. You would find, however, that the badge image would show the name of the workflow yml file, which probably isn’t what you want. To remedy this, give your workflow a name — add something like name: build near the top of the file. This name will now be used in the badge instead of the filename.
And with all that, your final test.yml file (or whatever you chose to call it, in .github/workflows/) would look something like this:
name: build
on:
- push
jobs:
test:
runs-on: ubuntu-latest
continue-on-error: ${{ matrix.experimental }}
strategy:
matrix:
racket-variant: ['BC', 'CS']
racket-version: ['7.8', 'stable']
experimental: [false]
include:
- racket-version: 'current'
racket-variant: 'CS'
experimental: true
name: Test on Racket ${{ matrix.racket-variant }} ${{ matrix.racket-version }}
steps:
- name: Checkout
uses: actions/checkout@master
- name: Install Racket
uses: Bogdanp/setup-racket@v1.8.1
with:
architecture: 'x64'
distribution: 'full'
variant: ${{ matrix.racket-variant }}
version: ${{ matrix.racket-version }}
- name: Install Package and its Dependencies
run: raco pkg install --auto --batch
- name: Run Tests
run: raco test main.rkt
coverage:
needs: test
runs-on: ubuntu-latest
name: Report Coverage
environment: test-env
env:
COVERALLS_REPO_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }}
steps:
- name: Checkout
uses: actions/checkout@master
- name: Install Racket
uses: Bogdanp/setup-racket@v1.8.1
with:
architecture: 'x64'
distribution: 'full'
variant: 'CS'
version: 'stable'
- name: Install package and its dependencies
run: raco pkg install --auto --batch
- name: Install coverage tools
run: raco pkg install --auto --batch cover cover-coveralls
- name: Report coverage
run: raco cover -b -f coveralls .Hosting Your Own Documentation
[Added in June 2022, following a tip from Sam Phillips]
Racket’s package server automatically builds docs for every listed Racket package on a daily basis. It’s great to have high-quality, searchable, crosslinked documentation that your library is a part of. This convenience comes at a cost, however — a build failure in your package would mean that your docs would become unavailable. As the package refresh cycle is 24 hours, this means that even if you fixed the issue immediately, your docs could still be down for a day during which time your users would be left out in the cold. Added to this, the build failure could be because of something you did, or even because of a change in a dependency of your library, so that the availability of your own docs is not entirely in your hands.
But, it could be!
GitHub includes a site hosting service called GitHub Pages which can be integrated into your CI workflows, so that you can build a standalone version of your docs as part of your CI workflows, and have those docs be deployed to the GitHub Pages site for your repo. This will ensure that the documentation for your library will always have a backup that you control. Here’s how you can set that up:
(This follows the recipe in the racket-keyring repo.)
First, create a GitHub Pages site for your repo (in a nutshell: just create an orphan branch named gh-pages and push it up to GitHub. You can verify the site is recognized by visiting the “Pages” section of your repo settings). Then, add a GA workflow to your repo which (1) builds a standalone version of your docs, (2) pushes those docs to the branch in your repo from which the GitHub Pages site is built (typically the gh-pages branch), but (3) only when you’ve made changes to your docs or to the docs workflow itself, and (4) only when these changes have been made on the main branch.
That ends up looking something like this:
name: docs
on:
push:
branches:
- main
paths:
- 'repo-relative/path/to/scribblings/**'
- '.github/workflows/docs.yml'
defaults:
run:
shell: bash
jobs:
deploy-docs:
runs-on: ubuntu-latest
name: Build and deploy backup docs
steps:
- name: Checkout
uses: actions/checkout@master
- name: Install Racket
uses: Bogdanp/setup-racket@v1.8.1
with:
architecture: 'x64'
distribution: 'full'
variant: 'CS'
version: 'stable'
- name: Install Package and its Dependencies
run: raco pkg install --auto --batch
- name: Build docs for hosting
run: scribble +m --redirect-main http://pkg-build.racket-lang.org/doc/ --htmls --dest ./docs repo-relative/path/to/scribblings/my-project-name.scrbl
- name: Push to GitHub Pages
uses: JamesIves/github-pages-deploy-action@v4.3.3
with:
folder: docs/my-project-name
branch: gh-pagesPut that in .github/workflows/docs.yml and, assuming you’ve already set up your GitHub Pages site, you should find your docs there in a few minutes.
Tom P
Very helpful! But missing from your coverage workflow: `raco pkg install –auto –batch cover`
sid
Nice catch! I’ve incorporated the change 👍