Whither scala/scala CI?

[SethTisue]

Not sure where to put this. If you notice anything missing that could use improvement, feel free to edit this directly, or comment with suggestions or questions. Once we feel it's complete, perhaps we can find a place to put it.

This ticket replaces the similar older #507.

See also other tickets labeled CI/publishing/infra.

Basics for contributors

1. Jenkins and Travis-CI mostly test the same things

The redundancy is partly intentional (each system serves to check/verify that the other one is functioning as expected) and partly a historical accident (we are still experimenting with both and the experimentation hasn't concluded).

2. As a rule, every PR should pass both CIs before being merged

In particular, every commit in a PR must pass Jenkins. (Travis-CI only tests the last (edit-dnw) merge commit.)

For certain PRs, a maintainer might also choose to manually a trigger a Windows run (via GitHub Actions) and/or a community build run before merge.

3. If they both fail, feel free to investigate using whichever CI system you prefer or is more familiar

The Jenkins build is monolithic, which means you only see "pass/fail", and you have to go digging in the logs to see where the failure was. On the other hand, if the problem is a test failure, the Jenkins UI splits out each tests for you, so it's more digging initially but then less digging later.

The Travis-CI build is split into jobs: build and bootstrap, run partests, run junit and other tests, compile on Dotty... and we could plausibly split it up even further. You can see it a glance in the GitHub UI which part failed.

When digging through logs, there are other minor ergonomic differences between the two UIs.

4. If only one fails, you have some digging to do

See "Differences..." below.

5. Jenkins publishes your changes to a "validation snapshots" resolver

This happens even if some tests fail. See the scala/scala README for information.

Differences between Jenkins and Travis-CI for PR validation

Jenkins, in combination with Scabot (which we built ourselves and operate ourselves), tests every commit in a PR. Travis-CI only tests the last commit. It is perhaps not strictly necessary that we require every commit in a PR to pass CI, but it is desirable.

Jenkins tests each commit in the PR's branch. Travis-CI tests a temporary merge commit of the PR's branch and the target branch (e.g. 2.13.x). When we hit "merge" the HEAD of the target branch may already have moved on, so that result may be stale.

Jenkins uses older, substantially more complicated scripts for bootstrapping (see the scripts directory). Travis-CI uses a newer, simpler method (see .travis.yml). The simpler method also more closely resembles how we advise contributors to bootstrap locally. In the long run, we should standardize on the simpler method, but the work of getting rid of the old stuff remains to be done.

For no special reason, only Travis-CI includes the compileWithDotty test, which verifies that the standard library compiles with the latest Scala 3 release.

Only Travis-CI builds the language spec.

How did we get here?

Originally, Jenkins (https://scala-ci.typesafe.com) was our only CI system. But we have to set up and maintain Jenkins ourselves (https://github.com/scala/scala-jenkins-infra) and pay to operate the EC2 instances, so Jenkins is costly for us in both labor and money.

So when the free Travis-CI service came into existence, we thought, let's try it! But we weren't ready to commit to it, so we kept Jenkins around.

Contributors only need to think about PR validation, but the core Scala team also needs a way to publish releases. Originally Scala releases were published from Jenkins, but circa 2018 we decided to move 2.12.x and 2.13.x publishing to Travis-CI, where it has remained ever since.

Why have we kept Jenkins?

Jenkins is a pain to maintain and a pain to expand our CI matrix on (e.g. to other JDK versions), and it's less familiar to most contributors these days than Travis-CI or GitHub Actions. Why do we still have it?

Reasons related to PR validation

• Jenkins+Scabot tests every commit, not just the last commit.
• Jenkins knows the publishing secrets necessary to publish PR validation snapshots. Travis-CI doesn't allow pull request jobs to have access to secrets. (Dotty has this problem too: Automatically publish artifacts for PRs scala3#6145 remains open, as of April 2021.) PR validation snapshots are especially valuable for running the community build against before a PR is merged, but they're also helpful for other kinds of PR review.

Other reasons

• Jenkins also runs the community build, which is too big for a free service like Travis-CI. Thus, the cost of also having it do other things is amortized.
• Free services may come and go, and their free service tiers may get more restrictive. Running our own Jenkins insulates us from such changes.
• Sometimes it's helpful to troubleshoot Jenkins by ssh'ing to the workers ourselves. This kind of troubleshooting is less available on other services.

Why have we kept Travis-CI?

We can't easily get rid of Travis-CI without redoing release publishing, which is complex and fragile.

(But there's nothing in particular stopping us from switching from Travis-CI to GitHub Actions for PR validation. That part is just inertia.)

[SethTisue]

Jenkins also runs our test suite nightly on Windows. We could and probably should move that to another service, but doing so is work.

trying out Travis-CI's Windows support here: scala/scala#9493

[SethTisue]

for Windows, @griggt's GitHub Actions PR (scala/scala#9496) looks more promising than Travis-CI

[harpocrates]

It might be worth looking at what GHC has done: they're in a somewhat similar position of having limited resources but aspiring to check as many configurations as possible.

• some of the initial discussion of alternatives: https://gitlab.haskell.org/ghc/ghc/-/wikis/continuous-integration
• sample CI pipeline: https://gitlab.haskell.org/ghc/ghc/-/pipelines/33456 (a lot of resources end up being saved by gating the costly configurations to only run after the fast configurations/linters succeed, or after someone explicitly requests)
• they end up relying on dedicated machines whose time is donated by various orgs that use Haskell (see Our Sponsors). They use CircleCI (I think), but GitHub Actions has similar support for custom runners,

As far as the test suite goes, it would be super helpful to have some way of tagging particular tests so they only run on a specific configuration (JDK version, OS, etc.). Even if we don't have a CI configuration that tests those, the tests provide a good way of bisecting for regressions (or running costly configurations infrequently).

[lrytz]

I tried out https://github.com/philips-labs/terraform-aws-github-runner a few weeks (months?) ago and it worked very well. In short it sets up everything in an AWS account to use on-demand EC2 (spot) instances as custom runners for GitHub Actions.

Some challenges

• we currently build every commit in a PR, find a way to do that (if we still want that)
• manage permissions to allow pr builds to publish to our artifactory

One thing the project doesn't currently support is starting different instance types based on the runner label in GitHub, the GitHub API doesn't provide the necessary data (https://github.com/philips-labs/terraform-aws-github-runner/discussions/518). This might be useful for the community build, and for windows testing (https://github.com/philips-labs/terraform-aws-github-runner/issues/347).

[SethTisue]

several excellent PRs from @harpocrates are increasing the pressure on us to figure something out for JDK 11 and 16 (scala/scala#9551, scala/scala#9556, scala/scala#9548)

[SethTisue]

note that if anyone (perhaps someone whose initials are M.H.) is thinking of adding anything to our CI currently, favor adding it to Travis-CI, not Jenkins. the Jenkins build is monolithic and there are some pretty archaic scripts involved. whereas the way Travis-CI is set up is pretty close to how we would do it on GitHub Actions

[SethTisue]

moving Windows testing off Jenkins and onto GitHub Actions is happening at scala/scala#9496 and scala/scala#9485 (merged on 2.12.x, will be merged forward soon). it is run on merged PRs; it is not part of PR validation

and adding JDK 16 (and perhaps 11) to the 2.13 Travis-CI matrix has landed at scala/scala#9579

I have updated the issue description above to remove out-of-date information.

[SethTisue]

we replaced JDK 16 with 17