Use sccache for builds

[jrmuizel]

mozilla/sccache#179 has some instructions.

When I last looked at these I believe I couldn't figure out how exactly to communicate the encrypted AWS key to sccache. The details are fuzzy though.

[jrmuizel]

In talking with kats I realized that we might be better off just trying to get sccache to work with task-cluster directly instead of getting it working with travis first.

[jrmuizel]

Here's an example of using secrets with taskcluster:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/upload-generated-sources/kind.yml#35
https://dxr.mozilla.org/mozilla-central/source/build/upload_generated_sources.py#94

and upload the secret using https://tools.taskcluster.net/secrets

[jrmuizel]

Also:
jrmuizel: you can set env vars for sccache for AWS creds (assuming your CI won't leak them in logs), or you could have a little script to rewrite the JSON secret into ~/.aws/credentials format
ted
https://github.com/mozilla/sccache/blob/master/src/simples3/credential.rs
ted
is the code that sccache uses to find AWS creds (forked from rusoto)

and https://dxr.mozilla.org/mozilla-central/source/taskcluster/scripts/builder/build-linux.sh#55

[glennw]

@jrmuizel @metajack What is the likelihood of getting someone at Mozilla who works on this stuff to look into this officially (running CI on task cluster)? I certainly don't know enough about that stuff to look into it.

We haven't been able to merge anything for ~3 days with the latest travis issues, which is following another fortnight of similar issues...

[jrmuizel]

@staktrace is looking at this a bit right now and we can probably get @luser to help before he goes on PTO next week.

[staktrace]

At the moment I only have basic linux64 taskcluster integration working. There are two steps involved:

1. Install the integration github app on the repo at https://github.com/apps/taskcluster (I have this app installed for my fork, staktrace/webrender)
2. Apply the change from my taskcluster-ci branch - this just adds the .taskcluster.yml file and tweaks one of the reftest fuzz numbers.

After doing this, each PR update or push will trigger the taskcluster job. You can see a sample one (from my taskcluster-ci commit) at https://tools.taskcluster.net/groups/GNsTmjKaQyeF5v623NM6eQ - it runs the debug and release commands on whatever the current stable rust version is. @jrmuizel said that for now the "nightly" rust commands we can just leave on travis.

If we want to pin to a specific rust version, we can update the docker image to have that rust version preinstalled, and remove the rustup commands from the .taskcluster.yml script.

Next steps are figuring out how to hook up sccache and getting OS X jobs running. I think OS X is probably more important at this point, since with that we can start using taskcluster "in production", and then work on getting it faster with sccache.

I'd like to merge my taskcluster-ci branch as soon as possible but I'm not sure what effect that will have on bors/homu and the regular workflow. We should probably set up a "maintenance window" or something where we can do the merge, and ensure things are working or roll back if they aren't. Or if there is a test repo somewhere with bors/homu we can use that to try this out.

[metajack]

sccache works fine, so I'm not sure what in paritcular is causing problems, or if your request is actually related to sccache builds or just builds being messed up in general. It's pretty easy for us to move a particular repo over to our buildbot instances if that is what is needed.

[staktrace]

I talked to taskcluster folks and I have steps on setting up OS X worker machines for taskcluster, so that we can run our own CI farm. It's fairly straightforward and I have it running using my laptop as a test. I think we can rustle up some OS X machines in the Toronto office or hosted remotely somewhere and use them as dedicated CI machines for webrender.

As a bonus, since the OS X setup doesn't use docker, it doesn't reset the machine state after each job is run. This means even if we just use a local sccache we should get a good speedup.

[jrmuizel]

The easiest thing to do is probably just get a dedicated mac mini or two from macstadium and expense it.

[staktrace]

I set up the worker on the mac mini that jrmuizel rented from macstadium. It seems to be working ok.

Next step to move this along is to try it on servo/webrender instead of just my clone of the repo. Whoever owns servo/webrender needs to install the github-taskcluster integration tool from https://github.com/apps/taskcluster.

Then we need to get :jonasfj to add the necessary scopes to this repo, so that it can spawn the "kats-webrender-ci-osx" type worker via the "localprovisioner" TC provisioner.

And then after that I can make a PR from my branch with the .taskcluster.yml file and see how bors/homu deal with it.

[glennw]

@metajack @larsbergstrom Is ^ something you can help with (enabling the TC tool on the WR repo)?

[metajack]

Should be done.

[staktrace]

Thanks. I got :jonasfj to add the scopes as well so we should be good to try the PR. I'll submit that shortly.

While I was waiting I installed sccache on the OS X worker but I ran into a rustc internal compiler error when building WR with it. I'll investigate that more but for now let's do this without sccache.

[luser]

We've seen that error before elsewhere:

thread 'rustc' panicked at 'failed to acquire jobserver token: Error { repr: Os { code: 35, message: "Resource temporarily unavailable" } }', src/libcore/result.rs:860:4

I think this is because cargo creates a make-style jobserver now, and it will pass it down to rustc (for use when you use codegen-units=N). There's some weird interaction here with how the jobserver fd gets passed down and I don't quite understand it.

[staktrace]

Yeah, I just commented in rust-lang/rust#42867 which appears to be tracking this problem.