Periodic ENOENT running in CI

[nathancahill]

This is an error that happens >50% of the time on CircleCI. No changes are made between one run and the next. One run might pass, the next will fail with this error. All versions are committed in yarn.lock.

12 06 2020 08:29:13.851:ERROR [SaucelabsLauncher]: Error: spawn /home/circleci/repo/node_modules/saucelabs/build/.sc-v4.5.4/bin/sc ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:268:19)
    at onErrorNT (internal/child_process.js:468:16)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

{
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn /home/circleci/repo/node_modules/saucelabs/build/.sc-v4.5.4/bin/sc',
  path: '/home/circleci/repo/node_modules/saucelabs/build/.sc-v4.5.4/bin/sc',
  spawnargs: [ '--version' ],
  killed: false,
  stdout: '',
  stderr: '',
  failed: true,
  signal: null,
  cmd: '/home/circleci/repo/node_modules/saucelabs/build/.sc-v4.5.4/bin/sc --version',
  timedOut: false
}

[christian-bromann]

@nathancahill can you try to run the pipeline without cache? I am afraid that the cache that includes the Sauce Connect binary is used for different environments.

[christian-bromann]

ping @nathancahill

[nathancahill]

Thanks, I think removing the cache solved the issue. I'll reopen in the future if it reappears.

[nathancahill]

Even with the cache disabled we're still getting periodic errors.

Previously, with the cache, the error rate was probably 50%. Without the cache, it's around 10%.

[Sam55555]

The error occurred too on each run after gitlab runner cached on gitlab ci using the docker executer. No error during the first run and clean cache.

[enriquegh]

Haven't been able to reproduce on a private Gitlab instance with a docker runner so far.

I was able to reproduce, however, if I was using the node alpine image:

 $ NODE_OPTIONS=--trace-warnings ts-node index.ts
Error: spawn /builds/enrique/sauce-sc-example/node_modules/saucelabs/build/.sc-v4.5.4/bin/sc ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:268:19)
    at onErrorNT (internal/child_process.js:468:16)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn /builds/enrique/sauce-sc-example/node_modules/saucelabs/build/.sc-v4.5.4/bin/sc',
  path: '/builds/enrique/sauce-sc-example/node_modules/saucelabs/build/.sc-v4.5.4/bin/sc',
  spawnargs: [ '--version' ],
  killed: false,
  stdout: '',
  stderr: '',
  failed: true,
  signal: null,
  cmd: '/builds/enrique/sauce-sc-example/node_modules/saucelabs/build/.sc-v4.5.4/bin/sc --version',
  timedOut: false
}

This gave an error 100% of the time though and not intermittently.

[enriquegh]

From @joventuraz:

it looks like the node module is not available, if there happened to be more than one concurrent build and one is finishing and cleaning up, it might affect the other build if they are using the same workspace which it seems like they might from the path in the error

Adding to this I think the issue lies somewhere in bin-wrapper and how it does its checks for the file/folder.

It seems like bin-wrapper is a collection of small modules so it was a bit hard to pinpoint exactly what is missing.

Some things we could try and do:

• Do more checks on whether the binary is actually there or not and retry to download if its not
• Allow for users to specify a folder where sc should be looked for and the user can download this manually
• use something other than bin-wrapper since it seems to not be very active

[christian-bromann]

I proposed a change in the Sauce Connect Launcher to retry starting Sauce Connect if it fails: karma-runner/karma-sauce-launcher#219

Let's see if we can fix this issue with that.