Building project graph on postinstall causes process to incorrectly exit with code 1 when libraries with individual project.json files are not yet present

[markpeterfejes]

Current Behavior

Building the project graph in the postinstall step breaks workspace dependency installation.

Expected Behavior

Even if an error occurs while building the project graph nx should not error and break the whole installation as stated here.
It seems this started to happen in v13 after this PR.

Steps to Reproduce

1. Create a fresh nx repo.
2. Create a couple of libraries with independent project.json files.
3. Delete the libs folder and the node modules folder
4. Try to install the dependencies of the repo.

After this it will say that the daemon couldn't finish the graph build, but in a docker container this will cause the whole installation to error out as per this condition

My suspicion is that despite that we are catching any errors in the init.ts, it will still stop the installation since we are exiting the process here.

The reason why the libraries are actually not there at the point of installation is that we are copying them in our Dockerfile after the installation step in order to take advantage of docker layer caching. (changing application code can still reuse the dependency-install layer of the workspace)

   Node : 16.14.0
   OS   : darwin x64
   yarn : 1.22.17

   nx : 13.9.2
   @nrwl/angular : 13.9.2
   @nrwl/cypress : 13.9.2
   @nrwl/detox : undefined
   @nrwl/devkit : 13.9.2
   @nrwl/eslint-plugin-nx : 13.9.2
   @nrwl/express : undefined
   @nrwl/jest : 13.9.2
   @nrwl/js : 13.9.2
   @nrwl/linter : 13.9.2
   @nrwl/nest : undefined
   @nrwl/next : undefined
   @nrwl/node : 13.9.2
   @nrwl/nx-cloud : 13.1.6
   @nrwl/nx-plugin : undefined
   @nrwl/react : undefined
   @nrwl/react-native : undefined
   @nrwl/schematics : undefined
   @nrwl/storybook : 13.9.2
   @nrwl/web : undefined
   @nrwl/workspace : 13.9.2
   typescript : 4.6.2
   rxjs : 7.5.5
   ---------------------------------------
   Community plugins:
   	 @ionic/angular: 6.0.12
   	 @ngrx/component: 13.0.2
   	 @ngrx/component-store: 13.0.2
   	 @ngrx/data: 13.0.2
   	 @ngrx/effects: 13.0.2
   	 @ngrx/entity: 13.0.2
   	 @ngrx/router-store: 13.0.2
   	 @ngrx/store: 13.0.2
   	 @ngrx/store-devtools: 13.0.2
   	 ts-node-builder: 2.2.0
   	 @funxtion/schematics: 0.0.1
   	 @ionic/cordova-builders: 6.1.0
   	 @ngrx/schematics: 13.0.2
   	 @nxtend/ionic-angular: 13.0.0

[tommazzo89]

We are having the same problem when building our Dockerfiles (e.g., here). Thank you for creating this issue @markpeterfejes!

With Docker the issue seems to depend on whether BuildKit is enabled or now. To reproduce the problem with our Docker builds you can follow these steps:

git clone https://github.com/polaris-slo-cloud/polaris-demos.git
cd polaris-demos/efficiency-demo

# This fails
DOCKER_BUILDKIT=0 docker build -f ./apps/eff-controller/Dockerfile --build-arg POLARIS_APP_TYPE=slo --build-arg POLARIS_APP_NAME=eff-controller -t polarissloc/eff-controller:latest .

# This works
DOCKER_BUILDKIT=1 docker build -f ./apps/eff-controller/Dockerfile --build-arg POLARIS_APP_TYPE=slo --build-arg POLARIS_APP_NAME=eff-controller -t polarissloc/eff-controller:latest .

I agree with @markpeterfejes that exiting the process with an error code here in buildProjectGraphWithoutDaemon() seems to be the problem.

The reason why the Docker build works with DOCKER_BUILDKIT=1 seems to be that createProjectGraphAsync() doesn't correctly detect the Docker build here when using BuildKit.

[tommazzo89]

Hi guys,

Is there any update on this issue? I've tried upgrading to Nx v13.10.2 and the problem still persists. Is there maybe a way

1. to disable the project graph computation for Docker builds (e.g., by setting an environment variable) or
2. to keep all project configurations in the workspace.json/angular.json file, like it was in older Nx versions?

Best regards,
Thomas

[ianldgs]

FYI, what happened to me was the OS was reporting 256 CPUs. That was a larger number than the amount of files that it had to calculate the dependency graph. That was causing a strange error and the stack trace was being swallowed. My fix was to set the environment variable for the number of workers to a reasonable number (4 in my case).

[tommazzo89]

Thank you for the hint @ianldgs, but unfortunately the problem seems to be a different one here. Nx tries to compute the project graph in a postinstall hook that gets executed as part of npm install (actually it's npm ci in the Dockerfile). This graph computation tries to open all project.json files, which are not available at this stage of the Docker build, because we only copy nx.json, package.json, tsconfig.base.json, package-lock.json, and workspace.json to allow Docker to cache the result of npm ci.

Nevertheless, I did try Nx v13.10.3, which fixes the bug that you mentioned (#9801), but the result is still the same: npm ci exits with the following error message:

>  NX   ENOENT: no such file or directory, open '/usr/src/polaris/apps/eff-controller/project.json'

[tommazzo89]

I have created a pull request to fix this issue.

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.