Skip to content

Commit a80f24a

Browse files
fix(gradle): prevent Gradle and Maven daemon accumulation during project graph recalculation (#35143)
## Current Behavior When file changes trigger rapid project graph recalculations (e.g., during development with `nx graph` or a dev server running), each call to `populateProjectGraph` spawns a new Gradle process via `execGradleAsync`. If the previous Gradle daemon is still busy processing the prior request, Gradle spawns a **new daemon**. These daemons persist for 3 hours by default (Gradle's idle timeout), leading to dozens of orphaned `java.exe` processes consuming significant memory. Maven has a similar issue — `runMavenAnalysis` spawns a long-lived process with no timeout or cancellation support at all. ### Root Cause Analysis The daemon explosion happens due to three compounding issues: 1. **No cancellation of in-flight processes**: When a newer project graph request arrives, the previous invocation continues running. Each concurrent invocation finds the existing daemon busy and spawns a new one. 2. **Windows process tree issue**: On Windows, `execFile` with `shell: true` runs `cmd.exe → gradlew.bat → java.exe`. Node's `AbortSignal` only terminates `cmd.exe` (the immediate child process), leaving `java.exe` running as an orphan. 3. **No timeout for Maven**: Maven analysis could run indefinitely with no way to cancel or time out. ### Reproduction 1. Run `nx graph` in a workspace with `@nx/gradle` registered 2. Rapidly modify a `build.gradle.kts` file (e.g., `while true; do echo "// tick" >> build.gradle.kts; sleep 0.01; done`) 3. Watch `java.exe` processes accumulate: `tasklist | grep java` (Windows) or `ps aux | grep java` (Unix) 4. Without this fix: **15+ Gradle daemons** within seconds, persisting for 3 hours each 5. With this fix: **1 Gradle daemon** remains stable under the same conditions ## Expected Behavior When rapid file changes trigger multiple project graph recalculations: - The previous invocation is cancelled before starting a new one - Cancelled calls that have already spawned a process get their entire process tree killed (not just the shell wrapper) - Only 1 daemon remains active at any time, rather than accumulating dozens - Both Gradle and Maven have configurable timeouts with clear error messages ## Changes ### Gradle #### 1. Self-contained cancellation in `get-project-graph-lines.ts` Moved the `AbortController` from `get-project-graph-from-gradle-plugin.ts` into `get-project-graph-lines.ts`, closer to where processes are spawned. `getNxProjectGraphLines` now manages its own abort controller — cancelling any in-flight request before starting a new one. Uses `abort('cancelled')` reason to distinguish external cancellation from timeout. #### 2. Tree-kill on abort (`exec-gradle.ts`) Instead of passing the `AbortSignal` directly to Node's `execFile` (which only kills the immediate child process), we intercept the signal and use `tree-kill` to terminate the entire process tree. This ensures `java.exe` is killed along with `cmd.exe` and `gradlew.bat` on Windows. ### Maven #### 3. Timeout and cancellation support for `maven-analyzer.ts` Added the same abort controller + tree-kill + timeout pattern to Maven analysis: - Configurable timeout via `NX_MAVEN_ANALYSIS_TIMEOUT` env var (default: 120s local, 600s CI) - `cancelPendingMavenAnalysis()` cancels in-flight processes on repeated calls - `tree-kill` ensures the entire Maven process tree is killed on abort - Clear timeout error messages with actionable steps --------- Co-authored-by: nx-cloud[bot] <71083854+nx-cloud[bot]@users.noreply.github.com>
1 parent 2665550 commit a80f24a

File tree

8 files changed

+219
-89
lines changed

8 files changed

+219
-89
lines changed

packages/gradle/package.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@
4242
"dependencies": {
4343
"@nx/devkit": "workspace:*",
4444
"toml-eslint-parser": "^0.10.0",
45+
"tree-kill": "^1.2.2",
4546
"tslib": "catalog:typescript"
4647
},
4748
"devDependencies": {

packages/gradle/src/plugin/utils/get-project-graph-from-gradle-plugin.ts

Lines changed: 38 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,6 @@ let projectGraphReportCachePath: string = join(
7474
workspaceDataDirectory,
7575
'gradle-nodes.hash'
7676
);
77-
7877
export function getCurrentProjectGraphReport(): ProjectGraphReport {
7978
if (!projectGraphReportCache) {
8079
throw new AggregateCreateNodesError(
@@ -131,32 +130,44 @@ export async function populateProjectGraph(
131130
'gradleProjectGraphReport:start'
132131
);
133132

134-
const projectGraphLines = await gradlewFiles.reduce(
135-
async (
136-
projectGraphLines: Promise<string[]>,
137-
gradlewFile: string
138-
): Promise<string[]> => {
139-
const getNxProjectGraphLinesStart = performance.mark(
140-
`${gradlewFile}GetNxProjectGraphLines:start`
141-
);
142-
const allLines = await projectGraphLines;
143-
const currentLines = await getNxProjectGraphLines(
144-
gradlewFile,
145-
gradleConfigHash,
146-
normalizedOptions
147-
);
148-
const getNxProjectGraphLinesEnd = performance.mark(
149-
`${gradlewFile}GetNxProjectGraphLines:end`
150-
);
151-
performance.measure(
152-
`${gradlewFile}GetNxProjectGraphLines`,
153-
getNxProjectGraphLinesStart.name,
154-
getNxProjectGraphLinesEnd.name
155-
);
156-
return [...allLines, ...currentLines];
157-
},
158-
Promise.resolve([])
159-
);
133+
let projectGraphLines: string[];
134+
try {
135+
projectGraphLines = await gradlewFiles.reduce(
136+
async (
137+
projectGraphLines: Promise<string[]>,
138+
gradlewFile: string
139+
): Promise<string[]> => {
140+
const getNxProjectGraphLinesStart = performance.mark(
141+
`${gradlewFile}GetNxProjectGraphLines:start`
142+
);
143+
const allLines = await projectGraphLines;
144+
const currentLines = await getNxProjectGraphLines(
145+
gradlewFile,
146+
gradleConfigHash,
147+
normalizedOptions
148+
);
149+
const getNxProjectGraphLinesEnd = performance.mark(
150+
`${gradlewFile}GetNxProjectGraphLines:end`
151+
);
152+
performance.measure(
153+
`${gradlewFile}GetNxProjectGraphLines`,
154+
getNxProjectGraphLinesStart.name,
155+
getNxProjectGraphLinesEnd.name
156+
);
157+
return [...allLines, ...currentLines];
158+
},
159+
Promise.resolve([])
160+
);
161+
} catch (e) {
162+
if (
163+
e instanceof Error &&
164+
e.message === 'Gradle project graph generation was cancelled'
165+
) {
166+
// Cancelled by a newer populateProjectGraph call — silently return
167+
return;
168+
}
169+
throw e;
170+
}
160171

161172
const gradleProjectGraphReportEnd = performance.mark(
162173
'gradleProjectGraphReport:end'

packages/gradle/src/plugin/utils/get-project-graph-lines.ts

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,19 @@ import { GradlePluginOptions } from './gradle-plugin-options';
55

66
const DEFAULT_GRAPH_TIMEOUT_SECONDS = isCI() ? 600 : 120;
77

8+
let currentAbortController: AbortController | undefined;
9+
10+
/**
11+
* Cancel any in-flight Gradle project graph process.
12+
* Safe to call even if nothing is running.
13+
*/
14+
export function cancelPendingProjectGraphRequest(): void {
15+
if (currentAbortController) {
16+
currentAbortController.abort('cancelled');
17+
currentAbortController = undefined;
18+
}
19+
}
20+
821
export function getGraphTimeoutMs(): number {
922
const envTimeout = process.env.NX_GRADLE_PROJECT_GRAPH_TIMEOUT;
1023
if (envTimeout) {
@@ -30,7 +43,12 @@ export async function getNxProjectGraphLines(
3043

3144
const timeoutMs = getGraphTimeoutMs();
3245
const timeoutSeconds = timeoutMs / 1000;
46+
47+
// Cancel any in-flight Gradle process from a previous call, then create a fresh controller.
48+
cancelPendingProjectGraphRequest();
3349
const controller = new AbortController();
50+
currentAbortController = controller;
51+
const signal = controller.signal;
3452
const timer = setTimeout(() => controller.abort(), timeoutMs);
3553

3654
try {
@@ -48,10 +66,14 @@ export async function getNxProjectGraphLines(
4866
`-PworkspaceRoot=${workspaceRoot}`,
4967
process.env.NX_GRADLE_VERBOSE_LOGGING ? '--info' : '',
5068
],
51-
{ signal: controller.signal }
69+
{ signal }
5270
);
5371
} catch (e: any) {
54-
if (controller.signal.aborted) {
72+
// Cancelled by a newer populateProjectGraph call — let the caller handle it
73+
if (signal.reason === 'cancelled') {
74+
throw new Error('Gradle project graph generation was cancelled');
75+
}
76+
if (signal.aborted) {
5577
throw new AggregateCreateNodesError(
5678
[
5779
[

packages/gradle/src/utils/exec-gradle.ts

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import { dirname, join, isAbsolute } from 'node:path';
99
import { LARGE_BUFFER } from 'nx/src/executors/run-commands/run-commands.impl';
1010
import { GradlePluginOptions } from '../plugin/utils/gradle-plugin-options';
1111
import { signalToCode } from 'nx/src/utils/exit-codes';
12+
import treeKill from 'tree-kill';
1213

1314
export const fileSeparator = process.platform.startsWith('win')
1415
? 'file:///'
@@ -38,6 +39,10 @@ export function execGradleAsync(
3839
args: ReadonlyArray<string>,
3940
execOptions: ExecFileOptions = {}
4041
): Promise<Buffer> {
42+
// Extract signal so we can handle cancellation with tree-kill
43+
// instead of Node's default which only kills the immediate child.
44+
const { signal, ...restOptions } = execOptions;
45+
4146
return new Promise<Buffer>((res, rej: (stdout: Buffer) => void) => {
4247
const cp = execFile(
4348
gradleBinaryPath,
@@ -48,11 +53,20 @@ export function execGradleAsync(
4853
windowsHide: true,
4954
env: process.env,
5055
maxBuffer: LARGE_BUFFER,
51-
...execOptions,
56+
...restOptions,
5257
},
5358
undefined
5459
);
5560

61+
// Use tree-kill on abort to kill the entire process tree
62+
// (cmd.exe → gradlew.bat → java.exe), not just the shell.
63+
const onAbort = () => {
64+
if (cp.pid) {
65+
treeKill(cp.pid);
66+
}
67+
};
68+
signal?.addEventListener('abort', onAbort, { once: true });
69+
5670
let stdout = Buffer.from('');
5771
cp.stdout?.on('data', (data) => {
5872
stdout += data;
@@ -61,8 +75,9 @@ export function execGradleAsync(
6175
stdout += data;
6276
});
6377

64-
cp.on('exit', (code, signal) => {
65-
if (code === null) code = signalToCode(signal);
78+
cp.on('exit', (code, s) => {
79+
signal?.removeEventListener('abort', onAbort);
80+
if (code === null) code = signalToCode(s);
6681
if (code === 0) {
6782
res(stdout);
6883
} else {

packages/maven/package.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@
4949
"dependencies": {
5050
"@nx/devkit": "workspace:*",
5151
"@xmldom/xmldom": "^0.8.10",
52+
"tree-kill": "^1.2.2",
5253
"tslib": "catalog:typescript"
5354
},
5455
"devDependencies": {

packages/maven/src/plugins/maven-analyzer.ts

Lines changed: 116 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,37 @@ import { join } from 'path';
22
import { existsSync } from 'fs';
33
import { spawn } from 'child_process';
44
import { logger, readJsonFile } from '@nx/devkit';
5+
import { isCI } from 'nx/src/devkit-internals';
56
import { workspaceDataDirectory } from 'nx/src/utils/cache-directory';
67
import { MavenAnalysisData, MavenPluginOptions } from './types';
78
import { detectMavenExecutable } from '../utils/detect-maven-executable';
9+
import treeKill from 'tree-kill';
10+
11+
const DEFAULT_ANALYSIS_TIMEOUT_SECONDS = isCI() ? 600 : 120;
12+
13+
let currentAbortController: AbortController | undefined;
14+
15+
/**
16+
* Cancel any in-flight Maven analysis process.
17+
* Safe to call even if nothing is running.
18+
*/
19+
export function cancelPendingMavenAnalysis(): void {
20+
if (currentAbortController) {
21+
currentAbortController.abort('cancelled');
22+
currentAbortController = undefined;
23+
}
24+
}
25+
26+
function getAnalysisTimeoutMs(): number {
27+
const envTimeout = process.env.NX_MAVEN_ANALYSIS_TIMEOUT;
28+
if (envTimeout) {
29+
const parsed = Number(envTimeout);
30+
if (!Number.isNaN(parsed) && parsed > 0) {
31+
return parsed * 1000;
32+
}
33+
}
34+
return DEFAULT_ANALYSIS_TIMEOUT_SECONDS * 1000;
35+
}
836

937
/**
1038
* Run Maven analysis using our Kotlin analyzer plugin
@@ -60,67 +88,102 @@ export async function runMavenAnalysis(
6088
);
6189
}
6290

91+
// Cancel any in-flight Maven process from a previous call, then create a fresh controller.
92+
cancelPendingMavenAnalysis();
93+
const controller = new AbortController();
94+
currentAbortController = controller;
95+
const signal = controller.signal;
96+
const timeoutMs = getAnalysisTimeoutMs();
97+
const timeoutSeconds = timeoutMs / 1000;
98+
const timer = setTimeout(() => controller.abort(), timeoutMs);
99+
63100
// Run Maven plugin
64101
logger.verbose(`[Maven Analyzer] Spawning Maven process...`);
65-
await new Promise<void>((resolve, reject) => {
66-
const child = spawn(mavenExecutable, mavenArgs, {
67-
cwd: workspaceRoot,
68-
windowsHide: true,
69-
shell: true,
70-
stdio: 'pipe', // Always use pipe so we can control output
71-
});
72-
73-
logger.verbose(`[Maven Analyzer] Process spawned with PID: ${child.pid}`);
74-
75-
let stdout = '';
76-
let stderr = '';
77-
78-
// In verbose mode, forward output to console in real-time
79-
if (isVerbose) {
80-
child.stdout?.on('data', (data) => {
81-
const text = data.toString();
82-
stdout += text;
83-
process.stdout.write(text); // Forward to stdout
84-
});
85-
child.stderr?.on('data', (data) => {
86-
const text = data.toString();
87-
stderr += text;
88-
process.stderr.write(text); // Forward to stderr
89-
});
90-
} else {
91-
child.stdout?.on('data', (data) => {
92-
const text = data.toString();
93-
stdout += text;
94-
logger.verbose(`[Maven Analyzer] Stdout chunk: ${text.trim()}`);
102+
try {
103+
await new Promise<void>((resolve, reject) => {
104+
const child = spawn(mavenExecutable, mavenArgs, {
105+
cwd: workspaceRoot,
106+
windowsHide: true,
107+
shell: true,
108+
stdio: 'pipe', // Always use pipe so we can control output
95109
});
96-
child.stderr?.on('data', (data) => {
97-
const text = data.toString();
98-
stderr += text;
99-
logger.verbose(`[Maven Analyzer] Stderr chunk: ${text.trim()}`);
100-
});
101-
}
102110

103-
child.on('close', (code) => {
104-
logger.verbose(`[Maven Analyzer] Process closed with code: ${code}`);
105-
if (code === 0) {
106-
logger.verbose(
107-
`[Maven Analyzer] Maven analysis completed successfully`
108-
);
109-
resolve();
111+
// Use tree-kill on abort to kill the entire process tree
112+
const onAbort = () => {
113+
if (child.pid) {
114+
treeKill(child.pid);
115+
}
116+
};
117+
signal.addEventListener('abort', onAbort, { once: true });
118+
119+
logger.verbose(`[Maven Analyzer] Process spawned with PID: ${child.pid}`);
120+
121+
let stdout = '';
122+
let stderr = '';
123+
124+
// In verbose mode, forward output to console in real-time
125+
if (isVerbose) {
126+
child.stdout?.on('data', (data) => {
127+
const text = data.toString();
128+
stdout += text;
129+
process.stdout.write(text); // Forward to stdout
130+
});
131+
child.stderr?.on('data', (data) => {
132+
const text = data.toString();
133+
stderr += text;
134+
process.stderr.write(text); // Forward to stderr
135+
});
110136
} else {
111-
let errorMsg = `Maven analysis failed with code ${code}`;
112-
if (stderr) errorMsg += `\nStderr: ${stderr}`;
113-
if (stdout && !isVerbose) errorMsg += `\nStdout: ${stdout}`;
114-
console.error(`[Maven Analyzer] Error: ${errorMsg}`);
115-
reject(new Error(errorMsg));
137+
child.stdout?.on('data', (data) => {
138+
const text = data.toString();
139+
stdout += text;
140+
logger.verbose(`[Maven Analyzer] Stdout chunk: ${text.trim()}`);
141+
});
142+
child.stderr?.on('data', (data) => {
143+
const text = data.toString();
144+
stderr += text;
145+
logger.verbose(`[Maven Analyzer] Stderr chunk: ${text.trim()}`);
146+
});
116147
}
117-
});
118148

119-
child.on('error', (error) => {
120-
console.error(`[Maven Analyzer] Process error: ${error.message}`);
121-
reject(new Error(`Failed to spawn Maven process: ${error.message}`));
149+
child.on('close', (code) => {
150+
signal.removeEventListener('abort', onAbort);
151+
logger.verbose(`[Maven Analyzer] Process closed with code: ${code}`);
152+
if (code === 0) {
153+
logger.verbose(
154+
`[Maven Analyzer] Maven analysis completed successfully`
155+
);
156+
resolve();
157+
} else {
158+
let errorMsg = `Maven analysis failed with code ${code}`;
159+
if (stderr) errorMsg += `\nStderr: ${stderr}`;
160+
if (stdout && !isVerbose) errorMsg += `\nStdout: ${stdout}`;
161+
console.error(`[Maven Analyzer] Error: ${errorMsg}`);
162+
reject(new Error(errorMsg));
163+
}
164+
});
165+
166+
child.on('error', (error) => {
167+
signal.removeEventListener('abort', onAbort);
168+
console.error(`[Maven Analyzer] Process error: ${error.message}`);
169+
reject(new Error(`Failed to spawn Maven process: ${error.message}`));
170+
});
122171
});
123-
});
172+
} catch (e: any) {
173+
if (signal.reason === 'cancelled') {
174+
throw new Error('Maven analysis was cancelled');
175+
}
176+
if (signal.aborted) {
177+
throw new Error(
178+
`Maven analysis timed out after ${timeoutSeconds} ${timeoutSeconds === 1 ? 'second' : 'seconds'}.\n` +
179+
` 1. If the issue persists, set the environment variable NX_MAVEN_ANALYSIS_TIMEOUT to a higher value (in seconds) to increase the timeout.\n` +
180+
` 2. If the issue still persists, set NX_MAVEN_DISABLE=true to disable the Maven plugin entirely.`
181+
);
182+
}
183+
throw e;
184+
} finally {
185+
clearTimeout(timer);
186+
}
124187

125188
// Read and parse the JSON output
126189
logger.verbose(`[Maven Analyzer] Checking for output file: ${outputFile}`);

0 commit comments

Comments
 (0)