Skip to content

[vropkg] (#839) Fix 'Too many files open' Error on Windows for Package with many Elements#1066

Open
akantchev wants to merge 9 commits intovmware:mainfrom
akantchev:fix/839-too-many-files-win
Open

[vropkg] (#839) Fix 'Too many files open' Error on Windows for Package with many Elements#1066
akantchev wants to merge 9 commits intovmware:mainfrom
akantchev:fix/839-too-many-files-win

Conversation

@akantchev
Copy link
Copy Markdown
Contributor

@akantchev akantchev commented Mar 18, 2026

Description

The archive() function in packaging.ts was creating archiver instances without properly managing the output stream lifecycle. When callers used archive.finalize(), the Promise returned by finalize() didn't wait for the stream to close, leaving file handles open.

Solution:

AddedfinalizeArchive() Promise settling and listener usage (packaging.ts)
• Added settled flag with settleOnce() guard to prevent multiple Promise settlements
• Changed all on() listeners to once() to prevent listener accumulation
• Added comprehensive error handling for both output stream and archiver errors
• All error paths now provide descriptive messages with output path for debugging

Replaced sync FS with true async operations
• tree.ts: Replaced Promise.resolve().then(() => fs.copySync(...)) with fs.copy(...)
• util.ts: Replaced Promise.resolve().then(() => fs.copySync(...)) with fs.copy(...)
• Both files now use non-blocking async I/O operations

Eliminated all unsafe direct .finalize() calls
• tree.ts: Replaced manual Promise-based archiver code with p.archive() + p.finalizeArchive()
• util.ts: Replaced manual Promise-based archiver code with p.archive() + p.finalizeArchive()
• flat.ts: Already using p.finalizeArchive() correctly (no changes needed)
• All archiver operations now use the centralized, safe finalizeArchive() function

Benefits:
1. Prevents EMFILE errors - Proper async/await ensures file handles are closed before proceeding
2. No unhandled exceptions - Event handlers don't throw, errors propagate through Promises
3. No memory leaks - Using once() prevents listener accumulation
4. No race conditions - Guard against multiple Promise settlements
5. True async I/O - No blocking sync operations disguised as Promises
6. Consistent error handling - All archive operations use the same safe pattern### Checklist

  • I have added relevant error handling and logging messages to help troubleshooting
  • I have added necessary documentation, relevant usage information (if applicable)
  • I have updated the PR title with affected component, related issue number and a short summary of the changes introduced
  • I have my changes rebased and squashed to the minimal number of relevant commits. Notice: don't squash all commits
  • I have added a descriptive commit message with a short title, including a Fixed #XXX - or Closed #XXX - prefix to auto-close the issue

Testing

Manual testing is required.

Related issues and PRs

#839

@akantchev akantchev requested a review from a team as a code owner March 18, 2026 15:05
@akantchev akantchev requested a review from bcpmihail March 23, 2026 15:06
@akantchev
Copy link
Copy Markdown
Contributor Author

akantchev commented Mar 23, 2026

Please add the following labels to this PR:

kind/bug

area/vropkg

lang/typescript

version/patch

@VenelinBakalov VenelinBakalov added kind/bug Something isn't working lang/typescript Related to typescript code area/vropkg Relates to `vropkg` module version/patch The change is a non-breaking bugfix labels Mar 25, 2026
@VenelinBakalov VenelinBakalov changed the title [vropkg] Fix 'Too many files open' Error on Windows for Package with many Elements [vropkg] (#839) Fix 'Too many files open' Error on Windows for Package with many Elements Mar 25, 2026
@joroaf joroaf self-requested a review March 25, 2026 13:17
Copy link
Copy Markdown
Contributor

@joroaf joroaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Required changes (blocking) to address before approval:

  1. Fix finalizeArchive() Promise settling / listener usage

    • Ensure the Promise settles exactly once (guard against both finalize() reject + later stream close, etc.).
    • Use once('close' | 'error') instead of on(...) to avoid accumulating listeners.
    • Capture/reject on both output stream errors and archiver errors (not just rely on thrown errors elsewhere).
  2. Stop throwing inside stream/event handlers in archive()

    • In packaging.ts, don’t throw from output.on('error') / instance.on('error').
    • Log there, but let finalizeArchive() be the mechanism that rejects and propagates errors to callers.
  3. Replace “sync FS inside Promise” with true async

    • In tree.ts and serialize/util.ts, replace fs.copySync(...) wrapped in Promise.resolve().then(...) with await fs.copy(...) / fs.copy(...) (or fs.copyFile(...)), so we don’t block and we keep handle usage predictable.
  4. Ensure no remaining unsafe direct .finalize() calls on p.archive()

    • Repo-wide check: any code doing p.archive(...).finalize() should be switched to p.finalizeArchive(arch) to avoid reintroducing the EMFILE issue.

… removed throw within archive() method, added true async operations in file io operations within a promise, updated usage of the finalizeArchive() method\nSigned-off-by:akantchev
@akantchev
Copy link
Copy Markdown
Contributor Author

Hi @joroaf, I've addressed the changes you've requested. Thanks for the detailed comments.

@akantchev akantchev requested a review from joroaf March 25, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/vropkg Relates to `vropkg` module kind/bug Something isn't working lang/typescript Related to typescript code version/patch The change is a non-breaking bugfix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants