Bundling a multithreading program, without public URL

[Andonvr]

I have a problem using multithreading and packaging the resulting WASM in a reusable node package.
But let's start with what works so far:

I have a C++ program that I compile with emscripten, utilizing a CMakeLists.txt. I then want to use the resulting WASM in an npm package, for me to later publish.
These are my compiler flags:

            -O0\
            --bind\
            --no-entry\
            -s ALLOW_MEMORY_GROWTH=1\
            -s MALLOC=emmalloc\
            -s EXPORT_ES6=1\
            -s MODULARIZE=1\
            -s EXPORT_NAME=${PROJECT_NAME}\
            -s SINGLE_FILE=1\
            -s ENVIRONMENT=web,worker,shell\
            --emit-tsd=${BUILD_FOLDER}/${PROJECT_NAME}.d.ts\

It outputs a .js and a .d.ts file, and I can use it like this:

import MainModule from "../wasm/wasmbuild/output/hello_world_web"

export const testHelloWorld = async () => {
  const myMainModule = await MainModule()
  return myMainModule._main(0, 0)
}

This works well!
However, now I want to add multithreading. I adjust my code, and add the following compiler flags to my CMakeLists.txt:

            -pthread\
            -s PTHREAD_POOL_SIZE=16\

When running the same way I do above, I get these errors in my console:

I have the strong suspicion, that it is because of this block in the generated .js file:

 allocateUnusedWorker() {
    var worker;
    var workerOptions = {
      "type": "module",
      // This is the way that we signal to the Web Worker that it is hosting
      // a pthread.
      "name": "em-pthread"
    };
    // If we're using module output, use bundler-friendly pattern.
    // We need to generate the URL with import.meta.url as the base URL of the JS file
    // instead of just using new URL(import.meta.url) because bundler's only recognize
    // the first case in their bundling step. The latter ends up producing an invalid
    // URL to import from the server (e.g., for webpack the file:// path).
    worker = new Worker(new URL("hello_world_web.js", import.meta.url), workerOptions); // <------ THIS
    PThread.unusedWorkers.push(worker);
  },

As I said, I use this in a package, I don't host the hello_world_web.js in any specific place. How would "hello_world_web.js" actually be found? Is there any specific way of handling this?

I already gave "locateFile" a go, like this:

const myMainModule = await MainModule({
  locateFile: (path: string) => {
    console.log("Trying to locate:")
    return path
  },
})

But nothing was logged, and since I don't see any "locateFile" referenced in the codeblock I mentioned, I didn't pursue this any further.

I also looked into webpack, and whether it would solve my issues. But that feels like overkill for what I want to achieve, and I have no clue if that is even the right path. So I didn't get further into that yet either.

Any help would be highly appreciated! Here's a simple repo to try things out: https://github.com/Andonvr/emscripten-threading-example.
Cheers.

[sbc100]

This issue has come up a few times recently.

In you case does replacing new Worker(new URL("hello_world_web.js", import.meta.url), workerOptions) with new Worker(import.meta.url, workerOptions) solve the issue?

I've been wanting to switch to that form for a while now but have been blocked on support in bundlers for this simpler syntax.

[sbc100]

See #23769, which was fixed in #23890.

Also see #22521 and webpack/webpack#12638

[Andonvr]

Thanks for the quick reply!
The error I now get is:
Security Error: Content at http://localhost:3000/ may not load data from file:///ROOT/hello_world_web/src/wasm/wasmbuild/output/hello_world_web.js.

But since you say this came up recently, and something related was fixed recently, I wanna point out that I'm still using 3.1.64, as that is the latest version available as a nix package.
If it's related to the version, I think I will work around it for a while, and revisit this once I actually need to package all my stuff :) Maybe by then the latest version is available, or I get around to setting it up myself.

[sbc100]

Thanks for the quick reply! The error I now get is: Security Error: Content at http://localhost:3000/ may not load data from file:///ROOT/hello_world_web/src/wasm/wasmbuild/output/hello_world_web.js.

But since you say this came up recently, and something related was fixed recently, I wanna point out that I'm still using 3.1.64, as that is the latest version available as a nix package. If it's related to the version, I think I will work around it for a while, and revisit this once I actually need to package all my stuff :) Maybe by then the latest version is available, or I get around to setting it up myself.

Are you using a bundler? If so which one? Presumably file:///ROOT/hello_world_web/src/wasm/wasmbuild/output/hello_world_web.js comes from the bundling process?

[Andonvr]

I'm not using any bundler :)
I set up a quick example repo if you wanna have a closer look: https://github.com/Andonvr/emscripten-threading-example

Structure:

• myPackage : C++ code is compiled to WASM using emscripten and a CMakeLists.txt, to be used as a package in other projects.
• demo : An example Nextjs project, simply using the package mentioned above.

(Note that this setup totally works, if I don't use multithreading, which one can test by removing the flags in the CMakeLists.txt. Obviously not an option for my real world usecase)

[sbc100]

Where does the file:///ROOT/hello_world_web/src/wasm/wasmbuild/output/hello_world_web.js string come from there? What does the new Worker() call look like in your final output?

[sbc100]

• demo : An example Nextjs project, simply using the package mentioned above.

Are you sure nextjs isn't doing some kind of bundling?

[Andonvr]

Are you sure nextjs isn't doing some kind of bundling?

Ohhh yeah that's a good point! After running next build, this is what's built:

I have no clue where the string comes from tho. If built by NextJS, shouldn't it at least bundle all dependencies in the resulting build-folder? I wonder if this has to do with pnpm-workspaces or something. The dependencies being weirdly linked. But I am so out of my depth in this regard.

[Andonvr]

I found the issue!! Emscripten is not at fault, and it's not even some bundling mishap. It's just Next.js.

TL;DR

This had nothing to do with any new URL(...) stuff.
In Next.JS, you have to specify the CORS headers not just for /, but also for /_next/static/:path*!

See demo/next.config.js in my Repo for the fix.

How I found it

The resulting wasm_example.js has this block:

worker.onerror = (e) => {
  var message = "worker sent an error!"
  if (worker.pthread_ptr) {
    message = `Pthread ${ptrToString(worker.pthread_ptr)} sent an error!`
  }
  err(`${message} ${e.filename}:${e.lineno}: ${e.message}`)
  throw e
}

The log prints worker sent an error! undefined:undefined: undefined. Why is every field of e undefined? I then logged the whole e, and it just said {"isTrusted":true}.
Google told me that this is a script error, and that it most likely had to do with CORS.
Of course, the network tab confirmed, that the request to /_next/static/media/wasm_example.0e243b49.js was blocked by CORS.
Adding the CORS headers fixed the issue.

Conclusion

It's always CORS.