Possible to run server side blocks on a cluster?

[voxpelli]

This proposal seems quite similar to the Swift Distributed Actor one: https://github.com/apple/swift-distributed-actors

It though, rather than being geared towards just something like WebWorkers, is also geared towards executing the distributed actors outside of the local machine and across a cluster of machines.

From the browser perspective executing stuff elsewhere but locally wouldn’t make sense, but from a server side perspective (Node.js, Deno and such) it could make equal sense to go beyond the local I machine as it does for Swift to do so.

The challenge that Web Workers and such require separate files to be loaded is also something that somewhat exists in the serverless function space (AWS Lambda and such) and could benefit as much from being able to define such functions online and then on execution time schedule and run those on a defined cluster.

I do realize that the specific logic for how to run something on a cluster would be out of scope for this proposal (just like it is out scope of the Swift Distributed Actor one) and would require a transport like https://github.com/apple/swift-distributed-actors to be built to do the scheduling of module blocks across a cluster.

What I do am wondering though is whether this proposal is written in a way that would facilitate such a mechanism and whether it is in line with the intention of this proposal or whether it should be seen as something module blocks would not be intended to be used for?

Original Twitter thread where @surma mentioned this proposal to me: https://twitter.com/voxpelli/status/1459127412860542976?s=21

[acutmore]

The challenge that Web Workers and such require separate files to be loaded

Nowhere near as clean and developer-tooling friendly as module-blocks but it is possible, in some situations, to run a worker without a separate file:

new Worker(`data:text/javascript,console.log('hi')`)
// 'hi' logged from worker thread

It's not a perfect analogy by any means, but imagine module-blocks as a special type of string that can only contain valid JavaScript. Now we can be sure when sending this 'string' to another realm/thread/process/machine to run - it was at least syntactically valid. So I don't think they necessarily provide a way to run code in a way that we couldn't before, but they do make for a much more robust and friendly experience.

[voxpelli]

imagine module-blocks as a special type of string that can only contain valid JavaScript

That + the fact that its javascript is never allowed to reference anything in its outside scope, which eg. the content of a toString() on a function would? And thus making it much more likely for such a function to be possible to launch in a totally separate context, right?

Also:

There is a Use with workers and such in the README, all being client side uses, would love for a non-client use to be documented as well, to show if the spec is meant to be used for use cases outside of those.

[Jamesernator]

There is a Use with workers and such in the README, all being client side uses, would love for a non-client use to be documented as well, to show if the spec is meant to be used for use cases outside of those.

It can't really help out of the box as even with .toString() there is no way to know what a relative specifier (i.e. import "./some/relative/url.js") would refer to when stringified. Like if you create the module block yourself it's fine as you can use import.meta.url but if you got a module block returned from a function, the actual module that this block came from might be completely unknown.

Although it does beg the question, should we have a meta object on module blocks that store this information? If we did you could just serialize both the module block (using .toString()) and the .meta object to know the base url i.e.:

const moduleBlock = module {
    export default class Foo {}
}

moduleBlock.meta.url; // whatever the import.meta.url of the module block would be

Whether or not people want to expose that information on all module blocks is another question, sometimes it's arguably a refactoring hazard or with ShadowRealms potentially a security hazard.

[surma]

I think anything cross-engine is out of scope for this proposal for now. But maybe a future proposal can bring this capability to JS.