Nilla loader auto-detection API is flawed and need to be reconsidered

[liferooter]

The current state

Nilla has a great loaders system that allows adding external dependencies of the project as inputs and automatically importing their Nix-enabled contents when possible, automatically detecting the layout and protocol of the dependency. It's done with loaders, which represent different ways to load a dependency from its source directory. There are some built-in loaders, but the user is free to define their own loaders. Custom loaders also support auto-detection via defining a check function.

The design looks great, but there's a serious flaw in it. The problem is what happens if the project supports many loaders, which is pretty common in the Nix world and probably will become more and more common if alternative non-flake solutions like Nilla are rising.

The current solution checks the project for all custom loaders. If none of them have worked, it loads it with built-in loaders, which are guaranteed to work because there's the last resort called a raw loader, which just passes the path as is without loading anything. If some of the custom loaders work, Nilla takes the first one, showing a warning if more than one is suitable.

Although it's always possible to manually set the loader, the idea of auto-detection is to avoid it as much as possible.

Problems

There are many problems with this design:

• It assumes that all built-in loaders are kind of a fallback for custom loaders, which is not always the case. For example, imagine the world where there's a project management solution for Nix called Snowheap (the name is made up, there's no such project known to me). Imagine that it's not as cool as Nilla, but some people still use it. Imagine that some people make projects with support for both Nilla and Snowheap. Some of them only support Snowheap, not Nilla. Imagine the projects which depend on both projects with Nilla AND Snowheap support and Snowheap-only projects. Imagine that Nilla has no native support for Snowheap projects, so the user somehow gets a custom Snowheap loader. It's a custom loader, so it automatically gets a higher priority in the loading process than the Nilla loader, which is not cool because, as I said, Nilla is much cooler than Snowheap and should be used in cases when both are available.
• When multiple options are available, it chooses a random one. When multiple loaders are possible, Nilla just gets the first one. The order of loaders is the order of keys in the attribute set of loaders, which is alphabetical. It has nothing to do with some user-defined priority and suitability. For means of use, it's a predictable, but still random order. It still can be played with via weird naming like 53-flake or 32-nilla to give them priority like 53 or 32, but I hope that this is not what we're going to do.
• It requires too many special cases. Built-in loaders are not detected via the check function. They're detected via custom code with case analysis. Although I agree that there are loaders that are special enough to differ from the concept of custom loaders, I think there's only one such loader. It's a raw loader, which is supposed to be the fallback used when no other loaders are suitable. Loaders like nixpkgs, flake, legacy, or nilla has nothing more special to require some special place in the loading system and be called built-in loaders than any other loaders that will appear in the future. nilla loader doesn't have more reasons to be a built-in loader than imaginary showheap loader from the first clause of my problem list. The necessity of case analysis for them just shows that the current design of loader auto-detection is flawed and cannot solve the problem of choosing between many possible options. The design should be good enough to enable these built-in loaders to become non-special, just some pre-defined custom loaders.

The problem can be reworded in short: the current design has no concept of loader priority, so it cannot handle well cases with many suitable loaders. Everything else is just a consequence of this problem.

Some thoughts about possible design

We need some means of setting priorities for loaders, which is about defining which one to choose when many are available. Unfortunately, it's not a trivial problem to solve.

In Nilla loaders generally come from the following three sources:

• Nilla itself. Some loaders are defined in Nilla, so they're global and well-known.
• External dependencies. Loaders can be imported from some external sources. Such loaders are not always well-known, they may not know about each other and so may not be able to define priorities between themselves, but sometimes they still can.
• The project code. Sometimes the user can define some brand new loaders just inside the project. Such loaders are defined by the user, so they can know about other loaders used inside the project and can set priorities between them and themselves.

The priority preferences also can come from two general sources: they can be defined with the loaders (e.g. nilla loader can definitely state that it has higher priority than the flake loader) or as part of the project, which reflects the preferences of the user (e.g. the user hates Flakes and so will prefer using legacy loader over flake loader).

It's obvious that in our case we cannot use a kind of global priority scale, like numeric priority, because we don't have a knowledge of all possible loaders and so we cannot order them from the least prioritized to the most prioritized. It still can be done if priorities are defined on the side of the user, but in that case we'll lose the ability to define priorities with the loader itself.

We need to define some kind of relation between the loaders. The relation that states "this loader is less/more preferred than that". In the language of order theory, we need some partial order on the loaders to be defined, so the problem of choosing between many possible loaders could be reduced to the problem of finding the greatest element of the set, which is pretty trivial. A partial order can be viewed as a directed acyclic graph, which is, fortunately, a pretty well-known abstraction in the Nix world and also well-supported by AuxLib, which is the standard library behind Nilla.

So a possible solution can be adding an option of DAG-type, which will define priority relations between different loaders. When many loaders are suitable, Nilla should try to find the greatest of the suitable loaders in means of this DAG. If there are no greatest elements in the set of suitable loaders, Nilla should fail with an error. Also, Nilla should validate the DAG and have some means to override it in a destructive way, which means removing or redirecting some of its edges.

[liferooter]

Just checked DAGs in AuxLib. Although there are types and utility functions for working with them, their are not well-supported (e.g. they're not checked and cannot be overridden as DAGs, not just like AttrSets) :(

[jakehamilton]

Thank you for the well thought out and written post! I will need to think about this a bit. Input loading has some issues that I have been trying to figure out overall, but haven't quite gotten anywhere.

[liferooter]

By the way, looks like the most hard problem to solve with the input system is inability to define option-dependent includes, which implies inability to include anything from inputs, which is makes impossible to write Nilla extensions which can be loaded with Nilla loaders.

[liferooter]

The problem is fundamental for Nix modules system. I wonder if it's even possible to rework modules in a way that will make this thing possible.