Ignore certain files (i.e. all-packages.nix)

[Atemu]

So my Emacs has been hanging for a few seconds now because I hovered over callPackage in all-packages.nix.

Could nixd have the option to not enable LSP operations inside certain paths or past a certain file size?

Or is that something the editor's LSP client should implement? (joaotavora/eglot#1391)

This also affects nixd however as its memory usage balloons to many dozen GiB when I do this. When attempting to jump to a definition of some thing in all-packages always appears to add a few GiB to nixd' RAM utilisation. Is that expected?

[inclyc]

This also affects nixd however as its memory usage balloons to many dozen GiB when I do this.

AFAIK my profiling data blames 'Document Symbol' for memory usage, for this file.

When attempting to jump to a definition of some thing in all-packages always appears to add a few GiB to nixd' RAM utilisation. Is that expected?

Not expected though.

Could nixd have the option to not enable LSP operations inside certain paths or past a certain file size?

Seems to be workaournds to add such 'option'. I think the correct solution is to 'reduce' the mem usage for such file, and reduce the file itself.

AFAIK all-packages.nix has been proved to be bad practice in nixpkgs. All nix language servers have trouble with it!

[inclyc]

If you can't disable just highlighting (that appears to be the primary performance limitation), can you disable eglot in certain paths only (or past a certain size)?

Does it comes from 'with'? Nixd consider all variables coming from with are 'related', thus try to highlight it for you. I think the BIG data for this IO operation may be the experience blocker for u.

highlighting

Did you mean semantic tokens or documentHighligjts?

[Atemu]

AFAIK my profiling data blames 'Document Symbol' for memory usage, for this file.

Hmm, that's curious; shouldn't document symbols be a one and done deal?

I'd also expect them to be within an order of magnitude of the file they're based on but I'm observing a delta of 10000x.

Seems to be workaournds to add such 'option'.

Certainly. Such "dumb" escape hatches aren't necessarily a bad thing though.

reduce the file itself.

AFAIK all-packages.nix has been proved to be bad practice in nixpkgs.

Sure, that'd be great but the file will stay around for a little longer and will always be in the git history if you need to do historical digging.

There is also the case of pkgs/development/haskell-modules/hackage-packages.nix which is an order of magnitude larger than all-packages.nix and has no obvious alternative.

All nix language servers have trouble with it!

Sure, I wouldn't expect any tooling to handle a dense code file in the megabytes without a hitch. Hence my desire to selectively disable tooling past certain limits.

This is a tad more than just a little hitch though ;)

Does it comes from 'with'? Nixd consider all variables coming from with are 'related', thus try to highlight it for you. I think the BIG data for this IO operation may be the experience blocker for u.

In the case of all-packages, yes, but in hackage-packages it comes from function args. I was able to make it grow memory uncontrollably by jumping to definitions in hackage-packages; it's very easy to repro.
Though at one point I was not able to repro after restarting nixd. Jumps worked like a charm but I was not able to get into that state again.

I think the BIG data for this IO operation may be the experience blocker for u.

What exactly do you mean by "BIG data"? The records for all top-level attrs of pkgs?

Did you mean semantic tokens or documentHighligjts?

textDocument/documentHighlight

How exactly does this one work? Does the LSP send over every instance that should be highlighted or is that up to the editor to decide?

[inclyc]

Hmm, that's curious; shouldn't document symbols be a one and done deal?

No, it should be flushed right after any text edits. I observed approximately 6GB memory usage, because the symbols in that file is such a large scale.

That large memory consumption could be traced by "high" JSON encoding usage, because that request is much less inefficient in comparison to "semantic tokens".

Certainly. Such "dumb" escape hatches aren't necessarily a bad thing though.

Maybe I can introduce a new CLI flag which controls whether or not disable AST operations if it become apparently too large. (That's need to discussed).

How exactly does this one work? Does the LSP send over every instance that should be highlighted or is that up to the editor to decide?

Sadly LSP specification explicitly said it should be very "fuzzy" to highlight all "related". That's why I choose to send a full list of variables coming from same "with".

‘textDocument/documentHighlight’ and ‘textDocument/references’ separate requests since the first one is allowed to be more fuzzy.

https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_documentHighlight

[inclyc]

Hi @Atemu I think you may reduce the number of highlights in the response body? Would that solve your problem?

To test this, you can fork this project and change the behavior here:

nixd/nixd/lib/Controller/DocumentHighlight.cpp

Line 30 in 39ff96c

Highlights.emplace_back(DocumentHighlight{

Read https://github.com/nix-community/nixd/blob/main/nixd/docs/dev.md#hack-nixd-from-source if you don't know how to setup the env :)

[inclyc]

Profiling:
massif.out.1943282.txt

2.8/3.0 GB comes from large document symbol information.

[arijoon]

We have a very large file (over 120k lines) which is auto generated from our yarn lock. Causing a diff on it will make nixd keep consuming more memory. On my machine this reaches over 30GB and hangs the entire system eventually. Is there an option to exclude specific files by name?

> nixd --version
nixd, version: 2.5.1