High level design documentation?

[madsmtm]

I'm currently reading the excellent documentation on rustc's query system, and wondered what your plan for Wild is regarding achieving incremental linking? Do you have some documentation or design drafts, or is this all still far out in the future?

[davidlattimore]

I am getting closer to the point where I'll probably start looking at implementing incremental compilation soon. I haven't decided on a concrete plan as yet. I do have two possible ways that I'm considering.

One is to keep the linker in memory. Effectively have a cargo-watch style command, that runs cargo, wraps rustc, then watches for changes to any source files. When it sees a source file change, reruns the relevant rustc command directly (without invoking cargo), then takes the new outputs from rustc and integrates them into the existing in-memory link.

The other approach is to persist bits of state to disk, then when invoked, load that state, see what input files have changed and reprocess those. I wouldn't want to use any serialisation for persisting state, since that would be too slow. Rather I'd just make the on-disk format be the same as the in-memory format, so that the files can just be mmapped. That would require that the in-memory format not use any references. Most of Wild's in-memory structures already don't use any references. The biggest exception is the symbol name map, which borrows string slices from the original input files. So that would need to change. The tricky part is changing it in a way that doesn't kill performance. I suspect I should continue to reference the original input files, but just do so in a way that doesn't use references.

[madsmtm]

Interesting, thanks for the answer!

Just curious now, would there be any idea in merging the linker into a fork of rustc itself, probably with cranelift too, so that all the parts of the compiler can share the query system, and we can have "truly" incremental compilation? But that'd probably be too big of an undertaking.

[davidlattimore]

I added some high level design docs in #475.

The comments on this issue are mostly asking about incremental linking, for which we've got #184.

So, I think we can call this done. Although, I just noticed that I never replied to your question about integrating the linker into rustc. That is something I'm considering experimenting with at some point. I wouldn't expect it to be too hard. The trickiest part is that usually rustc doesn't call the linker directly, but instead calls the C compiler, which then invokes the linker. So if the linker is built into rustc, we'd need some other way to locate the C runtime objects - that bit could get messy.

[madsmtm]

Cool, thanks again!

The trickiest part is that usually rustc doesn't call the linker directly, but instead calls the C compiler, which then invokes the linker. So if the linker is built into rustc, we'd need some other way to locate the C runtime objects - that bit could get messy.

See also rust-lang/rust#11937. I think nowadays, it's fairly well supported (e.g. -Clinker=ld usually just works).

[mati865]

I have been thinking about implementing a new bin+lib crate that would take care of guessing the paths and transforming the arguments.
I'll do such an experiment once wild reaches milestones that are important for me.