Parallelize some phases of compilation between a crate and its dependency

[brson]

With a single driver building multiple crates we can start running more parts of the compilation process in parallel.

The basic idea is to output the metadata about a crate before code generation so that the next compile can proceed based on that metadata.

rustc gets modified in a few ways:

• Just before trans we serialize the bulk of the metadata and insert a hook to give the multi-crate driver an opportunity to copy it to another task to begin the next compilation phase. There will still be some metadata that needs to be serialized after trans.
• Insert another hook into crate resulotion for reading the metadata binary. The multi-crate driver will use this to receive the metadata output from previous crates and feed them to the next crates.

The end result is that parsing and analysis will be largely done in parallel with code generation for multi-crate builds.

Beyond that we can also parse all crates in parallel.

This would require that we make sure we are using LLVM in a threadsafe way, unless we can get all the trans passes onto a single thread.

Related to #2237

[catamorphism]

Added far-future milestone

[emberian]

LLVM is now used threadsafely.

Still relevant, though I'm not sure how you plan to actually link the crate...

[thestinger]

There's a lot of potential for parallel compilation in LLVM itself. The function pass is an embarrassingly parallel task since it runs on each function in isolation.

http://lists.cs.uiuc.edu/pipermail/llvmdev/2013-July/063826.html

I think this would be a huge win for us.

[huonw]

Triage: no change, and the LLVM parallel function pass patch seems dead: http://llvm-reviews.chandlerc.com/D1152

[kmcallister]

Cargo now provides a multi-crate compilation manager, so the remaining task here is to improve parallelism between rustc processes. Perhaps they need a protocol to talk about which files aren't ready yet, and to notify when they become ready. Cargo would act as a router for this protocol. Alternately it could pass a list of not-ready files and each rustc process would watch them with inotify or similar (avoiding races will be tricky).

The hardest part of this may be the dynamic job management to balance CPU utilization and peak RAM use when many rustc processes are blocking on file updates.

[steveklabnik]

I'm pulling a massive triage effort to get us ready for 1.0. As part of this, I'm moving stuff that's wishlist-like to the RFCs repo, as that's where major new things should get discussed/prioritized.

This issue has been moved to the RFCs repo: rust-lang/rfcs#610