You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Looks to me like over half the time is spent on running the linker. Which doesn't surprise me, the dependencies are huge! In a debug-build, goblin is a 17MB rlib, indicativ is a 6.2MB rlib, and reqwest is a 28MB rlib. I'm guessing your compile times wouldn't be must better in C or C++.
If rust/cargo had better support for dynamic linking it might go a lot faster.
So I, personally, actually have no idea why we run dsymutil. I did a bit of digging to find out and I've never blamed a line of code so far back at this one in Rust's history. Turns out 196351aa introduced it, and as with many lines of code written seven (!) years ago it's changed a lot since then and unfortunately doesn't have a ton of rationale for why it exists (hey one can hope can't he?).
That being said a little testing locally makes it pretty obvious as to why it exists. At least lldb doesn't work without it! Or at least for me if I deleted the *.dSYM folder then lldb wasn't able to set a breakpoint. It looks as if dsymutil doesn't modify the binary as well, so I think it's just generating that extra debuginfo.
Despite that though I still have very little idea what dsymutil is doing. For example are there options we can compile with to make it take less time? Options to dsymutil? Something like -gsplit-dwarf to make it (I guess?) copy less data around? It seems to spend a lot fo dime in DwarfLinker::DIECloner (at least the LLVM implementation which I assume is the same as Apple's at some point). The source doens't seem too illuminating.
My guess is that we can't not run dsymutil if we want up-to-date debug information, but I could be wrong! With all the work being done it seems unlikely at least that lldb is going back to the original objects.
All that being said surely C++ runs into similar problems here. Or maybe we generate more debuginfo? Or maybe the executable here is too big? Unsure :(
So rustc deletes object files that go into rlibs, basically cleaning up after itself. If we don't do that, however, then lldb appears to work even if the *.dSYM is deleted.
And then even more interestingly!
It turns out that if I delete the *.dSYM folder then lldb does actually work for dependencies. That actually all makes sense to me I think. I believe that dsymutil is basically a "dwarf linker" which presumably just assembles everything into one location (the *.dSYM folder). The actual binary itself just contains a bunch of mappings to "go find my debuginfo in these files" (I believe) and lldb seems to process those if *.dSYM isn't available (not sure of the priority, but hey).
So debugging a binary doesn't actually work because the object files for the binary are cleaned up and deleted by rustc. Debugging a library of a binary works though because the rlib is still on the filesystem after the build.
It seems odd to try to change course with dsymutil now but it actually seems like we should:
Not delete object files for binaries with debuginfo (somehow)
Not run dsymutil by default if this happens.
I believe that means that we wouldn't run dsymutil at all by default as part of rustc itself. This should save time on incremental builds as we don't ever run anything, and if someone's build system needs to ship the *.dSYM folder they can always generated the debug symbols themselves by running dsymutil as a postprocessing step.
Now emitting object files from rustc by default is pretty tricky. Mostly in that we've never done it before and it's not clear if people want object files getting jettisoned out. What I think we could do though is not change rustc's defaults but change Cargo's defaults. For example we could have a flag to rustc like --leave-debuginfo-related-files-on-disk-i-am-managing-them and Cargo would pass that along. When rustc has that flag it wouldn't execute dsymutil, making the incremental builds here 38% faster! (ish)
@michaelwoerister curious what you think about this! I feel like it'd actually be pretty nice to not have to run dsymutil...
Keeping object files around would also help on Linux where we could use debuginfo fission to speed up linking. The same approach might be possible on OS X.
I believe that dsymutil is basically a "dwarf linker" which presumably just assembles everything into one location (the *.dSYM folder)
Yes, you're exactly correct. Apple's toolchain does the equivalent of -gsplit-dwarf by default, and dsymutil links the debug info. Since rustc runs dsymutil after every link it effectively nullifies the benefits of not linking the DWARF. :) I'm not surprised that lldb supports locating the object files and reading the debug info from there, Apple's gdb supported that as well. (FWIW, the stabs entries in the binaries that lldb/gdb use to locate the object files are the same ones causing the non-determinism you found!)
dsymutil is still useful because most other tools do not know how to locate the debug symbols from the object files, so people who want to run things like profilers or whatnot will probably need this step. It's not particularly onerous to tell people that they have to run dsymutil target/whatever/yourbinary to make those use cases work, though, especially if the tradeoff is much faster builds.
Just as an aside, dsymutil takes forever to run on Firefox's libxul. We don't run it as part of the normal build, but we run it as a post-build step so we can upload the dSYM to our symbol server and also so we can run our dump_syms tool on it to get symbols in Breakpad format for our crash reports. If we ever get to a point where we let cargo build libxul we'd definitely want this fixed!
Activity
Mark-Simulacrum commentedon Jan 6, 2018
Is this the project? https://github.com/jrmuizel/pdb-downloader/
jrmuizel commentedon Jan 6, 2018
Ah yes, sorry. I meant to include that link in the original report.
Thiez commentedon Jan 7, 2018
Looks to me like over half the time is spent on running the linker. Which doesn't surprise me, the dependencies are huge! In a debug-build, goblin is a 17MB rlib, indicativ is a 6.2MB rlib, and reqwest is a 28MB rlib. I'm guessing your compile times wouldn't be must better in C or C++.
If rust/cargo had better support for dynamic linking it might go a lot faster.
alexcrichton commentedon Jan 9, 2018
Some local timings:
cc
(the linker)dsymutil
(had to guess, this isn't timed by rustc by default)Opening up a profiler I see
DepGraph::try_mark_green
link
function (the libc functionlink
)memmove
mmap
, mostly from jemallocjrmuizel commentedon Jan 9, 2018
Given the object files should still be in place is it possible for us to avoid having to run dsymutil?
alexcrichton commentedon Jan 9, 2018
So I, personally, actually have no idea why we run dsymutil. I did a bit of digging to find out and I've never blamed a line of code so far back at this one in Rust's history. Turns out 196351aa introduced it, and as with many lines of code written seven (!) years ago it's changed a lot since then and unfortunately doesn't have a ton of rationale for why it exists (hey one can hope can't he?).
That being said a little testing locally makes it pretty obvious as to why it exists. At least lldb doesn't work without it! Or at least for me if I deleted the
*.dSYM
folder thenlldb
wasn't able to set a breakpoint. It looks as ifdsymutil
doesn't modify the binary as well, so I think it's just generating that extra debuginfo.Despite that though I still have very little idea what dsymutil is doing. For example are there options we can compile with to make it take less time? Options to dsymutil? Something like
-gsplit-dwarf
to make it (I guess?) copy less data around? It seems to spend a lot fo dime inDwarfLinker::DIECloner
(at least the LLVM implementation which I assume is the same as Apple's at some point). The source doens't seem too illuminating.My guess is that we can't not run dsymutil if we want up-to-date debug information, but I could be wrong! With all the work being done it seems unlikely at least that lldb is going back to the original objects.
All that being said surely C++ runs into similar problems here. Or maybe we generate more debuginfo? Or maybe the executable here is too big? Unsure :(
alexcrichton commentedon Jan 9, 2018
Interesting though!
So rustc deletes object files that go into rlibs, basically cleaning up after itself. If we don't do that, however, then
lldb
appears to work even if the*.dSYM
is deleted.And then even more interestingly!
It turns out that if I delete the
*.dSYM
folder then lldb does actually work for dependencies. That actually all makes sense to me I think. I believe thatdsymutil
is basically a "dwarf linker" which presumably just assembles everything into one location (the*.dSYM
folder). The actual binary itself just contains a bunch of mappings to "go find my debuginfo in these files" (I believe) and lldb seems to process those if*.dSYM
isn't available (not sure of the priority, but hey).So debugging a binary doesn't actually work because the object files for the binary are cleaned up and deleted by rustc. Debugging a library of a binary works though because the rlib is still on the filesystem after the build.
It seems odd to try to change course with
dsymutil
now but it actually seems like we should:dsymutil
by default if this happens.I believe that means that we wouldn't run
dsymutil
at all by default as part ofrustc
itself. This should save time on incremental builds as we don't ever run anything, and if someone's build system needs to ship the*.dSYM
folder they can always generated the debug symbols themselves by runningdsymutil
as a postprocessing step.Now emitting object files from rustc by default is pretty tricky. Mostly in that we've never done it before and it's not clear if people want object files getting jettisoned out. What I think we could do though is not change rustc's defaults but change Cargo's defaults. For example we could have a flag to rustc like
--leave-debuginfo-related-files-on-disk-i-am-managing-them
and Cargo would pass that along. Whenrustc
has that flag it wouldn't executedsymutil
, making the incremental builds here 38% faster! (ish)@michaelwoerister curious what you think about this! I feel like it'd actually be pretty nice to not have to run
dsymutil
...jrmuizel commentedon Jan 9, 2018
@luser
jrmuizel commentedon Jan 9, 2018
Yeah, we don't run dsymutil on the c++ part of Firefox builds because we still have all of the object files around.
michaelwoerister commentedon Jan 9, 2018
Keeping object files around would also help on Linux where we could use debuginfo fission to speed up linking. The same approach might be possible on OS X.
luser commentedon Jan 9, 2018
Yes, you're exactly correct. Apple's toolchain does the equivalent of
-gsplit-dwarf
by default, and dsymutil links the debug info. Since rustc runs dsymutil after every link it effectively nullifies the benefits of not linking the DWARF. :) I'm not surprised that lldb supports locating the object files and reading the debug info from there, Apple's gdb supported that as well. (FWIW, the stabs entries in the binaries that lldb/gdb use to locate the object files are the same ones causing the non-determinism you found!)dsymutil is still useful because most other tools do not know how to locate the debug symbols from the object files, so people who want to run things like profilers or whatnot will probably need this step. It's not particularly onerous to tell people that they have to run
dsymutil target/whatever/yourbinary
to make those use cases work, though, especially if the tradeoff is much faster builds.Just as an aside, dsymutil takes forever to run on Firefox's libxul. We don't run it as part of the normal build, but we run it as a post-build step so we can upload the dSYM to our symbol server and also so we can run our dump_syms tool on it to get symbols in Breakpad format for our crash reports. If we ever get to a point where we let cargo build libxul we'd definitely want this fixed!
jrmuizel commentedon Jan 9, 2018
FWIW, Instruments does not need you to run dysmutil to profile Firefox. I'm not sure what tools do.
17 remaining items