`DwarfCompileUnit::createAndAddScopeChildren` gobbles stack

[workingjubilee]

By far the most common stack overflow bug report nowadays for rustc is rustc dying in LLVM while emitting debuginfo. Lately DwarfCompileUnit::createAndAddScopeChildren has seemed to acquire the power of recursing on itself and destroying the stack essentially without assistance. People are setting hundreds of megabytes for RUST_MIN_STACK to get their code to compile. That's... closer to our resident set size in most executions?

Unlike other reports like #76920 this has been happening with our stock rustc distribution, as far as I can tell, and our bundled LLVM.

[llvmbot]

@llvm/issue-subscribers-debuginfo

Author: Jubilee (workingjubilee)

By far the most common stack overflow bug report nowadays for rustc is rustc dying in LLVM while emitting debuginfo. Lately `DwarfCompileUnit::createAndAddScopeChildren` has seemed to acquire the power of recursing on itself and destroying the stack essentially without assistance. People are setting hundreds of megabytes for `RUST_MIN_STACK` to get their code to compile. That's... closer to our resident set size in most executions?

Unlike other reports like #76920 this has been happening with our stock rustc distribution, as far as I can tell, and our bundled LLVM.

[dwblaikie]

Some reproducer(s) and, if it's a regression, an example of this not being a problem on earlier versions, would be helpful.

[workingjubilee]

Textual LLVMIR from a... I can't call it "minimal", but it's a reproducer: createandaddsigsev.ll

It's hard to tell whether this is a "regression" or not, honestly. I hope the reason this is the most common segfault lately is because we've won the game of whack-a-mole with the others.

[dwblaikie]

Might end up leaving this to someone else to look at, but I got a flame graph just cpu-based, didn't look at the memory usage and I didn't see a deep stack related to createAndAddScopeChildren, but a few related to the LLVM IR verifier (calling getSubprogram on the debugloc on each instruction - so it has to recurse a few times up for deeply inlined code - that could be linearized int oa loop rather than recursion), and some other getSubprogram recursions related to DwarfDebug::collectEntityInfo... collectVariableInfoFromMFTable

I guess maybe the deep recursion happened too quick/didn't get sampled. valgrind detects the recursion and diagnoses llvm::DwarfCompileUnit::applyConcreteDbgVariableAttributes(llvm::Loc::MMI const&, llvm::DbgVariable const&, llvm::DIE&) (DwarfCompileUnit.cpp:891) (or points to it at least)

I guess this is some deep inlining or many nested lexical scopes? (I haven't looked at the IR to try to find the shape of it... ) - generally LLVM won't scale well with very large functions, though I guess a lot of inlining doesn't necessarily mean a large function and deeply inlined code only really impacts DWARF handling - outside of DWARF, everything related to inlined code just "goes away"...

[dwblaikie]

Hmm, in a non-optimized build of the compiler it's stack-overflowing in llvm::AsmPrinter::emitDwarfDIE instead.

I'm guessing this is some deeply recursive code?

[dwblaikie]

Hmm, nope, all these nestings appear to be lexical blocks...

What is this code? We /might/ be able to make LLVM work better with it, but it seems pretty weird/not sure we'd want to do a lot of work to support it without understanding it better...

[workingjubilee]

Yeah, I mean, I wouldn't want you to do much more than the standard trick... I'd PR it myself except my C++ skills are bad enough I'd probably fuck it up somehow!... of hoisting recursion into iteration inside relevant functions, by having an inner function return the state of the "recursion" and then repeatedly calling that inner function to drive the "recursion" forward.

What is this code? We /might/ be able to make LLVM work better with it, but it seems pretty weird/not sure we'd want to do a lot of work to support it without understanding it better...

A Rust stack frame!

...in real programs, instead of this somewhat decontextualized example, this is "someone asked for full debuginfo and then declares MANY variables". The problem is that the breaking points get reached in programs people actually want to build rather than purely hypothetical ones. And not, like... absurd code like entire gate-by-gate CPU simulators, though I have gotten a report of that, but "ordinary" things like cryptography algorithms. My understanding is that yes, inlining does play some part in it.

But otherwise this is just a somewhat overly literalist interpretation of Rust.

Now, it might be that there's a better way to do this, but we started doing it this specific way because we had issues with getting values to be emitted into debuginfo in a way we can easily predict. Otherwise we might do something else.

It seems quite a bit has changed over time with LLVM's debug info handling, so if you have any recommendations as to how to reduce the frequency of this occurring on our side but which might require being aware of the last... 4 years of LLVM goings-on, I'm interested!

[dwblaikie]

Declaring a lot of variables shouldn't result in this sort of debug info - unless it's being generated somewhat esoterically.

lexical_blocks are for literal language-level lexical blocks like anything with "{}" in C++, roughly. Is rust using a separate lexical_block for every variable lifetime?

in C++, for instance, this code would have one lexical block:

int x = 3;
int a = 2;
{
  int x = 5;
  func();
  int y = 7;
  ...
}

Expressing the name overriding. One of the problems with this is that, at the call to func(), the debugger doesn't know that y doesn't exist yet - now there's a different, but unsupported-in-LLVM, DWARF feature that can be used to address this problem (DW_AT_scope_start).

But if rust is putting in new artificial scopes for every variable to avoid the "we're in the lexical scope, but the variable doesn't exist yet" problem - I'd /highly/ encourage not doing that, and accepting the less-than-ideal behavior that happens. Introducing a scope for every variable is likely not going to scale well with many pieces of the infrastructure here (compiler, object files, debuggers, symbolizers, etc)

Perhaps you could show me a smaller example of rust with a few variables, the source code, the LLVM IR, and the resulting dwarf (llvm-dwarfdump to dump it)?

[jryans]

I was curious about this, so I tried a small Rust example on CE:

#[no_mangle]
pub fn square(num: i32) -> i32 {
    let x = num;
    let y = num;
    x + y
}

There are indeed separate DILexicalBlocks in the IR for x and y:

!15 = !DILocalVariable(name: "x", scope: !16, file: !8, line: 3, type: !12, align: 4)
!16 = distinct !DILexicalBlock(scope: !7, file: !8, line: 3, column: 5)
!17 = !DILocalVariable(name: "y", scope: !18, file: !8, line: 4, type: !12, align: 4)
!18 = distinct !DILexicalBlock(scope: !16, file: !8, line: 4, column: 5)

But if rust is putting in new artificial scopes for every variable to avoid the "we're in the lexical scope, but the variable doesn't exist yet" problem - I'd /highly/ encourage not doing that, and accepting the less-than-ideal behavior that happens. Introducing a scope for every variable is likely not going to scale well with many pieces of the infrastructure here (compiler, object files, debuggers, symbolizers, etc)

Rust does seem to do this, but they are also not alone... Swift makes the same choice, as described in this 2023 blog post from @adrian-prantl and others.

[workingjubilee]

Yes. Rust supports shadowing, so it's possible... even common, in some styles of code I've seen... to have like 6 different declarations of a variable named x, with three different types and twelve different values.

I don't know of another way to model that so a debugger can understand except by saying "fuck it, new scope time".