BOLT: clang segfault after instrumenting libclang-cpp.so or libLLVM.so when built with -DLLVM_LINK_{CLANG,LLVM}_DYLIB=ON

[joebonrichie]

Tested as of db1a762, the last worked was with llvm-17

Repro:

$ export LDFLAGS="-Wl,-O1 -Wl,-z,relro -Wl,-z,now -Wl,-z,max-page-size=0x1000 -Wl,-Bsymbolic-functions -Wl,--sort-common -Wl,-z,pack-relative-relocs -Wl,-q"
$ cmake -G Ninja -B stage1 -S llvm -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=/usr/lib64/ccache/bin/clang -DCMAKE_CXX_COMPILER=/usr/lib64/ccache/bin/clang++ -DCMAKE_ASM_COMPILER=/usr/lib64/ccache/bin/clang -DLLVM_ENABLE_PROJECTS="clang;lld;bolt" -DLLVM_ENABLE_RUNTIMES="compiler-rt" -DCOMPILER_RT_BUILD_SANITIZERS=OFF -DCOMPILER_RT_BUILD_XRAY=OFF -DCOMPILER_RT_BUILD_LIBFUZZER=OFF -DCMAKE_INSTALL_PREFIX=stage1/install -DLLVM_LINK_LLVM_DYLIB=ON -DCLANG_LINK_CLANG_DYLIB=ON
$ ninja -C stage1 install
$ ./stage1/install/bin/llvm-bolt -instrument ./stage1/install/bin/../lib/libclang-cpp.so.20.0git -o ./stage1/install/bin/../lib/libclang-cpp.so.20.0git.instr
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: db1a76206902e6926464e30cc47249e217405dab
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x4200000, offset 0x4200000
BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling lite mode
BOLT-WARNING: Failed to analyze 16 relocations
BOLT-WARNING: 6 collisions detected while hashing binary objects. Use -v=1 to see the list.
BOLT-INFO: 0 out of 112356 functions in the binary (0.0%) have non-empty execution profile
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 39341
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 110954
BOLT-INSTRUMENTER: Number of function descriptors: 110954
BOLT-INSTRUMENTER: Number of branch counters: 1350926
BOLT-INSTRUMENTER: Number of ST leaf node counters: 656205
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 2007131
BOLT-INSTRUMENTER: Total size of counters: 16057048 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 16615228 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 133046532 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /tmp/prof.fdata
BOLT-INFO: 550535 instructions were shortened
BOLT-INFO: removed 4300 empty blocks
BOLT-INFO: UCE removed 80769 blocks and 4856578 bytes of code
BOLT-INFO: padding code to 0xac00000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0xc93a7a0
BOLT-INFO: clear procedure is 0xc936340
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting __bolt_runtime_start to 0xc93a760
BOLT-INFO: setting __bolt_runtime_fini to 0xc93a7a0
BOLT-INFO: setting __hot_start to 0x4400000
BOLT-INFO: setting __hot_end to 0xab075d7
$ mv ./stage1/install/bin/../lib/libclang-cpp.so.20.0git.instr ./stage1/install/bin/../lib/libclang-cpp.so.20.0git
$ ./stage1/install/bin/clang -v
Segmentation fault (core dumped)

backstrace

                Stack trace of thread 307074:
                #0  0x00007fb7b7fd7043 elf_get_dynamic_info (ld-linux-x86-64.so.2 + 0x7043)
                #1  0x00007fb7b7fd7fe1 _dl_map_object (ld-linux-x86-64.so.2 + 0x7fe1)
                #2  0x00007fb7b7fd29b5 openaux (ld-linux-x86-64.so.2 + 0x29b5)
                #3  0x00007fb7b7fd14dd __GI__dl_catch_exception (ld-linux-x86-64.so.2 + 0x14dd)
                #4  0x00007fb7b7fd2d14 _dl_map_object_deps (ld-linux-x86-64.so.2 + 0x2d14)
                #5  0x00007fb7b7ff2a7f dl_main (ld-linux-x86-64.so.2 + 0x22a7f)
                #6  0x00007fb7b7fef343 _dl_sysdep_start (ld-linux-x86-64.so.2 + 0x1f343)
                #7  0x00007fb7b7ff0c7e _dl_start_final (ld-linux-x86-64.so.2 + 0x20c7e)
                #8  0x00007fb7b7fef988 _start (ld-linux-x86-64.so.2 + 0x1f988)
                ELF object binary architecture: AMD x86-64

Lemme know if I can provide any more details.

[joebonrichie]

bughunter

$ BOLT=./stage1/install/bin/llvm-bolt BOLT_OPTIONS="-v=1" INPUT_BINARY=./stage1/install/lib/libclang-cpp.so.20.0git bolt/utils/bughunter.sh 
Verify input binary passes
  INPUT_BINARY: : &&  ./stage1/install/lib/libclang-cpp.so.20.0git  |& cat >& /tmp/libclang-cpp.so.20.0git.3tX.bolt.out
bolt/utils/bughunter.sh: line 97: 698665 Segmentation fault      (core dumped) $TIMEOUT_OR_CMD $INPUT_BINARY $COMMAND_LINE 2>&1
     698666 Done                    | $POST_COMMAND >&$OUTPUT_FILE
  Warning: input binary failed
Verify optimized binary fails
  Bolt crashes while generating optimized binary.
egrep: warning: egrep is obsolescent; using grep -E
egrep: warning: egrep is obsolescent; using grep -E
Iteration 0, trying /tmp/func-names.eiR.txtaa / 56983 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.eiR.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 1, trying /tmp/func-names.rJG.txtaa / 28492 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.rJG.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 2, trying /tmp/func-names.9rf.txtaa / 14246 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.9rf.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 3, trying /tmp/func-names.OOh.txtaa / 7123 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.OOh.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 4, trying /tmp/func-names.Emo.txtaa / 3562 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.Emo.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 5, trying /tmp/func-names.Cc6.txtaa / 1781 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.Cc6.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 6, trying /tmp/func-names.yo9.txtaa / 891 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.yo9.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 7, trying /tmp/func-names.xZG.txtaa / 446 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.xZG.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 8, trying /tmp/func-names.ZjP.txtaa / 223 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.ZjP.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 9, trying /tmp/func-names.hOm.txtaa / 112 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.hOm.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 10, trying /tmp/func-names.6Gf.txtaa / 56 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.6Gf.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 11, trying /tmp/func-names.Wrg.txtaa / 28 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.Wrg.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 12, trying /tmp/func-names.NdQ.txtaa / 14 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.NdQ.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 13, trying /tmp/func-names.hPG.txtaa / 7 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.hPG.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 14, trying /tmp/func-names.2dz.txtaa / 4 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.2dz.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 15, trying /tmp/func-names.YxM.txtaa / 2 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.YxM.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
Iteration 16, trying /tmp/func-names.Zte.txtaa / 1 functions
  BOLT: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.Zte.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt >& /tmp/bolt3A4.log
  BOLT failure=1
The function(s) that failed are in /tmp/func-names.xv2.txtaa
To reproduce, run: ./stage1/install/bin/llvm-bolt -v=1 ./stage1/install/lib/libclang-cpp.so.20.0git -funcs-file-no-regex=/tmp/func-names.Zte.txtaa -o /tmp/libclang-cpp.so.20.0git.3tX.bolt
rm: cannot remove '/tmp/libclang-cpp.so.20.0git.3tX.bolt': No such file or directory
rm: cannot remove '/tmp/libclang-cpp.so.20.0git.3tX.bolt.out': No such file or directory
[ble: exit 1]
 $ cat /tmp/func-names.xv2.txtaa -p
__bolt_fini_trampoline

[joebonrichie]

bisected to 0053cb8

partial revert of 0053cb8 fixes clang segfaulting after libclang/libLLVM has been instrumented

diff --git a/bolt/lib/Rewrite/RewriteInstance.cpp b/bolt/lib/Rewrite/RewriteInstance.cpp
index 32ec7abe8b66..f7f63b2ea956 100644
--- a/bolt/lib/Rewrite/RewriteInstance.cpp
+++ b/bolt/lib/Rewrite/RewriteInstance.cpp
@@ -5068,11 +5068,9 @@ void RewriteInstance::patchELFAllocatableRelrSection(
   const uint8_t PSize = BC->AsmInfo->getCodePointerSize();
   const uint64_t MaxDelta = ((CHAR_BIT * DynamicRelrEntrySize) - 1) * PSize;
 
-  auto FixAddend = [&](const BinarySection &Section, const Relocation &Rel,
-                       uint64_t FileOffset) {
+  auto FixAddend = [&](const BinarySection &Section, const Relocation &Rel) {
     // Fix relocation symbol value in place if no static relocation found
-    // on the same address. We won't check the BF relocations here since it
-    // is rare case and no optimization is required.
+    // on the same address
     if (Section.getRelocationAt(Rel.Offset))
       return;
 
@@ -5081,6 +5079,10 @@ void RewriteInstance::patchELFAllocatableRelrSection(
     if (!Addend)
       return;
 
+    uint64_t FileOffset = Section.getOutputFileOffset();
+    if (!FileOffset)
+      FileOffset = Section.getInputFileOffset();
+    FileOffset += Rel.Offset;
     OS.pwrite(reinterpret_cast<const char *>(&Addend), PSize, FileOffset);
   };
 
@@ -5102,7 +5104,7 @@ void RewriteInstance::patchELFAllocatableRelrSection(
       RelOffset = RelOffset == 0 ? SectionAddress + Rel.Offset : RelOffset;
       assert((RelOffset & 1) == 0 && "Wrong relocation offset");
       RelOffsets.emplace(RelOffset);
-      FixAddend(Section, Rel, RelOffset);
+      FixAddend(Section, Rel);
     }
   }
 
-- 

@yota9 hopefully you don't mind the ping