Advanced Symbols

Speed Up Native Symbol Resolution

When a WPA table has enabled a Stack column, most symbols may need to resolve before analysis can proceed.

Q: Why might it take a long time for WPA to resolve symbols?
What's it doing!?

A: WPA is doing the following:

1. Searching for hundreds of PDB symbol files (which may still not exist)
2. Downloading large PDB files
3. Translating PDB files to SymCache format
4. Finding / loading each SymCache file

Solutions:

• Close as many apps as possible before capturing a trace.
  With less code active on the system, there will be fewer modules referenced once the trace is captured.
• Launch WPA (View command) with the -FastSym switch. This eliminates the slowdown from steps 1-3 above.
      TraceCPU View -FastSym
  This works by launching WPA with: -symbols -symcacheonly
  At the same time, a CMD window is launched (minimized) to slowly download other native symbols in the background. (This requires XPerf on the system.) Then to force WPA to reload symbols, including those recently downloaded in the background, do: Trace > Load Symbols - turn off and on
• Manually download native symbols:
  Use the GetSymbols† script to download symbols for all or specific modules referenced in a trace:
      BETA\GetSymbols.bat "<path>\MyTrace.etl" -v
    BETA\GetSymbols.bat "<path>\MyTrace.etl" Module1.exe Module2.dll ...
  
• Limit WPA to load symbols for just a few processes.
  In WPA: Trace > Configure Symbol Paths > Load Settings tab > Enable: Load symbols per the following restrictions:
      "Load symbols for the following processes:"
      Add the binary names of the processes of interest: Excel.exe, PowerPnt.exe, WinWord.exe, etc.
• Windows Defender has been observed to scan and slow symbol downloads.
  You can control this using the SymbolScan script:
      - To exclude ETW symbol/trace files from Defender scans, run: SymbolScan -Prevent
      - To restore Defender to scanning ETW symbol/trace files, run: SymbolScan -Allow
      - To list ETW symbol/trace files excluded from Defender scans, run: SymbolScan
  
• Be patient! The second round is always faster, since most symbol files will have already been cached.

Share Symbols with Others

It may not be enough to share a .ETL trace file with colleagues if they don't have access to the same symbol files, such as when some of the referenced modules were built locally.

Solution 1: Embed the symbols within the trace file.

• Within WPA: Open the trace and load symbols. Make sure all symbols of interest are loaded, but you can exclude from loading (for efficiency) modules which may be extraneous: Trace > Configure Symbol Paths > Load Settings
• In WPA, select: File > Export Package
  This creates a .WPAPK file, which is a zipped archive of the .ETL trace, the .SymCache symbol files, and the .wpaProfile view configuration. You load a .WPAPK file into WPA much like you would an ordinary .ETL file.
  NOTE: Those embedded symbols, once loaded into WPA, will also be available to other traces on the same machine thereafter.

Solution 2: Cache generated SymCache files to a common server share.

• Set up a commonly accessible UNC server share: \\centralserver\symcache

• Set that UNC share path as the shared cache store for SymCache files:
      - set _NT_SYMCACHE_PATH = \\centralserver\symcache*<any_additional_paths>
      - Within WPA: Trace > Configure Symbol Paths > SymCache > SymCache Generation Folder = \\centralserver\symcache

  Note that WPA also supports downloading SymCache files from an http symcache server that you set up:
  _NT_SYMCACHE_PATH = c:\symcache*http://OurSymCacheServer

  You can manually transcode .PDB symbol files (emitted by the C/C++/ASM linker) to .SymCache format (used by WPA) using the SymCacheGen tool, available in the ADK > Windows Performance Toolkit.

Troubleshoot Symbol Resolution

On occasion you need more visibility into a trace's symbol information, perhaps to resolve symbol errors or to generate a module disassembly.

• For symbol resolution to work at all, the trace must be collected with System Keywords: Loader, Process Thread
  • To see the System Keywords of a running profile (for example): TraceCPU Status -verbose
  • Or to see the System Keywords of a particular WPR Profile: WPR -ProfileDetails <path>\<Name>.wprp!ProfileName
• WPA's 'Symbols Hub' table (Windows menu) lists symbols found within the trace, filterable by name or by status.
• WPA's 'Images' (Modules) table lists each module per process with detailed information.
• WPA does not launch, or is acting strangely: reset its caches.
  Run the script: MSO-Scripts\ResetWPA [-All]
• A very large symbol file failed to download. Could it have timed out?
  Set the environment variable (in seconds): set SYMSRV_HTTP_RECEIVE_TIMEOUT=240 OR $Env:SYMSRV_HTTP_RECEIVE_TIMEOUT=240
• GetSymbols† finds and downloads symbol files (.PDB) of modules referenced within a given ETW Trace Log (.ETL), and converts them to SymCache format:
  • Run: MSO-Scripts\BETA\GetSymbols.bat /?
• SymChk finds and downloads PDB symbol files for specific modules on your machine:
  • Get SymChk.exe : Install only the Windows WDK.
    Then find it at: c:\Program Files (x86)\Windows Kits\10\Debuggers\x64
  • Ensure the environment variable: _NT_SYMBOL_PATH
    Example: _NT_SYMBOL_PATH = cache*c:\symbols;srv*https://msdl.microsoft.com/download/symbols
  • Run: SymChk "<path>\<module>" /v /od
    

Note

For the C/C++/ASM linker to emit .PDB symbol files requires certain switches.
Even then, symbol names in highly optimized code are notoriously difficult to resolve unambiguously.
This is why symbolic stack walks (call stacks) may occasionally include function names which appear to make no sense.

Missing ImageId event

All I see in the Stack column is:  MyModule!<Missing ImageId event>
Q: Why does this happen, and how can I fix it!?
A: When WPR puts together the .ETL trace file (WPR -stop MyTraceFile.etl ...), it finds each referenced native module, loads the module header, and reads the embedded signature of the debug information, thereby generating an "ImageId event" within the trace. Later, if WPA does not find a particular ImageId event within that trace then it cannot resolve those symbols, and it will report simply: MyModule!<Missing ImageId event>

Even if the trace was collected on a different machine, you can fix it, as long as you have access to the exact same version of that module:

1. Recent versions of WPA provide this table: Graph Explorer > Other > ImageId Errors
   Alternately, go to the "Images" (modules) table, remove any filters, and enable the "Image Name" and "Image Path" columns.
2. If the module of interest is not in the location specified in the table, copy it to there.
   Create the specified path, if needed. If that drive does not exist on this machine, use: subst
3. Run SymChk (optional, as above) on the module. Make sure that you see in the output:
       [SYMCHK] Searching for symbols to <Module Path> in path <Symbol Server Path> ...
       [SYMCHK] PDB Matched: TRUE ...
       PdbFilename <Path\Name>.pdb
    PdbSignature {<GUID>}...
        etc.
   If this doesn't work, either the native module wasn't built with symbols, or the .PDB file is no longer available on the symbol path.
4. Run: WPR -merge <MyTraceFile>.etl <MyTraceFile>_Fixed.etl -compress
5. Open the _Fixed.etl file in WPA and verify that the symbols appear in the Stack column for the module of interest:
        MSO_Scripts\TraceCPU View -Path <MyTraceFile>_Fixed.etl
   Note that the ImageId Errors table is no longer valid after the -merge, so don't go off of that.
   Instead you can confirm by running (as below):
       xperf -i <MyTraceFile>_Fixed.etl -a symcache -dbgid | findstr /i <ModuleName>

Digging Deeper

XPerf can perform module/symbol operations on an ETL trace.
(XPerf.exe is often adjacent to WPA.exe, or available in the ADK / Windows Performance Toolkit.)

• List module names with timestamp & date:
  xperf -i <TRACE>.etl -a symcache -imageid
• List .PDB names with signature (Name + GUID + Age):
  xperf -i <TRACE>.etl -a symcache -dbgid
• Download PDB files and translate to SymCache† all referenced symbols using the environment variables: _NT_SYMBOL_PATH and _NT_SYMCACHE_PATH
  xperf -i <TRACE>.etl -symbols -a symcache -build
• Download PDB files and translate to SymCache† the symbols of specific modules using the environment variables:
  xperf -i <TRACE>.etl -symbols -a symcache -build -image <Module1> <Module2> ...
xperf -i <TRACE>.etl -symbols verbose -a symcache -build -image <Module1> <Module2> ... // with diagnostic output
• More info on the trace requirements and environment variables:
  xperf -help symbols

Note

If XPerf complains of lost events then add (before -i): -tle
If XPerf complains of time inversions then add (before -i): -tti

Tip

† Use the GetSymbols script to download symbols (.PDB files transcoded to .SymCache format) for all or specific modules in a given ETW trace log (.ETL):
Run:  MSO-Scripts\BETA\GetSymbols.bat mypath\mytrace.etl -v
Run:  MSO-Scripts\BETA\GetSymbols.bat mypath\mytrace.etl Module1.exe Module2.dll ...
Run:  MSO-Scripts\BETA\GetSymbols.bat /?

Missing Stackwalks

Symbol resolution is of no value if stackwalk information is missing from WPA's Stack column.

Here are several reasons for finding N/A in the Stack column:

• Stackwalking is not enabled for that ETW provider in the WPR recording profile (.wprp).
  To see (while tracing) which providers have stackwalking enabled: Trace<Name> Status -Verbose
• Stackwalking among sampled and event-driven ETW providers is intermittently colliding.
  For example, collecting both CPU samples and FileIO events with stackwalks in the same trace.
• Stackwalks are intermittently timing out. ETW allows only so much time per stackwalk collection (default 400 ms).
  This is controlled via registry value (REG_DWORD in milliseconds):
  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\WMI\Trace!StackCaptureTimeout
• Stackwalks are getting dropped due to a very high data logging rate.
• Unknowable reasons.