If you use Visual Studio to debug your code you will be used to it magically loading the correct symbols as it debugs your running process. In fact, you may be so used to it that it might not have occurred to you that loading symbols are something that the debugger has to do, never mind it being something that you have to do manually in some situations.
But if you are used to debugging crash dumps for software that has been released in the wild, attaching a debugger to remote processes running a CI-built deployment, or have used other, more manual debugging systems like gdb or WinDbg, then you will be a lot more aware of the need for a debugger to load the correct symbols in order for it to make sense of all the binary data that makes up a running (or most likely crashing) program.
Luckily for most of us, Visual Studio does a really good job of loading symbols automatically (specifically of loading the correct symbols automatically) such that a lot of the time you never even notice that a job is being done, never mind that it is doing it correctly. And so long as your CI system and Visual Studio symbol paths are set up correctly the system will work, letting you trawl the depths of a crash dump from any version of your software that has been deployed.
What if it doesn’t though? What if you are faced with the dreaded “Symbols not loaded” screen? And why, when you point Visual Studio at the dll or pdb that you are sure is the one it needs, will it refuse to load it? Well, lets peel back the curtain and take a peek at how the magic works for loading symbols and debugging a crashdump.
For this deep-dive I will be using Hex Editor Neo Free although you can use other tools like CFF explorer or any other hex editor or PE explorer, to do the same thing. Also, I have to give full credit to this post from Oleg Starodumov which helped me to decipher much of the stuff that is left out of the official documentation.
So what happens when you get a minidump, load it into visual studio and start debugging? When debugging a full memory dump, or when attaching to a running process, Visual Studio can inspect the loaded modules (the running exe and any loaded dlls) directly. With a minidump all that is stored is the name of the module and some further info in a
MINIDUMP_MODULE structure. Visual Studio extracts the name, timestamp and image size and will use all three to try and find the correct module in the symbol store. It will also use that information to verify any module before it loads it.
Let’s load a dmp file and independently find that data to try and debug a failed load (if you want to give it a go and don’t have a minidump handy, you can create one by right-clicking on any process in the
Details pane of Task Manager and selecting
Create dump file from the menu). Using Hex Editor Neo do the following:
0x00005A7C) as we are now going to find the
MINIDUMP_MODULEthat refers to this
ULONG32directly before your first hit is the TimeDateStamp of the dll that VS is looking for. In this case
ULONG32directly before that is the
ULONG32directly before that is the
SizeOfImage, in this case
00D03600or 3592192 bytes
VS Uses the concatenation of the
SizeOfImage as the unique key to look for in the symbol store and will then verify both these values in any binary module, it attempts to load. This is the first part of ensuring that VS does not load the wrong symbols for the debugging session.
Now we can look in the location Visual Studio will expect to find the binary module it is trying to load. Go to your symbol store (the location your CI build spits out symbols) and find a sub-directory with the name of the module you are looking for – in our case this is a folder called
d3d11.dll. If you do not have a symbol store to look at, you can look in your local symbol cache which is normally in
%LOCALAPPDATA%\Temp\SymbolCache as this stores cached symbols and binaries in the same format. In the d3d11.dll folder, we can see a number of folders with hex values as names. We are looking for one that is the concatenation of the two values we found earlier:
SizeOfImage (note that for the
SizeOfImage the least significant byte is omitted). In our case that gives us
FECB8D5ED03600 and we can see that there is the folder we want.
It is when this folder is missing that Visual Studio asks you for the location of the binary file. Now we can open the module and look for the matching information inside the binary, which is what Visual Studio uses to verify that it has, in fact, found the correct binary module. It is this step failing that leads to the most hair-pulling as you point Visual Studio at the file you think it should want and it just refuses to load it! At least now you will know why it isn’t loading (and if you really want to force it to load, then you will know what bit of the file to edit to force it)!
If you are wondering where all these folders come from, they are generated by a tool called SymStore that can be run directly from your CI build or from the PublishSymbols task in TFS / Azure DevOps. For more information see the end of this post.
Again we open the file in HexEditorNeo but this time instead of looking at the raw hex we can use the Structure Viewer panel (you may have to enable advanced mode to see this) to explore the PE headers. You can also use another tool like CFF explorer to do this. We now simply navigate to dos_header → e_lfanew → FileHeader ->TimeDateStamp to read the
TimeDateStamp value. Clicking on it will take us to the corresponding location in the hex editor which allows us to easily verify that the hex value matches.
Next we navigate to dos_header → e_lfanew → OptionalHeaderSelector → OptionalHeader → SizeOfImage to read the
SizeOfImage value. Again clicking on it will take us to the matching part of the binary file so we can directly compare the hex values and check for a match.
If you have been forced to rebuild a module to match a historic crashdump then this value should match while the timestamp will be different. Simply modify the
TimeDateStamp in your newly built binary file to match the expected value (or edit the dump file to match your newly built module) and Visual Studio will load your new binary file and then fetch your newly created symbols!
Finally, a reminder that if we are debugging a live process or a full memory dump, then Visual Studio can directly inspect the module in memory and so these first two steps are not required and we can move straight on to the next step.
Once we have the correct module loaded things get a little simpler. This is because Visual Studio has moved from working with signatures inferred from general-purpose module headers into working with explicit debug information. In this case, we start by looking in the
DebugDirectory part of the PE headers which contains a reference to the exact version of the symbols that Visual Studio wants to load.
Again we can use the Structure Viewer tab or other PE structure explorer to investigate the
DebugDirectory and now we are looking for the raw data stored there. So we navigate to DebugDirectory → AddressOfRawData → Data → data and click on it to move to the corresponding location in the main hex editor. We can see that this section starts with the magic string “RSDS” followed by 16 bytes of GUID followed by the age value (normally “1”) as a uint32 and finally the original file path of the symbol file. Note these values down and we can now use them to get the correct symbols to load! For this example the bytes are
F82BEBBA3FFA9C4D84643895BFA0295E for the GUID and
01000000 for the age.
If you are debugging the process on the machine where the binary file was built, things are a little simpler as we have just found the path that leads directly to the symbol files to load! But what if the binary was built on another machine, say a CI server? Well, again Visual Studio will turn to the symbol store.
Visual Studio will now look for a folder with the same name as the symbols file it is looking for – in our case this will be
d3d11.pdb. If you open this folder you will find a directory full of GUID values. The matching symbols will be found in a folder that’s name is the concatenation of the GUID from the previous step and the age value (with no leading zeros). To decode the GUID remember that it is in the format
uint32-uint16-uint16-byte so you will need to re-order the first 8 hex values to get
BAEB2BF8-FA3F-4D9C-84643895BFA0295E (the hyphens are just for readability, don’t put them in the path) and put a 1 on the end to get
BAEB2BF8FA3F4D9C84643895BFA0295E1. Looking in this folder we can find the file
d3d11.pdb which is the symbols file we were looking for!
I bet you are wondering if Visual Studio will just blindly load these symbols or if it first confirms that they are indeed the correct symbols for the running process. You are, of course, correct – there is one final validation step before Visual Studio will actually use the symbols it has just found (or if both the above searches draw a blank, the symbols file you have just asked it to load from a file dialog). Visual Studio confirms they are the correct symbols by looking for the GUID inside the PDB file itself. From my own cursory investigations, it seems to be stored at offset
0x0000500C in the file, with the age at
0x00005008. However, it is easy enough to just use a hex editor and search for the entire set of bytes that make up the GUID – (in this case,
F82BEBBA3FFA9C4D84643895BFA0295E). If it doesn’t match anywhere in the file then these are definitely not the right symbols!
So that is how Visual Studio does its magic and matches up symbols to a minidump. Hopefully now if one of those steps doesn’t work (due to a corrupt file, symbol store failure, or some other CI shenanigans), you’ll have something to go on when trying to diagnose the issue!
For a more in-depth delve into the binary structures involved and how they are used by different versions of Visual Studio read this blog post on debuginfo.com
For more information on source indexing and symbol, servers read this article on CodeProject