Typically, when you receive a crash report from iTunes connect or a 3rd party service that provides mobile crash collection and reporting (like Apteligent), the service will take care of symbolicating the crash for you. If you did not upload symbols though, you may find yourself with an unsymbolicated crash and nothing else to go on. Such a crash file is not very useful for debugging an issue that may be impacting a large number of your users.

In this case, you must symbolicate the crash report by resolving the backtrace stack addresses to symbols in order to garner useful information about the crash.

Luckily, it’s entirely possible (if a bit tedious) to manually symbolicate a crash report. This article will outline what you information you need, show you how to interpret the crash report, and go over some of the tools available on OSX and XCode to symbolicate a crash.

There are only two sections in the crash report that are relevant for symbolicating the exception trace. The first is the ‘Exception Backtrace’ section. This shows the call stack for your application at the time of the crash. This particular crash log snippet is showing the backtrace of a crash inside our ApteligentExampleApp application. (For this app, crashing really is a feature, not a bug)

Last Exception Backtrace:
0   CoreFoundation 0x000000018708b100 0x186f80000 + 1093888
1   libobjc.A.dylib            0x00000001939441fc 0x19393c000 + 33276
2   CoreFoundation 0x000000018708b040 0x186f80000 + 1093696
3   ApteligentExampleApp 0x000000010003acc4 0x10002c000 + 60612

The second section of interest is the ‘Binary Images’ section, at the bottom of the crash report, which gives you additional useful information. This section lists the binaries that were loaded at the time of the crash.

Binary Images:
0x10002c000 - 0x1000dffff ApteligentExampleApp arm64  <3759a98e880336108b1a799afa3c1adc> /var/mobile/Applications/46FB38F8-0E69-459F-B96A-CEEA21B77D55/ApteligentExampleApp.app/ApteligentExampleApp

Unfortunately, this crash report is unsymbolicated. We can see the point where our application crashed (line 3 in the exception backtrace), but it is missing details like function names, function parameters and line numbers that will help a developer debug the issue. In order to turn the various addresses in the crash report into something human-readable, we need to map these addresses to symbols. For that, we will need debug symbols (a dSYM file) and a symbolication tool along with information gathered from the crash report itself.

Collecting the Information Needed

Here’s an example line that needs to be symbolicated. We know that this is the point at which the program failed, but the address itself doesn’t tell us anything useful.

3  ApteligentExampleApp  0x000000010003acc4  0x10002c000 + 60612

This line in the exception backtrace gives you the stack address, the binary load address within virtual memory for the application, as well as an offset. This last value is simply the difference between the stack address and the load address.

Nearer to the bottom of the crash report, you will see the binary images section. Typically, the crashed application will be at the top of the list. This entry will give you the load address (again), the dSYM UUID for this crash, and the architecture of the system that the application crashed on.

Binary Images:
0x10002c000 - 0x1000dffff +ApteligentExampleApp arm64 <3759a98e880336108b1a799afa3c1adc>

Now we can gather up everything needed to symbolicate this line. Using this data, you can symbolicate any stack address within a particular crash.

Verifying your Symbols File

A dSYM file is an ELF file that contains DWARF debug information (among other things) for your application. If you have the “DWARF with dSYM File” option set in XCode, a dSYM file is generated by the compiler and is stored alongside your build.

If you want to symbolicate a particular crash, you need to locate the matching dSYM file. It is a good idea to have some sort of archiving mechanism to store the dSYM and application binary for every release to the app store, because it is important to match up your crash dSYM UUID with the correct dSYM file. If the UUID does not match exactly, your symbolicated results will be unreliable at best. The crash report will tell you what dSYM you need in order to symbolicate. If you’re not sure if your dSYM matches the crash, you can check the UUID using dwarfdump.

dwarfdump -u ApteligentExampleApp.dSYM
UUID: 3759A98E-8803-3610-8B1A-799AFA3C1ADC (arm64) ApteligentExampleApp.dSYM

 

Getting the Slide Value

For some tools, you may need to provide the slide value instead of a load address. ATOS will handle the slide computation for you if you give it a load address (0x10002c000) and a stack address (0x10003acc4). However, dwarfdump and lldb take the file address (0x10000ECC4), so you will need to take the slide value into account for these tools.

One way to get the slide value from the dSYM is to use “otool”, which is available with the XCode developer tools on OSX.

You need to look for the LC_SEGMENT_64 (arm64) or LC_SEGMENT (armv7, armv7s) segment and the “vmaddr” entry. For iOS, this is usually 0x4000 for 32-bit, and 0x100000000 for 64-bit architectures, but this is liable to change.

otool -l ApteligentExampleApp.dSYM > ApteligentExampleApp.otool.output
Load command 3
cmd LC_SEGMENT_64
cmdsize 1032
segname __TEXT
vmaddr 0x0000000100000000

Now that we’ve compiled all the information needed, we can start symbolicating the stack trace.

Symbolicating a Crash Report

Symbolicating with ATOS

ATOS is a console symbolication tool from Apple. It converts numeric addresses to symbolicated strings from a binary image.

This is the easiest tool to use to quickly get symbolicated output on OSX. Now that you have all of the information collected from your crash file, you simply need to plug in the addresses and build information and you should get a symbolicated line. ATOS can process multiple addresses at a time, so you can enter every stack address from the stack trace if you choose.

atos -arch <architecture> -o <binary filename> -l <load address> <stack address 1> <stack address 2> ...
atos -arch arm64 -o ApteligentExampleApp.dSYM -l 0x10002c000 0x000000010003acc4
-[ApteligentExampleClass buggyFunction] (in ApteligentExampleApp.dSYM) (ApteligentExampleClass.m:181)

Symbolicating with lldb

lldb is the default debugger in XCode on OSX and can be used to symbolicate lines from a crash. lldb is again included with XCode for OSX, and there are ports for Linux, FreeBSD and Windows available. You can get it from the llvm project site here.

If you’d like more information about what lldb can do, you can read up on it here.

(lldb) target create --arch arm64 ApteligentExampleApp.dSYM
Current executable set to ApteligentExampleApp.dSYM' (arm64).
(lldb) image lookup --address 0x10000ECC4
Address: ApteligentExampleApp.dSYM[0x000000010000ecc4] (ApteligentExampleApp.dSYM.__TEXT.__text + 29916)
Summary: ApteligentExampleApp.dSYM`-[ApteligentExampleClass buggyFunction] + 68 at ApteligentExampleClass.m:181

Symbolicating with Dwarfdump

Dwarfdump is a utility which dumps DWARF debug information from an ELF object – for iOS this is typically a dSYM file. Dwarfdump is an extremely verbose tool which is often used to debug DWARF debug information producers (like the compiler in XCode) or verifying the output of symbolication tools like ATOS. For simply symbolicating a crash dump, using this tool is utterly overkill, but sometimes you might want to dig a bit further into the DWARF debug information.

Dwarfdump uses the file address (0x10000ECC4) to locate a matching subprogram “Debug Information Entry”. This entry has DWARF attributes which give you information about the subprogram. For Objective C, this subprogram entry will usually represent a function inside a class, and you can get information like the line number, filename along with the class/function name.

A version of Dwarfdump is shipped with the XCode developer tools, but you can also get it, along with libdwarf, from the libdwarf project page here.

dwarfdump --lookup 0x10000ECC4 --arch arm64 ApteligentExampleApp.dSYM
----------------------------------------------------------------------
File: ApteligentExampleApp.dSYM (arm64)
----------------------------------------------------------------------
Looking up address: 0x000000010000ecc4 in .debug_info... found!

0x000516d2: Compile Unit: length = 0x00000e9f  version = 0x0002  abbr_offset = 0x00000000  addr_size = 0x08  (next CU at 0x00052575)

0x000516dd: TAG_compile_unit [99] *
AT_producer( "Apple LLVM version 7.0.2 (clang-700.1.81)" )
AT_language( DW_LANG_ObjC )
AT_name( "/Users/kcrawford/src/apteligent-example-apps/ios/ApteligentExampleApp/ApteligentExampleClass.m" )
AT_stmt_list( 0x00008e5a )
AT_comp_dir( "/Users/kcrawford/src/apteligent-example-apps/ios" )
AT_APPLE_major_runtime_vers( 0x02 )
AT_low_pc( 0x000000010000d704 )
AT_high_pc( 0x000000010000f234 )

0x00051cbd:     TAG_subprogram [116] *
AT_low_pc( 0x000000010000ec80 )
AT_high_pc( 0x000000010000ecd0 )
AT_frame_base( reg29 )
AT_object_pointer( {0x00051cdb} )
AT_name( "-[ApteligentExampleClass buggyFunction]" )
AT_decl_file( "/Users/kcrawford/src/apteligent-example-apps/ios/ApteligentExampleApp/ApteligentExampleClass.m" )
AT_decl_line( 179 )
AT_prototyped( 0x01 )
Line table dir : '/Users/kcrawford/src/apteligent-example-apps/ios/ApteligentExampleApp'
Line table file: 'ApteligentExampleClass.m' line 181, column 1 with start address 0x000000010000ecc4
Looking up address: 0x000000010000ecc4 in .debug_frame... not found.

SymbolicateCrash

So far, we’ve looked at tools that symbolicate specific addresses within a crash, or at best a series of addresses in the case of ATOS. To make this process easier, Apple ships a script with XCode that expedites the symbolication process of a crash report in its entirety. If you have a dSYM, your app binary and a crash report, this is probably the easiest method of symbolication. You don’t have to worry about any of the addresses – this script will parse the whole crash dump file and use ATOS to resolve all of the addresses into symbols for you.

  • Locate the “symbolicatecrash” on your system
cd /Applications/Xcode.app
find . -name symbolicatecrash
  • Export the DEVELOPER_DIR environment variable, if it doesn’t exist
export DEVELOPER_DIR="/Applications/Xcode.app/Contents/Developer"
  • Copy your .app binary, the crash report, and the .dSYM file to a temporary folder (ex. ~/tmp)
  • Run the script like our example below
/Applications/Xcode.app/Contents/SharedFrameworks/DVTFoundation.framework/Versions/A/Resources/symbolicatecrash -v ApteligentExampleApp.crash ApteligentExampleApp.app.dSYM/

If all goes well, the script should symbolicate your entire crash file and output the result to your terminal window. This script doesn’t do anything you couldn’t do manually with ATOS or another tool, but it will get you what you need much quicker.