Tombstones

Understanding Tombstones

When a native C/C++ process in Android crashes due to a fatal signal (like a segmentation fault or an abort), the system generates a post-mortem crash report known as a Tombstone. Tombstones are the primary diagnostic tool for debugging native crashes in AOSP and NDK applications.

Tombstone File Location and Format

Tombstones are plain text files stored in the /data/tombstones/ directory on the device. They are named sequentially: tombstone_00, tombstone_01, up to a system-defined limit (usually 10 or 50), after which they rotate.

You can access them via ADB:

adb shell ls -l /data/tombstones/
adb pull /data/tombstones/tombstone_00 .

The tombstone format is standardized and contains sections detailing the build fingerprint, the crashed process, the thread that caused the crash, the signal received, register states, and a backtrace.

Reading the Crash Reason: Signal, Fault Address, and Registers

The top of the tombstone provides the immediate cause of the crash.

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'google/pixel_6/oriole:13/TP1A.220624.021/8805294:user/release-keys'
Revision: 'MP1.0'
ABI: 'arm64'
Timestamp: 2023-10-27 10:15:30+0000
pid: 12345, tid: 12360, name: RenderThread  >>> com.example.app <<<
uid: 10150
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0

• pid and tid: Identifies the exact process and thread that crashed.
• signal: The POSIX signal that terminated the process. Common signals include:
  • SIGSEGV (11): Segmentation fault (invalid memory access).
  • SIGABRT (6): Abort (often triggered by an assertion failure or Scudo allocator panic).
  • SIGILL (4): Illegal instruction (corrupted memory or executing data).
• fault addr: The specific memory address the thread attempted to access. 0x0 indicates a classic null pointer dereference.

Below this, the tombstone lists the state of the CPU registers (e.g., x0-x30, sp, pc on ARM64) at the exact moment of the crash.

Reading the Backtrace

The most critical part of the tombstone is the backtrace (call stack). It shows the sequence of function calls that led to the crash.

backtrace:
    #00 pc 000000000004a1b4  /system/lib64/libc.so (abort+164)
    #01 pc 000000000004f2c0  /system/lib64/libc.so (__assert2+112)
    #02 pc 0000000000012348  /data/app/~~XYZ==/com.example.app-ABC==/lib/arm64/libnative-lib.so (MyClass::doWork()+80)
    #03 pc 0000000000012500  /data/app/~~XYZ==/com.example.app-ABC==/lib/arm64/libnative-lib.so (Java_com_example_app_MainActivity_startWork+120)

The trace starts at #00 (the function where the crash actually occurred) and works its way backward. Notice the pc (program counter) addresses. These are relative offsets within the shared library (.so), which are necessary for later symbolization.

The Crash Handling Architecture: debuggerd and crash_dump

Android handles native crashes using a client-server architecture:

1. Signal Handler Setup: When a process starts (typically forked by the Zygote), Bionic sets up signal handlers for fatal signals (SIGSEGV, SIGABRT, etc.).
2. debuggerd: A system daemon that listens for crash notifications.
3. Crash Execution: When a fatal signal occurs, the process's signal handler contacts debuggerd.
4. crash_dump: debuggerd spawns a helper process called crash_dump (e.g., crash_dump64).
5. Ptrace and Analysis: crash_dump uses the ptrace system call to attach to the dying process. It freezes the process, reads its memory, unwinds the stack, formats the tombstone text, writes it to /data/tombstones/, and dumps a summarized version to Logcat. Finally, it lets the kernel terminate the dying process.