Tombstone & ANR Mastery

Tombstone and ANR Diagnostics

Application crashes and Application Not Responding (ANR) events are critical production issues. Android provides detailed telemetry to diagnose these failures at the system level.

Reading Complex Tombstones

When a native crash occurs (e.g., a SIGSEGV), the debuggerd daemon generates a "tombstone" file in /data/tombstones/.

A tombstone contains:

• The crashing signal and fault address.
• CPU registers at the time of the crash.
• A backtrace of the crashing thread.
• Backtraces of all other threads in the process.

When reading tombstones with multiple threads, it is crucial to identify the crashing thread. The top of the file explicitly names it: Cmdline: com.example.app (Crashing thread: Thread-42). You must examine the backtrace of that specific thread. Often, a crash in libc (like abort() or memcpy()) is just the symptom; the actual bug lies higher in the stack trace within the application's native libraries.

ANRs: Binder Deadlocks

An ANR occurs when the main UI thread is blocked for 5 seconds. The system drops a trace file in /data/anr/traces.txt.

A common and complex ANR cause is a Binder deadlock. Identification:

1. Open traces.txt. Find the main thread ("main" prio=5).
2. Observe if it is blocked in BinderProxy.transactNative().
3. Look at the system server trace within the same file. Find the corresponding Binder thread executing the incoming call.
4. If that system server thread is blocked waiting for a lock held by another thread, and that thread is waiting for an IPC back to the original application, you have a classic cyclic deadlock.

ANRs: Slow Broadcasts

Broadcast receivers must execute onReceive() within 10 seconds (for foreground broadcasts) or an ANR is triggered. Identification:

1. Check the logcat for the AMS warning: ActivityManager: Broadcast of Intent { act=android.intent.action.BOOT_COMPLETED } took 11000ms.
2. Open traces.txt and look at the main thread. If it is blocked in a database transaction, a network call, or Thread.sleep() inside the onReceive method, the solution is to move the work to a background thread using goAsync() or WorkManager.

System Server ANR Analysis

A System Server ANR is catastrophic: it results in a soft reboot (the Android logo appears). Diagnosing a System Server ANR requires examining /data/anr/traces.txt specifically for the system_server process. Common causes include:

• The ActivityManager thread is blocked holding the global ActivityManagerService lock, starving all other IPC requests.
• A critical system thread is blocked waiting for a slow hardware response via a HAL.
• A memory leak in the system server triggering continuous Garbage Collection pauses.

To force a system server crash for testing recovery logic:

adb shell kill -9 `pidof system_server`