Complete Guide to Android AOSP Log Analysis and Debugging

Introduction

Log analysis is the cornerstone of effective Android AOSP debugging. This comprehensive guide covers all essential log types, debugging commands, keywords for different issues, and systematic troubleshooting approaches for Android Open Source Project (AOSP) development. Whether you're dealing with boot failures, application crashes, system performance issues, or security policy violations, understanding logs is crucial for successful AOSP development and customization.

Core Log Types in Android AOSP

1. Logcat Logs

Logcat is the primary logging system for Android applications and framework components.

Log Buffers:

• main: Application logs and most system messages
• system: Android OS and system service messages
• crash: Crash-specific logs and tombstone references
• radio: Radio and telephony-related logs
• events: Binary-formatted system event logs
• kernel: Kernel messages (requires root access)

Log Levels (Priority):

• V (Verbose): Detailed debugging information
• D (Debug): Debug messages for development
• I (Info): General information messages
• W (Warning): Warning conditions
• E (Error): Error conditions
• F (Fatal): Fatal errors that cause crashes

Basic Logcat Commands:

# View all logs with timestamps
adb logcat -v time

# Filter by log level (show warnings and above)
adb logcat *:W

# Save logs to file
adb logcat > logcat.txt

# View specific buffer
adb logcat -b system

# Multiple buffers
adb logcat -b main -b system -b crash

# Clear log buffer
adb logcat -c

# View logs from specific app
adb logcat | grep "com.example.app"

2. Kernel Logs (dmesg/kmsg)

Kernel logs contain low-level system information, hardware driver messages, and kernel panics.

Access Methods:

# Current kernel messages
adb shell dmesg

# Continuous kernel log monitoring  
adb shell cat /proc/kmsg

# Export kernel logs
adb shell dmesg > kernel.log

# Kernel log from previous boot
adb shell cat /proc/last_kmsg

# For newer Android versions (7.0+)
adb shell cat /sys/fs/pstore/console-ramoops

3. Tombstone Files

Tombstones are generated when native processes crash and contain detailed crash information.

Location and Structure:

• Path: /data/tombstones/tombstone_XX
• Contains: Process info, signal details, CPU registers, call stack, memory maps
• Retention: Up to 10 tombstones (cycling 00-09)

Accessing Tombstones:

# List available tombstones
adb shell ls /data/tombstones/

# Pull specific tombstone
adb pull /data/tombstones/tombstone_01 .

# Pull all tombstones
adb pull /data/tombstones .

4. ANR (Application Not Responding) Traces

ANR traces help debug application unresponsiveness issues.

ANR Trace Locations:

• Path: /data/anr/traces.txt or /data/anr/anr_*
• Contains: Thread dumps, stack traces, lock information

Collecting ANR Traces:

# Pull ANR traces
adb pull /data/anr/traces.txt .

# Monitor ANR events in logcat
adb logcat ActivityManager:E *:S

5. System Event Logs

System events provide binary-formatted information about system activities.

# View system events
adb logcat -b events -v time

# Specific event monitoring
adb logcat -b events | grep "am_proc_start"

Essential Debugging Commands

ADB (Android Debug Bridge) Commands

Device Management:

# List connected devices
adb devices

# Connect over network
adb connect <ip>:5555

# Root access (userdebug/eng builds)
adb root

# Remount system partition
adb remount

# Reboot into different modes
adb reboot
adb reboot-bootloader
adb reboot recovery

File Operations:

# Push files to device
adb push <local> <remote>

# Pull files from device
adb pull <remote> <local>

# Shell access
adb shell

# Execute single command
adb shell <command>

dumpsys - System Service Information

The dumpsys tool provides comprehensive information about system services.

Key dumpsys Commands:

# List all available services
adb shell dumpsys -l

# Battery information
adb shell dumpsys battery

# Memory information
adb shell dumpsys meminfo

# Activity manager state
adb shell dumpsys activity

# Window manager state  
adb shell dumpsys window

# Package manager information
adb shell dumpsys package

# Input system state
adb shell dumpsys input

# Graphics information
adb shell dumpsys gfxinfo <package>

# Network statistics
adb shell dumpsys netstats

System Properties

# View all properties
adb shell getprop

# Specific property
adb shell getprop ro.build.version.release

# Set property (requires root)
adb shell setprop <key> <value>

# Enable verbose logging for specific components
adb shell setprop log.tag.ActivityManager VERBOSE

Issue-Specific Keywords and Debugging Approaches

Boot Issues

Boot problems are among the most challenging to debug in AOSP development.

Keywords to Search:

• "kernel panic"
• "Unable to mount"
• "init: service"
• "Fatal signal"
• "Kernel Oops"
• "out of memory"
• "SELinux"
• "mount failed"
• "init: critical process"

Boot Debugging Commands:

# Early boot kernel logs
adb shell cat /proc/last_kmsg

# Boot animation logs
adb logcat | grep -i "bootanim"

# Init process logs
adb logcat | grep -i "init"

# Mount point information
adb shell cat /proc/mounts

# Check boot reason
adb shell getprop sys.boot.reason

Common Boot Issues:

1. Kernel Panic: Hardware initialization failures, driver issues
2. Mount Failures: Partition corruption, filesystem errors
3. Init Process Crashes: Service configuration errors
4. SELinux Denials: Security policy violations

Application Crashes

Application crashes require systematic log analysis to identify root causes.

Keywords to Search:

• "FATAL EXCEPTION"
• "AndroidRuntime"
• "Force finishing activity"
• "Process <pid> crashed"
• "segmentation fault"
• "abort()"
• "signal 11"

Crash Debugging Process:

# Monitor crash events
adb logcat AndroidRuntime:E *:S

# Java/Kotlin crashes
adb logcat | grep -E "(FATAL|AndroidRuntime)"

# Native crashes
adb logcat | grep -E "(DEBUG|tombstone)"

# Application lifecycle events
adb logcat ActivityManager:I *:S

ANR (Application Not Responding) Issues

ANRs occur when the main thread is blocked for more than 5 seconds.

Keywords to Search:

• "ANR in"
• "Application is not responding"
• "Input dispatching timed out"
• "Broadcast of intent"
• "Service not responding"

ANR Types and Debugging:

1. Input Dispatch Timeout (5 seconds):

# Monitor input events
adb logcat WindowManager:I *:S

# Check main thread activity
adb logcat | grep "Input dispatch"

2. Broadcast Timeout (10 seconds foreground, 60 seconds background):

# Broadcast receiver monitoring
adb logcat ActivityManager:I | grep "Broadcast"

3. Service Timeout:

# Service lifecycle events
adb logcat ActivityManager:I | grep "Service"

Google Play Services (GMS) Issues

GMS-related problems require specific logging approaches.

Keywords to Search:

• "com.google.android.gms"
• "GooglePlayServices"
• "Gms"
• "Play Services"
• "ConnectionResult"
• "GoogleApiClient"

GMS Debugging Commands:

# Enable GMS verbose logging
adb shell setprop log.tag.Games VERBOSE

# GMS-specific logs
adb logcat | grep -i "gms"

# Play Services logs
adb logcat -s GooglePlayServices

# Authentication issues
adb logcat | grep -i "auth"

Performance Issues

System performance problems require comprehensive analysis.

Keywords to Search:

• "skipped frames"
• "GC_"
• "lowmemorykiller"
• "OutOfMemoryError"
• "watchdog"
• "ANR"
• "slow"

Performance Debugging Tools:

1. Systrace:

# Capture system trace
python systrace.py sched freq idle am wm gfx view binder_driver -t 10 -b 96000

# Specific categories
python systrace.py -l  # List available categories

2. Memory Analysis:

# System memory information
adb shell dumpsys meminfo

# Process-specific memory
adb shell dumpsys meminfo <package>

# Native heap information
adb shell dumpsys meminfo --unreachable <process>

SELinux Policy Issues

SELinux denials are common in AOSP development and require careful policy adjustments.

Keywords to Search:

• "avc: denied"
• "SELinux"
• "audit"
• "type=1400"
• "scontext"
• "tcontext"

SELinux Debugging:

# Monitor SELinux denials
adb logcat | grep -i "avc"

# Kernel SELinux messages
adb shell cat /proc/kmsg | grep avc

# Check SELinux mode
adb shell getenforce

# Set permissive mode (for debugging)
adb shell setenforce 0

# Disable denial rate limiting
adb shell auditctl -r 0

Understanding SELinux Denials:

avc: denied { read write } for pid=1234 comm="myprocess" 
path="/dev/mydevice" scontext=u:r:mydomain:s0 
tcontext=u:object_r:device:s0 tclass=chr_file

• scontext: Source context (process attempting access)
• tcontext: Target context (resource being accessed)
• tclass: Object class (file, directory, socket, etc.)
• Action: Permission being requested (read, write, execute)

Network and Connectivity Issues

Network problems require monitoring various system components.

Keywords to Search:

• "ConnectivityService"
• "NetworkAgent"
• "WifiManager"
• "TelephonyManager"
• "NetworkTimeUpdateService"
• "DnsManager"

Network Debugging:

# Network statistics
adb shell dumpsys netstats

# Wi-Fi information
adb shell dumpsys wifi

# Connectivity service
adb shell dumpsys connectivity

# Network interfaces
adb shell netcfg
adb shell ip addr show

Advanced Debugging Techniques

Memory Debugging

Memory issues require specialized tools and approaches.

Memory Profiling Tools:

# Address Sanitizer (ASan) - requires special build
# Add to Android.mk: LOCAL_SANITIZE := address

# Malloc debug
adb shell setprop libc.debug.malloc.options backtrace

# Heap profiling with perfetto
adb shell perfetto -c heap_profile.cfg -o trace.pb

Performance Profiling

Comprehensive performance analysis using multiple tools.

Systrace Analysis:

1. CPU Scheduling: Look for thread starvation
2. Frame Drops: Identify rendering bottlenecks
3. I/O Wait: Check for storage/network delays
4. Memory Pressure: Monitor garbage collection events

Native Debugging

Debugging native code requires specific approaches.

GDB/LLDB Debugging:

# Attach to running process
lldbclient.py -p <pid>

# Debug binary
lldbclient.py -r /system/bin/mybinary

# Core dump analysis
lldbclient.py -c core.dump

Bug Report Generation and Analysis

Comprehensive Bug Reports

Bug reports contain all essential debugging information.

Generating Bug Reports:

# Full bug report
adb bugreport bugreport.zip

# Legacy text format
adb bugreport > bugreport.txt

# From device UI
Settings > Developer Options > Take bug report

Bug Report Contents:

• Complete logcat output
• dumpsys information from all services
• System properties
• Process list and memory usage
• Network configuration
• Battery information

Log File Locations

Understanding log file locations is crucial for manual analysis.

Standard Log Locations:

/data/anr/              # ANR traces
/data/tombstones/       # Native crash dumps
/data/dontpanic/        # Panic logs (some devices)
/sys/fs/pstore/         # Persistent store logs
/proc/last_kmsg         # Previous boot kernel log
/dev/kmsg              # Current kernel messages
/proc/kmsg             # Kernel message buffer

Systematic Debugging Workflow

1. Initial Assessment

When encountering any issue:

1. Identify the symptom: Crash, hang, performance degradation
2. Reproduce the issue: Create consistent reproduction steps
3. Gather baseline information: Device state, build information
4. Enable relevant logging: Increase verbosity for affected components

2. Log Collection

Systematic log gathering approach:

# Pre-issue setup
adb logcat -c  # Clear logs
adb shell dmesg -c  # Clear kernel buffer

# Reproduce issue while logging
adb logcat -v threadtime > logs_during_issue.txt &
adb shell cat /proc/kmsg > kernel_during_issue.txt &

# Post-issue collection  
adb bugreport post_issue_bugreport.zip
adb pull /data/tombstones .
adb pull /data/anr .

3. Log Analysis Strategy

Systematic analysis approach:

1. Start with fatal errors: Search for "FATAL", "crash", "panic"
2. Work backwards from symptoms: Find the triggering event
3. Follow the timeline: Understand the sequence of events
4. Correlate across log types: Match timestamps between different logs
5. Focus on relevant processes: Filter by PID or component names

4. Common Analysis Patterns

For Crashes:

# Find the crash
grep -i "fatal\|crash\|exception" logcat.txt

# Get the process info  
grep -B 20 -A 10 "FATAL EXCEPTION" logcat.txt

# Check for related tombstone
grep "tombstone" logcat.txt

For Boot Issues:

# Check init process
grep "init:" kernel.log

# Look for mount failures
grep -i "mount\|failed" kernel.log  

# SELinux issues
grep "avc:" kernel.log

Build System Debugging

AOSP Build Issues

Build problems require understanding of the Soong build system.

Common Build Keywords:

• "FAILED"
• "error:"
• "No such file"
• "undefined reference"
• "permission denied"

Build Debugging Commands:

# Show detailed build commands
make showcommands

# Build with verbose output
make -j1 showcommands

# Clean build for specific module
make clean-<module>

# Check build environment
printconfig

Best Practices for Log Analysis

1. Log Management

Effective log collection and organization:

• Use timestamps: Always include time information (-v time)
• Buffer management: Use appropriate buffer sizes (-b option)
• Focused collection: Enable only necessary log levels during analysis
• Structured storage: Organize logs by date, device, and issue type

2. Search Strategies

Efficient log searching techniques:

# Case-insensitive search with context
grep -i -A 5 -B 5 "keyword" logfile.txt

# Multiple keywords
grep -E "keyword1|keyword2|keyword3" logfile.txt

# Timestamp filtering
grep "2024-01-15 14:" logfile.txt

# Process-specific filtering  
grep "pid: 1234" logfile.txt

3. Automation Tools

Creating scripts for repetitive debugging tasks:

#!/bin/bash
# Log collection script
DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p logs_$DATE

adb logcat -v threadtime > logs_$DATE/logcat_$DATE.txt &
adb shell cat /proc/kmsg > logs_$DATE/kernel_$DATE.txt &
adb bugreport logs_$DATE/bugreport_$DATE.zip

echo "Logs being collected in logs_$DATE/"

Conclusion

Effective AOSP debugging requires mastery of multiple log types, commands, and analysis techniques. Success depends on:

1. Understanding log hierarchy: From kernel messages to application logs
2. Systematic approach: Following consistent debugging workflows
3. Tool proficiency: Mastering adb, dumpsys, systrace, and other tools
4. Pattern recognition: Identifying common failure modes and their signatures
5. Cross-correlation: Connecting information across different log sources

By following this comprehensive guide and developing expertise in log analysis, you'll be well-equipped to diagnose and resolve complex issues in Android AOSP development. Remember that debugging is an iterative process—each issue teaches valuable lessons that improve your debugging capabilities for future challenges.

The key to successful AOSP debugging lies in understanding that logs are not just text files but detailed forensic evidence of system behavior. Master the art of reading this evidence, and you'll unlock the ability to solve even the most challenging Android system issues.