Building a Voice-Controlled Assistant for LineageOS: Part 1 - The Foundation

If you've ever used Google Assistant, Siri, or Bixby, you know how convenient voice control can be. But what if you want that same experience on LineageOS, without relying on Google's services? That's exactly what I set out to build - a fully functional, privacy-focused voice assistant that works directly with your device hardware.

After months of development and testing on my Redmi Note 7 Pro running LineageOS 20, I've successfully integrated a voice-controlled assistant with over 50 commands. In this three-part series, I'll walk you through everything: from the initial architecture decisions to the final implementation details.

Why Build This?

The motivation was simple: I wanted hands-free control of my device without sending data to cloud services. Most voice assistants require internet connectivity and send your voice data to remote servers. For a privacy-focused ROM like LineageOS, this doesn't align well with the philosophy.

I also wanted something that worked offline, understood natural language (not just exact commands), and could control device hardware directly - Bluetooth, WiFi, camera, flashlight, alarms, and more. The goal was to make it feel natural, like talking to a friend who understands what you mean, even if you don't phrase things perfectly.

The Architecture Challenge

The biggest challenge wasn't implementing individual features - it was designing a system that could understand natural language without requiring massive ML models or cloud services. I needed something lightweight that could run entirely on-device.

Here's the architecture I settled on:

Voice Input -> Speech Recognition -> Text Conversion
    v
Natural Language Understanding (Pattern Matching)
    v
Intent Classification (50+ different commands)
    v
Parameter Extraction (time, location, app names)
    v
Hardware Control / Intent Execution
    v
User Feedback

The key insight was using a pattern-based approach for natural language understanding. Instead of training a massive neural network or integrating ML models (which would require significant build system changes and dependencies), I created a database of intent patterns and used word-based similarity matching. This approach is lightweight, fast, and works entirely on-device without requiring TensorFlow, PyTorch, or other ML frameworks.

Core Components

The project consists of three main components:

1. MainActivity.java - The main UI and command execution engine This handles the user interface, processes commands, and executes actions. It's about 1,700 lines of code that manages everything from Bluetooth control to alarm setting.

2. AICommandProcessor.java - The natural language understanding engine This is where the magic happens. It takes raw text input like "turn on bluetooth" or "set alarm for 7 AM" and figures out what the user wants. It uses pattern matching with confidence scoring to handle variations in how people phrase commands.

3. AlwaysOnService.java - Background listening service For hands-free operation, I implemented a service that runs in the background, listening for the wake word "Hey Buddy". This uses Android's ForegroundService API to keep running even when the app isn't visible. The architecture is in place, though wake word detection currently uses simple string matching rather than a dedicated ML model.

The Speech Recognition Problem

One of the first hurdles I hit was speech recognition. Android's SpeechRecognizer API requires either Google Speech Services or a system-level speech recognition service. On LineageOS without GApps, this can be problematic.

I solved this with a two-pronged approach:

1. Use the system's speech recognition if available
2. Provide a text input fallback that works perfectly fine for all commands

The text input actually works great - you can type "turn on flashlight" and it works exactly the same as voice input. This makes the assistant accessible even when speech recognition isn't available.

Building for AOSP

Since this needed to be integrated into LineageOS, I built it as a system app using AOSP's build system. The Android.bp file configures it as a privileged system app with platform certificate, which gives it the permissions needed for direct hardware control.

Here's the build configuration:

android_app {
    name: "SimpleBuddyAssistant",
    srcs: [
        "MainActivity.java",
        "AICommandProcessor.java",
        "AlwaysOnService.java",
    ],
    platform_apis: true,
    certificate: "platform",
    privileged: true,
    system_ext_specific: true,
}

This ensures the app has system-level permissions for things like controlling Bluetooth, WiFi, and camera hardware directly, without opening settings screens.

The Intent Classification System

The heart of the natural language understanding is the intent classification system. I created a database of 50+ intents, each with multiple pattern variations. For example, the "BLUETOOTH_ON" intent matches patterns like:

• "turn on bluetooth"
• "enable bluetooth"
• "activate bluetooth"
• "bluetooth on"
• "open bluetooth"

When a user says something, the system:

1. Normalizes the input (lowercase, remove special characters)
2. Calculates similarity scores against all patterns
3. Selects the best match with a confidence score
4. Falls back to fuzzy matching if confidence is too low

The confidence scoring uses word-based matching - it counts how many words from the input match words in the pattern, then divides by the total word count. This gives a score between 0.0 and 1.0, where 0.6+ is considered confident enough to execute.

Parameter Extraction

Understanding the intent is only half the battle. Commands like "set alarm for 7 AM" need to extract the time (7) and period (AM). I implemented regex-based extraction for common parameters:

• Time: "7 AM", "9 PM" -> extracts hour and AM/PM
• Duration: "5 minutes", "1 hour" -> extracts number and unit
• App names: "open camera" -> extracts "camera"
• Locations: "navigate to work" -> extracts "work"
• Messages: "send message hello" -> extracts "hello"

This parsing happens in the extractParameters() method, which uses regex patterns and word parsing to pull out structured data from natural language.

Direct Hardware Control

One of the most satisfying parts of this project was implementing direct hardware control. Instead of opening settings screens, the assistant controls hardware directly using Android APIs:

• Flashlight: CameraManager.setTorchMode() - instant on/off
• Bluetooth: BluetoothAdapter.enable() / disable() - direct control
• WiFi: WifiManager.setWifiEnabled() - immediate toggle
• Volume: AudioManager.setStreamVolume() - precise control
• Brightness: Settings.System.putInt() - direct brightness setting

This makes the assistant feel responsive and powerful - when you say "turn on flashlight", it happens instantly, not after navigating through menus.

The Always-On Service Challenge

Implementing always-on listening was tricky. Android has strict restrictions on background services, especially for microphone access. Starting with Android 10, you need a foreground service with a persistent notification.

I implemented AlwaysOnService as a foreground service that:

1. Creates a notification channel (required for Android O+)
2. Shows a persistent notification ("Always-on listening active")
3. Continuously runs speech recognition in a loop
4. Detects the "Hey Buddy" wake word using simple string matching
5. Broadcasts commands to MainActivity for processing

The service uses START_STICKY so it automatically restarts if killed by the system. The continuous listening loop restarts after each recognition result or error, ensuring it's always ready to detect the wake word. Note that wake word detection uses basic string matching rather than a dedicated ML model - a more sophisticated approach would require additional integration work.

Permissions and Security

With great power comes great responsibility - and a lot of permissions. The app needs 30+ permissions for all the features:

• RECORD_AUDIO - for voice input
• BLUETOOTH_CONNECT, BLUETOOTH_SCAN - for Bluetooth control
• CAMERA, FLASHLIGHT - for camera and flashlight
• WRITE_SETTINGS - for system settings modification
• FOREGROUND_SERVICE_MICROPHONE - for always-on listening
• CALL_PHONE, SEND_SMS - for communication features
• ACCESS_FINE_LOCATION - for navigation
• And many more...

Since it's a system app with platform certificate, these permissions are automatically granted. But the app still respects Android's permission model and requests user confirmation where required (like Bluetooth enable).

What's Working So Far

After months of development, I've successfully implemented 50+ voice commands covering:

Core Utilities: Bluetooth, WiFi, Flashlight, Camera, Alarm, Timer, DND, Theme, Volume, Mobile Data, Hotspot, Airplane Mode, Screenshot, App Launcher, Settings

Personalization: Brightness, Wallpaper, Auto Rotate, NFC, Location, Battery Saver, Sound Profiles, Screen Recording, Notes, Lists

Communication: Phone Calls, SMS, Read Messages, Calendar Events, Reminders

Media: Play/Pause, Next/Previous, Media Control

Navigation: Weather, Navigation, Commute Time, Translation

Lifestyle: Pomodoro Timer, Automation

Each command works with multiple phrasings - you can say "turn on bluetooth", "enable bluetooth", "bluetooth on", or "activate bluetooth" and they all work the same way.

Testing and Debugging

Testing voice commands during development was interesting. I used ADB broadcasts to simulate voice input:

adb shell "am broadcast -a com.buddy.assistant.VOICE_COMMAND --es command 'hey buddy turn on bluetooth'"

This let me test commands quickly without actually speaking. The logs show the full processing pipeline:

MainActivity: Received voice command: hey buddy turn on bluetooth
AICommandProcessor: Final result: BLUETOOTH_ON with confidence: 0.6

The confidence scores help debug why certain commands aren't matching - if confidence is too low, the system falls back to fuzzy matching or returns "unknown command".

Challenges Encountered

Not everything went smoothly. Here are some issues I ran into:

SELinux Policy Conflicts: Initially tried to add custom SELinux rules for AI model access, but this caused policy freeze test failures. Had to remove those and work within existing policies.

Speech Recognition Availability: On devices without Google Speech Services, speech recognition might not be available. The text input fallback handles this gracefully.

Alarm App Detection: Different devices have different alarm apps. I implemented multiple fallback mechanisms - try the standard AlarmClock intent first, then try to launch the clock app directly, then fall back to settings.

Build System Integration: Getting the app to build correctly as a system app took some trial and error. The Android.bp file needed specific flags for platform APIs and privileged access.

Always-On Service Limitations: The always-on service architecture is implemented, but wake word detection currently uses simple string matching rather than a dedicated wake word detection model. This works but isn't as efficient or accurate as a proper ML-based wake word detector would be. Battery optimization is also an area for future improvement.

What's Next?

In Part 2, I'll dive deep into the implementation details - how the natural language processing actually works, the parameter extraction algorithms, and how each hardware control method is implemented. I'll also cover the always-on service architecture and how wake word detection works.

Part 3 will cover testing results, performance metrics, lessons learned, and future enhancements. While the current implementation uses pattern-based NLU and system speech recognition, future improvements could include offline speech recognition models like WhisperTiny and ML-based intent classification - though these weren't successfully integrated in this version due to build system complexities and dependency management challenges.

For now, the foundation is solid. The assistant understands natural language, controls hardware directly, and works entirely on-device. It's not perfect - there's always room for improvement - but it's functional and privacy-focused, which was the goal.

Stay tuned for Part 2, where we'll get into the technical nitty-gritty of how everything works under the hood.

Have questions or want to discuss the implementation? Drop a comment below or reach out on our forums. The source code and build instructions will be available in Part 3.