Dive deep into the technical implementation of natural language understanding, parameter extraction, and direct hardware control. Includes code examples for pattern matching algorithms, intent classification, and Android API integration for Bluetooth, WiFi, camera, and more. Step-by-step guide for implementing voice commands.
Welcome back! In Part 1, we covered the architecture and foundation of the Buddy Assistant. Now let's dive into the technical implementation details - how natural language understanding works, how parameters are extracted, and how hardware control is implemented.
The Natural Language Understanding Engine
The AICommandProcessor class is where the magic happens. It's a static utility class that takes raw text input and figures out what the user wants to do.
The Intent Pattern Database
At the core is a massive HashMap that maps intent names to lists of pattern variations. Here's how it's structured:
private static final Map<String, List<String>> INTENT_PATTERNS = new HashMap<>();
static {
INTENT_PATTERNS.put("BLUETOOTH_ON", Arrays.asList(
"turn on bluetooth", "enable bluetooth", "activate bluetooth",
"switch on bluetooth", "bluetooth on", "open bluetooth",
"start bluetooth", "connect bluetooth"
));
INTENT_PATTERNS.put("ALARM_SET", Arrays.asList(
"set alarm", "create alarm", "add alarm", "schedule alarm",
"wake me up", "remind me", "alarm for", "set reminder",
"set alarm for", "create alarm for", "add alarm for",
"schedule alarm for", "wake me up at", "remind me at"
));
// ... 50+ more intents
}
I found that having multiple pattern variations is crucial. People phrase things differently - some say "turn on", others say "enable" or "activate". By including all common variations, the system becomes more robust.
The Processing Pipeline
When a command comes in, here's what happens:
public static CommandResult processCommand(String userInput) {
// Step 1: Normalize input
String normalizedInput = normalizeInput(userInput);
// Step 2: Calculate similarity for each intent pattern
CommandResult bestMatch = null;
float bestConfidence = 0.0f;
for (Map.Entry<String, List<String>> entry : INTENT_PATTERNS.entrySet()) {
String intent = entry.getKey();
List<String> patterns = entry.getValue();
for (String pattern : patterns) {
float similarity = calculateSimilarity(normalizedInput, pattern);
if (similarity > bestConfidence) {
bestConfidence = similarity;
bestMatch = new CommandResult(intent,
extractParameters(normalizedInput, intent),
similarity);
}
}
}
// Step 3: Fallback to fuzzy matching if confidence too low
if (bestConfidence < 0.6f) {
bestMatch = fuzzyMatch(normalizedInput);
}
return bestMatch;
}
The normalization step is important - it converts everything to lowercase, removes special characters, and normalizes whitespace. This makes matching more reliable.
Similarity Calculation Algorithm
The similarity algorithm is word-based, not character-based. Here's how it works:
private static float calculateSimilarity(String input, String pattern) {
String[] inputWords = input.split("\\s+");
String[] patternWords = pattern.split("\\s+");
int matches = 0;
for (String inputWord : inputWords) {
for (String patternWord : patternWords) {
if (inputWord.equals(patternWord) ||
inputWord.contains(patternWord) ||
patternWord.contains(inputWord)) {
matches++;
break;
}
}
}
return (float) matches / Math.max(inputWords.length, patternWords.length);
}
This calculates how many words from the input match words in the pattern, then divides by the maximum word count. So "turn on bluetooth" compared to "turn on bluetooth" gives 1.0 (perfect match), while "enable bluetooth" compared to "turn on bluetooth" gives 0.5 (2 out of 4 words match).
The algorithm also handles partial matches - if "bluetooth" contains "blue" or vice versa, it counts as a match. This helps with typos and variations.
Fuzzy Matching Fallback
If no pattern matches with confidence above 0.6, the system falls back to fuzzy matching. This extracts action words and target words from the input and tries to generate an intent:
private static CommandResult fuzzyMatch(String input) {
String[] words = input.split("\\s+");
// Extract action words
String action = "";
for (String word : words) {
if (word.matches("(turn|switch|enable|disable|open|close|start|stop)")) {
action = word;
break;
}
}
// Extract target words
String target = "";
for (String word : words) {
if (word.matches("(bluetooth|wifi|flashlight|torch|flash|camera|volume)")) {
target = word;
break;
}
}
// Generate intent from action + target
if (!action.isEmpty() && !target.isEmpty()) {
String intent = generateIntent(action, target, words);
return new CommandResult(intent, new HashMap<>(), 0.7f);
}
return null;
}
This handles cases where the input doesn't match any pattern exactly but still contains recognizable action and target words.
Parameter Extraction
Understanding the intent is only half the battle. Commands like "set alarm for 7 AM" need to extract the time (7) and period (AM). This is where parameter extraction comes in.
Time Extraction
For alarms and timers, I use regex patterns to extract time:
if (intent.equals("ALARM_SET") || intent.equals("TIMER_START")) {
Pattern timePattern = Pattern.compile("(\\d{1,2})\\s*(am|pm|AM|PM)");
Matcher matcher = timePattern.matcher(input);
if (matcher.find()) {
params.put("time", matcher.group(1));
params.put("period", matcher.group(2).toLowerCase());
}
// Also extract duration for timers
Pattern durationPattern = Pattern.compile("(\\d+)\\s*(minute|min|hour|hr|second|sec)");
matcher = durationPattern.matcher(input);
if (matcher.find()) {
params.put("duration", matcher.group(1));
params.put("unit", matcher.group(2).toLowerCase());
}
}
The regex (\\d{1,2})\\s*(am|pm|AM|PM) matches patterns like "7 AM", "9pm", "12 AM", etc. It captures the hour and the period separately, which makes conversion to 24-hour format easier.
App Name Extraction
For "open camera" or "launch gallery", the system extracts the app name:
if (intent.equals("OPEN_APP")) {
String[] words = input.split("\\s+");
for (int i = 0; i < words.length; i++) {
if (words[i].equals("open") || words[i].equals("launch")) {
if (i + 1 < words.length) {
params.put("app_name", words[i + 1]);
}
break;
}
}
}
This finds the word after "open" or "launch" and uses it as the app name. Then MainActivity tries to find and launch that app.
Location Extraction
For navigation commands like "navigate to work", the system extracts everything after "to":
if (intent.equals("NAVIGATE_TO")) {
String[] words = input.split("\\s+");
for (int i = 0; i < words.length; i++) {
if (words[i].equals("to") || words[i].equals("navigate")) {
if (i + 1 < words.length) {
StringBuilder location = new StringBuilder();
for (int j = i + 1; j < words.length; j++) {
location.append(words[j]).append(" ");
}
params.put("location", location.toString().trim());
}
break;
}
}
}
This captures multi-word locations like "New York" or "coffee shop near me".
Hardware Control Implementation
One of the most satisfying parts was implementing direct hardware control. Let me walk through a few examples.
Flashlight Control
The flashlight uses CameraManager API for direct control:
private String enableFlashlight() {
try {
CameraManager cameraManager =
(CameraManager) getSystemService(CAMERA_SERVICE);
String cameraId = cameraManager.getCameraIdList()[0];
cameraManager.setTorchMode(cameraId, true);
return "✅ Flashlight turned ON";
} catch (Exception e) {
return "❌ Could not enable flashlight: " + e.getMessage();
}
}
This is instant - no UI navigation, no settings screens. Just direct hardware control.
Bluetooth Control
Bluetooth is a bit trickier because enabling it requires user permission on newer Android versions:
private String enableBluetooth() {
try {
BluetoothAdapter bluetoothAdapter =
BluetoothAdapter.getDefaultAdapter();
if (bluetoothAdapter == null) {
return "❌ Bluetooth not available on this device";
}
if (!bluetoothAdapter.isEnabled()) {
Intent enableBtIntent = new Intent(
BluetoothAdapter.ACTION_REQUEST_ENABLE);
startActivityForResult(enableBtIntent, 1);
return "✅ Requesting Bluetooth permission...";
} else {
return "✅ Bluetooth is already ON";
}
} catch (Exception e) {
return "❌ Could not enable Bluetooth: " + e.getMessage();
}
}
For disabling, we can do it directly without permission:
private String disableBluetooth() {
try {
BluetoothAdapter bluetoothAdapter =
BluetoothAdapter.getDefaultAdapter();
if (bluetoothAdapter != null && bluetoothAdapter.isEnabled()) {
bluetoothAdapter.disable();
return "✅ Bluetooth turned OFF";
}
return "✅ Bluetooth is already OFF";
} catch (Exception e) {
return "❌ Could not disable Bluetooth: " + e.getMessage();
}
}
Volume Control
Volume control uses AudioManager with different stream types:
private String increaseVolume() {
try {
AudioManager audioManager =
(AudioManager) getSystemService(AUDIO_SERVICE);
int currentVolume = audioManager.getStreamVolume(
AudioManager.STREAM_MUSIC);
int maxVolume = audioManager.getStreamMaxVolume(
AudioManager.STREAM_MUSIC);
int newVolume = Math.min(currentVolume + 2, maxVolume);
audioManager.setStreamVolume(
AudioManager.STREAM_MUSIC, newVolume, 0);
return "✅ Volume increased";
} catch (Exception e) {
return "❌ Could not increase volume: " + e.getMessage();
}
}
I increment by 2 steps to make the change noticeable. The system supports separate control for ringtone volume, alarm volume, and media volume.
Alarm Setting with Multiple Fallbacks
Setting alarms was tricky because different devices have different alarm apps. I implemented multiple fallback mechanisms:
private String setAlarm(String time, String period) {
// Try 1: Standard AlarmClock intent
Intent alarmIntent = new Intent(
AlarmClock.ACTION_SET_ALARM);
alarmIntent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
if (time != null && period != null) {
int hour = Integer.parseInt(time);
if (period.equals("pm") && hour != 12) {
hour += 12;
} else if (period.equals("am") && hour == 12) {
hour = 0;
}
alarmIntent.putExtra(AlarmClock.EXTRA_HOUR, hour);
alarmIntent.putExtra(AlarmClock.EXTRA_MINUTES, 0);
alarmIntent.putExtra(AlarmClock.EXTRA_MESSAGE,
"Buddy Assistant Alarm");
}
if (alarmIntent.resolveActivity(getPackageManager()) != null) {
startActivity(alarmIntent);
return "✅ Setting alarm" +
(time != null ? " for " + time + " " + period : "");
}
// Try 2: Direct clock app launch
Intent clockIntent = new Intent(Intent.ACTION_MAIN);
clockIntent.addCategory(Intent.CATEGORY_LAUNCHER);
clockIntent.setPackage("com.android.deskclock");
clockIntent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
if (clockIntent.resolveActivity(getPackageManager()) != null) {
startActivity(clockIntent);
return "✅ Opening clock app for alarm setup";
}
// Try 3: Generic clock app
// ... more fallbacks
// Final fallback: Settings
Intent settingsIntent = new Intent(
Settings.ACTION_DATE_SETTINGS);
startActivity(settingsIntent);
return "❌ No alarm app found, opened settings instead";
}
This ensures alarms work on as many devices as possible, even if they don't have the standard AlarmClock app.
The Always-On Service Architecture
The AlwaysOnService is a foreground service that runs continuously, listening for the wake word "Hey Buddy". Here's how it works:
Service Initialization
@Override
public void onCreate() {
super.onCreate();
createNotificationChannel();
startForeground(1, createNotification());
initializeSpeechRecognizer();
startContinuousListening();
}
The service must start as a foreground service with a notification, otherwise Android will kill it. The notification shows "Always-on listening active - Say 'Hey Buddy'".
Continuous Listening Loop
private void startContinuousListening() {
if (speechRecognizer != null) {
Intent intent = new Intent(
RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
intent.putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, true);
intent.putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, 1);
speechRecognizer.startListening(intent);
isListening = true;
}
}
After each recognition result or error, the service automatically restarts listening:
@Override
public void onResults(Bundle results) {
ArrayList<String> matches = results.getStringArrayList(
SpeechRecognizer.RESULTS_RECOGNITION);
if (matches != null && !matches.isEmpty()) {
String command = matches.get(0).toLowerCase();
// Check for wake word
if (command.contains("hey buddy")) {
processCommand(command);
}
}
// Restart listening
startContinuousListening();
}
@Override
public void onError(int error) {
// Restart listening after error
startContinuousListening();
}
This creates a continuous loop that's always listening for the wake word.
Wake Word Detection
Currently, wake word detection is simple string matching - it checks if the recognized text contains "hey buddy". This works but isn't perfect - it can false-trigger on similar phrases or miss the wake word if speech recognition makes mistakes.
A better approach would be a dedicated wake word detection model (like what Google Assistant uses), but that's a future enhancement.
Broadcasting Commands
When the wake word is detected, the service extracts the actual command and broadcasts it:
private void processCommand(String command) {
// Remove "hey buddy" and process the actual command
String actualCommand = command.replace("hey buddy", "").trim();
// Send command to MainActivity
Intent intent = new Intent("com.buddy.assistant.VOICE_COMMAND");
intent.putExtra("command", actualCommand);
sendBroadcast(intent);
}
MainActivity has a BroadcastReceiver that listens for this intent and processes the command.
Command Execution Flow
When a command arrives at MainActivity, here's the execution flow:
private BroadcastReceiver mCommandReceiver = new BroadcastReceiver() {
@Override
public void onReceive(Context context, Intent intent) {
if (intent.getAction().equals("com.buddy.assistant.VOICE_COMMAND")) {
String command = intent.getStringExtra("command");
// Process with AI
AICommandProcessor.CommandResult result =
AICommandProcessor.processCommand(command);
// Execute command
String response = executeAICommand(result);
// Show feedback
showToast(response);
mResponseText.setText(response);
}
}
};
The executeAICommand() method is a large switch statement that handles all 50+ intents:
private String executeAICommand(CommandResult result) {
String intent = result.intent;
Map<String, String> params = result.parameters;
switch (intent) {
case "BLUETOOTH_ON":
return enableBluetooth();
case "BLUETOOTH_OFF":
return disableBluetooth();
case "WIFI_ON":
return enableWifi();
// ... 50+ more cases
default:
return "❌ AI Command not implemented: " + intent;
}
}
Each case calls a dedicated method that implements the specific feature.
Error Handling and Fallbacks
I've implemented extensive error handling throughout. Every hardware control method has try-catch blocks and returns user-friendly error messages. If direct control fails, many methods fall back to opening relevant settings screens.
For example, if Bluetooth enable fails, it might open Bluetooth settings. If alarm setting fails, it opens the clock app. This ensures the assistant is always helpful, even when direct control isn't possible.
Performance Considerations
The pattern matching approach is fast - it's just string comparisons and simple math. Even with 50+ intents and hundreds of patterns, processing takes milliseconds. This makes it suitable for real-time voice control.
The always-on service does consume battery, but it's manageable. The speech recognition API handles most of the optimization - it only processes audio when it detects speech, not continuously.
What's Working Well
After extensive testing, here's what works reliably:
- Intent Classification: The pattern matching handles most common phrasings well
- Hardware Control: Direct control of Bluetooth, WiFi, flashlight, volume works instantly
- Parameter Extraction: Time extraction for alarms works reliably
- Fallback Mechanisms: Multiple fallbacks ensure features work on different devices
- Text Input: The text input fallback works perfectly when speech recognition isn't available
Areas for Improvement
There are definitely areas that could be better:
- Wake Word Detection: Simple string matching isn't ideal - a dedicated model would be better
- Intent Classification: Pattern matching works but ML-based classification would be more accurate
- Parameter Extraction: Regex-based extraction is limited - NER would handle more cases
- Battery Optimization: The always-on service could be optimized further
- Offline Speech Recognition: Currently requires system STT - offline models would be better
These are all potential future enhancements, which we'll cover in Part 3.
Next Steps
In Part 3, we'll cover testing results, performance metrics, lessons learned, and future enhancements. I'll also share the complete source code and build instructions so you can try it yourself.
The implementation is solid and functional. It's not perfect - there's always room for improvement - but it demonstrates that you can build a privacy-focused voice assistant entirely on-device using pattern-based natural language understanding.
Stay tuned for the final part!
Questions about the implementation? Found a bug or have a suggestion? Let's discuss in the comments below.