Multi-Camera and Reverse Camera

Overview

Modern Android devices frequently feature multiple camera sensors (e.g., wide, ultra-wide, telephoto) to offer versatile photography capabilities. The Camera2 API introduced the logical multi-camera framework to manage these arrays seamlessly. Additionally, specialized environments like Android Automotive OS (AAOS) introduce unique requirements, such as the ultra-low latency necessary for rearview (backup) cameras, addressed by the External View System (EVS).

Logical Multi-Camera API

A "Logical Camera" is a software abstraction that groups two or more physical camera sensors facing the same direction into a single entity. To an application using standard Camera2 API, a logical camera appears as one device capable of extraordinary optical zoom ranges.

The Problem it Solves

Before logical cameras, applications had to manually open and switch between separate camera IDs when zooming, resulting in a noticeable glitch or freeze as the old stream was torn down and the new one configured.

How it Works

1. Abstraction: The HAL exposes a single logical camera ID.
2. Seamless Switching: As the application requests different zoom levels (via CaptureRequest.CONTROL_ZOOM_RATIO), the Camera HAL handles the complex logic of powering up the appropriate physical sensor and seamlessly routing its frames into the output stream.
3. Field of View (FoV) Matching: The HAL crops and scales the output from the different physical sensors to ensure a smooth transition without sudden jumps in perspective.

// Discovering a Logical Multi-Camera
CameraCharacteristics chars = manager.getCameraCharacteristics(cameraId);
int[] capabilities = chars.get(CameraCharacteristics.REQUEST_AVAILABLE_CAPABILITIES);
boolean isLogical = false;
for (int cap : capabilities) {
    if (cap == CameraCharacteristics.REQUEST_AVAILABLE_CAPABILITIES_LOGICAL_MULTI_CAMERA) {
        isLogical = true;
        break;
    }
}

if (isLogical) {
    Set<String> physicalIds = chars.getPhysicalCameraIds();
    Log.d("Camera", "Logical camera " + cameraId + " backs physical sensors: " + physicalIds);
}

Physical Sub-Cameras

While the logical camera abstraction simplifies zoom, advanced applications (like computational photography apps capturing a wide and telephoto shot simultaneously for depth estimation) need explicit control over the underlying hardware.

The API allows applications to request streams from specific physical sub-cameras.

Dual Streaming

An app can attach target Surfaces explicitly associated with a physical camera ID to a logical camera request.

// Creating an OutputConfiguration tied to a specific physical sensor
OutputConfiguration physicalConfig = new OutputConfiguration(surface);
physicalConfig.setPhysicalCameraId(specificPhysicalId);

// Add to session configuration
SessionConfiguration sessionConfig = new SessionConfiguration(
    SessionConfiguration.SESSION_REGULAR,
    Arrays.asList(mainConfig, physicalConfig),
    executor, callback);

The HAL must ensure synchronized capture across the active physical sensors, providing timestamps that align precisely to facilitate computational fusion algorithms.

Automotive: External View System (EVS)

In automotive contexts, camera systems serve safety-critical functions, most notably the rearview (backup) camera and surround-view systems.

The standard Camera Service is heavily tied to the Android framework, meaning it requires SurfaceFlinger, Binder, and the Java runtime to be fully initialized before displaying a frame. This boot process takes seconds, which violates automotive safety regulations requiring the rearview camera to display video within a fraction of a second (typically < 2 seconds) of the vehicle being turned on and shifted into reverse.

The EVS Solution

Android Automotive introduces the External View System (EVS) to bypass the standard heavy framework. EVS is a lightweight, specialized camera stack designed for rapid initialization and minimal overhead.

ExternalViewSystem (EVS) HAL

The EVS subsystem consists of its own HAL and native services, operating independently of the standard CameraService.

Architecture

1. Early Boot Execution: The EVS service (evsmanager) starts extremely early in the init sequence, long before the Android UI framework is ready.
2. EVS HAL (IEvsCamera): This HAL is specifically designed for simple video streaming. It lacks the complex metadata, 3A controls, and deep stream configuration negotiation of Camera HAL3. Its primary job is to get raw frames from a V4L2 device and pass them along as quickly as possible.
3. EVS Display (IEvsDisplay): EVS interacts directly with the hardware composer (HWC) or kernel DRM/KMS drivers to display frames, completely bypassing SurfaceFlinger. This ensures frames hit the screen immediately.

Workflow Example

1. Vehicle shifted to reverse.
2. The Vehicle HAL signals the EVS manager.
3. EVS manager requests IEvsCamera->startVideoStream().
4. Frames stream directly from the camera driver, through the EVS HAL, and are composited directly to the screen via IEvsDisplay.
5. Once the standard Android framework finishes booting and the user shifts out of reverse, the EVS system can yield control of the display and camera resources back to the standard Android subsystem.

// Conceptual flow in EVS Client
sp<IEvsEnumerator> evs = IEvsEnumerator::getService();
sp<IEvsCamera> camera = evs->openCamera(cameraId);
sp<IEvsDisplay> display = evs->openDisplay();

// Set up callback to receive frames
camera->setMaxFramesInFlight(2);
camera->startVideoStream(myFrameReceiver);

// In the receiver callback, frames are passed directly to the display
Return<void> receiveFrames(const hidl_vec<BufferDesc>& buffers) {
    display->returnDisplayBuffer(buffers[0]);
    return Void();
}

The EVS stack highlights Android's flexibility in providing specialized subsystems tailored for the strict latency and safety requirements of the automotive industry.