In the fast-paced world of automotive technology, every second counts, especially when it comes to ensuring the safety of both drivers and pedestrians. One crucial component in modern vehicles is the rearview camera, which provides drivers with a clear view of what’s behind them. However, the challenge arises when the camera system needs to be up and running within mere seconds of ignition, while the Android operating system, which controls many of the vehicle’s functions, takes significantly longer to boot. In this blog, we will explore a groundbreaking solution to this problem — the Exterior View System (EVS), a self-contained application designed to minimize the delay between ignition and camera activation.

Problem

In vehicles, there is a camera located at the rear (back) of the vehicle to provide the driver with a view of what’s behind them. This camera is useful for parking, reversing, and overall safety. However, there is a requirement that this rearview camera should be able to show images on the display screen within 2 seconds of the vehicle’s ignition (engine start) being turned on.

Challenge

The challenge is that many vehicles use the Android operating system to power their infotainment systems, including the display screen where the rearview camera’s images are shown. Android, like any computer system, takes some time to start up. In this case, it takes tens of seconds (meaning around 10 or more seconds) for Android to fully boot up and become operational after the ignition is turned on.

Solution is Exterior View System (EVS)

Exterior View System (EVS): To address the problem of the slow boot time of Android and ensure that the rearview camera can show images within the required 2 seconds, a solution called the Exterior View System (EVS) is proposed.

So, What is Exterior View System (EVS)

The Exterior View System (EVS) emerges as a pioneering solution to the problem of delayed camera activation. Unlike traditional camera systems that rely heavily on the Android OS, EVS is an independent application developed in C++. This approach drastically reduces the system’s dependency on Android, allowing EVS to become operational within a mere two seconds of ignition.

The Exterior View System (EVS) in Android Automotive is a hardware abstraction layer (HAL) that provides support for rearview and surround view cameras in vehicles. EVS enables OEMs to develop and deploy advanced driver assistance systems (ADAS) and other safety features that rely on multiple camera views.

The EVS HAL consists of a number of components, including:

• A camera manager that provides access to the vehicle’s cameras
• A display manager that controls the output of the camera streams
• A frame buffer manager that manages the memory used to store camera frames
• A sensor fusion module that combines data from multiple cameras to create a single, unified view of the vehicle’s surroundings.

EVS is a key component of Android Automotive’s ADAS and safety features. It enables vehicles to provide drivers with a comprehensive view of their surroundings, which can help to prevent accidents.

Here are some of the benefits of using EVS in Android Automotive:

• Improved safety: EVS can help to prevent accidents by providing drivers with a comprehensive view of their surroundings. This is especially helpful in low-visibility conditions, such as at night or in bad weather.
• Advanced driver assistance features: EVS can be used to power advanced driver assistance features, such as lane departure warning, blind spot monitoring, and parking assist. These features can help to make driving safer and more convenient.
• Enhanced user experience: EVS can be used to enhance the user experience of Android Automotive by providing drivers with a more immersive view of their surroundings. This can be helpful for navigation, entertainment, and other tasks.

If you are looking for a safe and advanced driving experience, then Android Automotive with EVS is a great option.

Here are some examples of how EVS can be used in Android Automotive:

• Rearview camera: A rearview camera can be used to provide drivers with a view of the area behind their vehicle. This can be helpful for backing up and parking.
• Sideview cameras: Sideview cameras can be used to provide drivers with a view of the area to the sides of their vehicle. This can be helpful for changing lanes and avoiding obstacles.
• Surround-view cameras: Surround-view cameras can be used to provide drivers with a 360-degree view of the area around their vehicle. This can be helpful for parking in tight spaces and maneuvering in difficult conditions.
• Lane departure warning: Lane departure warning uses EVS to detect when the vehicle is drifting out of its lane. If the vehicle starts to drift, the system will alert the driver and may even apply the brakes to help keep the vehicle in its lane.
• Blind spot monitoring: Blind spot monitoring uses EVS to detect vehicles in the driver’s blind spots. If a vehicle is detected in the blind spot, the system will alert the driver with a visual or audible warning.
• Parking assist: Parking assist uses EVS to help drivers park their vehicles. The system will provide guidance on how to steer and brake, and it may even automatically control the steering wheel and brakes.

These are just a few examples of how EVS can be used in Android Automotive. As the technology continues to develop, we can expect to see even more innovative and advanced uses for EVS in the future.

BTW, Why EVS is introduced?

The introduction of the Exterior View System (EVS) serves several key purposes, each of which contributes to its significance:

SIMPLE: Support camera and view display with a simplified design

EVS is designed to provide a straightforward and uncomplicated way to manage camera input and display views. Its primary goal is to make it easy for developers to work with cameras and show what they capture on the screen. By offering a simplified design, EVS reduces complexity, making it more efficient to integrate camera functionality into applications.

EARLY: Intend to show display very early in the Android boot process

One of the primary motivations behind EVS is to ensure that camera views can be displayed as quickly as possible after the ignition of the vehicle. Traditional Android boot times can be relatively long, potentially delaying the display of camera feeds. EVS addresses this issue by functioning independently of the Android operating system and initiating the camera display within just a few seconds of starting the vehicle. This capability enhances user experience by providing prompt access to crucial camera information.

EXTENSIBLE: Enables advanced features to be implemented in user apps

EVS is designed with extensibility in mind, allowing developers to implement advanced features and functionalities within their applications. By providing a framework that can be built upon, EVS empowers app creators to integrate innovative and sophisticated camera-related features, enhancing the overall capabilities of their applications. This extensibility promotes creativity and enables the development of unique and tailored user experiences.

Overall, the introduction of EVS is driven by the desire to simplify camera and view display functionality, ensure early access to camera views during the Android boot process, and provide a platform for the implementation of advanced and customizable features within user applications. This approach aims to enhance the efficiency, responsiveness, and versatility of camera-related functionalities in the context of Android-based systems.

EVS stack in Android

The EVS stack in Android consists of three main components that work together to facilitate the functioning of the Exterior View System:

EVS Application

The EVS application is composed of native code and is initiated by the init.rc (initialization script) during the system startup process. This application runs in the background even when it’s not actively being used by a user. Its primary purpose is to manage the processing and display of exterior camera views.

EVS Manager

The EVS Manager acts as an intermediary layer that connects the EVS application with the Hardware Abstraction Layer (HAL) and the user-facing applications. It essentially serves as a wrapper, facilitating communication and data exchange between these different components. Importantly, the EVS Manager can handle multiple concurrent clients, meaning that it can manage requests from multiple user applications that want to access and display camera views simultaneously.

EVS HAL (Hardware Abstraction Layer)

The EVS HAL is a crucial component that interacts with the underlying hardware and interfaces with the SurfaceFlinger module, which is responsible for rendering graphics on the screen. The EVS HAL is designed to be hardware-independent, meaning it can work with various types of hardware configurations. It plays a vital role in capturing camera data, processing it, and delivering it to the EVS Manager for further distribution to user applications.

Overall, the EVS stack in Android is structured to ensure efficient communication between the EVS application, the EVS Manager, and the EVS HAL. This stack enables the seamless management of exterior camera views, from capturing the data to processing it and finally displaying it to users through various applications.

Architecture

The Exterior View System’s architecture is designed to maximize efficiency and speed while maintaining a seamless user experience. The following system components are present in the EVS architecture:

EVS Application

There’s an EVS application example written in C++ that you can find at /packages/services/Car/evs/app. This example shows you how to use EVS. The job of this application is to ask the EVS Manager for video frames and then send these frames to the EVS Manager so they can be shown on the screen. It’s designed to start up as soon as the EVS and Car Service are ready, usually within two seconds after the car turns on. Car makers can change or use a different EVS application if they want to.

EVS Manager

The EVS Manager, located at /packages/services/Car/evs/manager, is like a toolbox for EVS applications. It helps these applications create different things, like showing a basic rearview camera view or even a complex 6DOF(Six degrees of freedom (6DOF) refers to the specific number of axes that a rigid body is able to freely move in three-dimensional space.) multi-camera 3D view. It talks to the applications through HIDL, a special communication way in Android. It can work with many applications at the same time.

Other programs, like the Car Service, can also talk to the EVS Manager. They can ask the EVS Manager if the EVS system is up and running or not. This helps them know when the EVS system is working.

EVS HIDL interface

The EVS HIDL interface is how the EVS system’s camera and display parts talk to each other. You can find this interface in the android.hardware.automotive.evs package. There’s an example version of it in /hardware/interfaces/automotive/evs/1.0/default that you can use to test things out. This example makes fake images and checks if they work properly.

The car maker (OEM) needs to make the actual code for this interface. The code is based on the .hal files in /hardware/interfaces/automotive/evs. This code sets up the real cameras, gets their data, and puts it in special memory areas that Gralloc (Gralloc is a type of shared memory that is also shared with the GPU) understands. The display part of the code has to make a memory area where the app can put its images (usually using something called EGL), and then it shows these images on the car screen. This display part is important because it makes sure the app’s images are shown instead of anything else on the screen. Car makers can put their own version of the EVS code in different places, like /vendor/… /device/… or hardware/… (for example, /hardware/[vendor]/[platform]/evs).

Kernel drivers

For a device to work with the EVS system, it needs special software called kernel drivers. If a device already has drivers for its camera and display, those drivers can often be used for EVS too. This can be helpful, especially for display drivers, because showing images might need to work together with other things happening in the device.

In Android 8.0, there’s an example driver based on something called v4l2 (you can find it in packages/services/Car/evs/sampleDriver). This driver uses the kernel for v4l2 support (a way to handle video) and uses something called SurfaceFlinger to show images.

It’s important to note that the sample driver uses SurfaceFlinger, which isn’t suitable for a real device because EVS needs to start quickly, even before SurfaceFlinger is fully ready. However, the sample driver is designed to work with different hardware and lets developers test and work on EVS applications at the same time as they develop EVS drivers.

EVS hardware interface description

In this section, we explain the Hardware Abstraction Layer (HAL) for the EVS (Exterior View System) in Android. Manufacturers need to create implementations of this HAL to match their hardware.

IEvsEnumerator

This object helps find available EVS hardware (cameras and the display) in the system.

• getCameraList(): Gets a list of all available cameras.
• openCamera(string camera_id): Opens a specific camera for interaction.
• closeCamera(IEvsCamera camera): Closes a camera.
• openDisplay(): Opens the EVS display.
• closeDisplay(IEvsDisplay display): Closes the display.
• getDisplayState(): Gets the current display state.

IEvsCamera

This object represents a single camera and is the main interface for capturing images.

• getCameraInfo(): Gets information about the camera.
• setMaxFramesInFlight(int32 bufferCount): Sets the maximum number of frames the camera can hold.
• startVideoStream(IEvsCameraStream receiver): Starts receiving camera frames.
• doneWithFrame(BufferDesc buffer): Signals that a frame is done being used.
• stopVideoStream(): Stops receiving camera frames.
• getExtendedInfo(int32 opaqueIdentifier): Requests driver-specific information.
• setExtendedInfo(int32 opaqueIdentifier, int32 opaqueValue): Sends driver-specific values.

BufferDesc

Describes an image passed through the API.

• width: Width of the image in pixels.
• height: Height of the image in pixels.
• stride: Number of pixels per row in memory.
• pixelSize: Size of a single pixel in bytes.
• format: Pixel format (compatible with OpenGL).
• usage: Usage flags for the image.
• bufferId: A unique identifier for the buffer.
• memHandle: Handle for the image data.

It’s important to note that these interfaces help EVS applications communicate with the hardware and manage camera and display functionality. Manufacturers can customize these implementations to match their specific hardware features and capabilities.

IEvsCameraStream

The client uses this interface to receive video frames asynchronously.

• deliverFrame(BufferDesc buffer): Called by the HAL whenever a video frame is ready. The client must return buffer handles using IEvsCamera::doneWithFrame(). When the video stream stops, this callback might continue as the pipeline drains. When the last frame is delivered, a NULL bufferHandle is sent, indicating the end of the stream. The NULL bufferHandle doesn’t need to be sent back using doneWithFrame(), but all other handles must be returned.

IEvsDisplay

This object represents the EVS display, controls its state, and handles image presentation.

• getDisplayInfo(): Gets basic information about the EVS display.
• setDisplayState(DisplayState state): Sets the display state.
• getDisplayState(): Gets the current display state.
• getTargetBuffer(): Gets a buffer handle associated with the display.
• returnTargetBufferForDisplay(handle bufferHandle): Informs the display that a buffer is ready for display.

DisplayDesc

Describes the basic properties of an EVS display.

• display_id: Unique identifier for the display.
• vendor_flags: Additional information for a custom EVS Application.

DisplayState

Describes the state of the EVS display.

• NOT_OPEN: Display has not been opened.
• NOT_VISIBLE: Display is inhibited.
• VISIBLE_ON_NEXT_FRAME: Will become visible with the next frame.
• VISIBLE: Display is currently active.
• DEAD: Display is not available, and the interface should be closed.

The IEvsCameraStream interface allows the client to receive video frames from the camera, while the IEvsDisplay interface manages the state and presentation of images on the EVS display. These interfaces help coordinate the communication between the EVS hardware and the application, ensuring smooth and synchronized operation.

EVS Manager

The EVS Manager is a component that acts as an intermediary between applications and the EVS Hardware API, which handles external camera views. The Manager provides shared access to cameras, allowing multiple applications to use camera streams concurrently. A primary EVS application is the main client of the Manager, with exclusive display access. Other clients can have read-only access to camera images.

The EVS Manager offers the same API as the EVS Hardware drivers, except that the EVS Manager API allows concurrent camera stream access. The EVS Manager is, itself, the one allowed client of the EVS Hardware HAL layer, and acts as a proxy for the EVS Hardware HAL.

IEvsEnumerator

• openCamera(string camera_id): Obtains an interface to interact with a specific camera. Multiple processes can open the same camera for video streaming.

IEvsCamera

• startVideoStream(IEvsCameraStream receiver): Starts video streams independently for different clients. The camera starts when the first client begins.
• doneWithFrame(uint32 frameId, handle bufferHandle): Returns a frame when a client is done with it. Other clients continue to receive all frames.
• stopVideoStream(): Stops a video stream for a client, without affecting other clients.
• setExtendedInfo(int32 opaqueIdentifier, int32 opaqueValue): Allows one client to affect another by sending driver-specific values.

IEvsDisplay

• The EVS Manager passes the IEvsDisplay interface directly to the underlying HAL implementation.

In essence, the EVS Manager acts as a bridge, enabling multiple clients to utilize the EVS system simultaneously, while maintaining independent access to cameras. It provides flexibility and concurrent access to camera streams, enhancing the overall functionality of the EVS system.

Typical control flow

The EVS application in Android is a C++ program that interacts with the EVS Manager and Vehicle HAL to offer basic rearview camera functionality. It’s meant to start early in the system boot process and can show appropriate video based on available cameras and the car’s state (gear, turn signal). Manufacturers can customize or replace this application with their own logic and visuals.

Since image data is provided in a standard graphics buffer, the application needs to move the image from the source buffer to the output buffer. This involves a data copy, but it also gives the app the flexibility to manipulate the image before displaying it.

For instance, the app could move pixel data while adding scaling or rotation. Alternatively, it could use the source image as an OpenGL texture and render a complex scene onto the output buffer, including virtual elements like icons, guidelines, and animations. More advanced applications might even combine multiple camera inputs into a single output frame for a top-down view of the vehicle surroundings.

Overall, the EVS application provides the essential connection between hardware and user presentation, allowing manufacturers to create custom and sophisticated visual experiences based on their specific vehicle designs and features.

Boot Sequence Diagram

The boot sequence diagram outlines the steps involved in the initialization and operation of the Exterior View System (EVS) within the context of an Android-based system:

Communication with EVS Manager and Vehicle HAL

The process begins by establishing communication between the EVS Application and both the EVS Manager and the Vehicle HAL (Hardware Abstraction Layer). This communication enables the EVS Application to exchange information and commands with these two key components.

Infinite Loop for Monitoring Camera and Gear/Turn Signal State

Once communication is established, the EVS Application enters an infinite loop. This loop serves as the core operational mechanism of the system. Within this loop, the EVS Application constantly monitors two critical inputs: the camera state and the state of the vehicle’s gear or turn signals. These inputs help determine what needs to be displayed to the user.

Reaction to Camera and Vehicle State

Based on the monitored inputs, the EVS Application reacts accordingly. If the camera state changes (e.g., a new camera feed is available), the EVS Application processes the camera data. Similarly, if there’s a change in the gear or turn signal state, the system responds by updating the displayed content to provide relevant information to the driver.

Use of Source Image as OpenGL Texture and Rendering a Complex Scene

The EVS Application utilizes the source image from the camera feed as an OpenGL texture. OpenGL is a graphics rendering technology that enables the creation of complex visual scenes. The EVS Application takes advantage of this capability to render a sophisticated and informative scene. This scene, which includes data from the camera feed and potentially other elements, is then composed and prepared for display.

Rendering to the Output Buffer

The rendered scene is finally placed into the output buffer, which is essentially a designated area of memory used for displaying content on the screen. This process ensures that the composed scene, which combines the camera feed and other relevant information, is ready for presentation to the user.

In essence, the boot sequence diagram illustrates how the EVS Application interacts with the EVS Manager, the Vehicle HAL, and the hardware to continuously monitor camera and vehicle states, react to changes, create a visually informative scene, and render that scene for display on the screen. This orchestration ensures that the driver receives real-time and relevant exterior view information during the operation of the vehicle.

Boot Time Evaluation

The evaluation of boot time for the application involves ensuring that it is initiated promptly by the system’s initialization process. Specifically, the goal is for the application to start running as soon as the EVS Manager and the Vehicle HAL become available. This initiation is targeted to occur within a time frame of 2.0 seconds from the moment the power is turned on.

In simpler terms, the aim is to have the application up and running very quickly after the vehicle is powered on. This swift start time helps ensure that the Exterior View System (EVS) becomes operational without unnecessary delays, allowing the system to provide timely and accurate information to the user based on the exterior camera feeds and other relevant data.

The evaluation of boot time is measured from the initial stage of the Android system’s initialization process. This means that the time it takes for the system to fully start up and become operational is calculated starting from the very beginning of the boot process. This measurement includes all the necessary tasks and processes that occur during the system’s startup sequence, such as loading essential components, initializing hardware, and launching applications.

In essence, boot time evaluation from Android first stage init provides a comprehensive view of the time it takes for the entire system to transition from a powered-off state to a fully functional state, including the initiation and execution of various components like the Exterior View System (EVS) application, EVS Manager, Vehicle HAL, and other crucial elements. The goal is to optimize and minimize this boot time to ensure efficient and timely access to the system’s functionalities and services.

Quick Boot Optimization

Here’s an improved version with three steps to enhance the boot time of the system and the operation of the Exterior View System (EVS):

EVS App: Concurrent Camera Stream and GL Preparation

The EVS Application can be optimized to start the camera stream and simultaneously prepare the OpenGL (GL) rendering. By executing these tasks concurrently, the system can make more efficient use of available resources, reducing overall initialization time. This means that as the camera stream begins, the OpenGL components responsible for rendering the visuals are already being prepared, allowing for a smoother and quicker transition to displaying the camera views.

EVS HAL: Display Frames via Composer Service Early

The EVS Hardware Abstraction Layer (HAL) can be enhanced to leverage the composer service to display frames even before SurfaceFlinger is fully ready. By doing so, the system can start showing visual content sooner, improving the responsiveness of the user interface. This approach allows for an early display of camera frames and other graphics, enhancing the user experience by reducing any perceptible delay in visual feedback.

Android Init: Start EVS Services/HALs on Early-Init

To further expedite the boot process and enable faster access to EVS-related functionalities, consider moving the initialization of EVS services and HALs to the “early-init” phase of the Android boot sequence. This adjustment ensures that essential EVS components are initiated earlier in the startup process, reducing the overall time it takes for the system to become fully operational. Starting EVS-related services and HALs at an earlier stage streamlines the boot process, making the EVS capabilities available to users more quickly after powering on the device.

By implementing these three steps, the boot time of the system can be significantly improved, and the operation of the Exterior View System can become more seamless and responsive, enhancing the overall user experience.

Optimization Breakdown

The optimization efforts yield significant improvements in the launch time of the Exterior View System (EVS) and the overall system boot time. Here’s a breakdown of the results:

Optimized EVS Launch Time

By implementing the proposed optimizations, the EVS launch time has been reduced to 1.1 seconds from the Android first stage initialization. This signifies a substantial improvement in getting the EVS system up and running promptly after the Android boot process starts.

Total System Boot Time

The total time required for the entire system to boot up, including bootloader and kernel time, has been reduced to approximately 3.0 seconds. This represents an impressive reduction in the time it takes for the system to become fully operational and ready for use.

Additionally, there’s an alternative scenario to consider:

Reduced Texture Operations for Faster EVS Launch

If the GL (OpenGL) preparation and texture operations are removed from the EVS App, the launch time of EVS can be further decreased to 0.7 seconds. This change has a notable impact on getting the EVS system up and running even more swiftly after the Android boot process initiates.

Total Boot Time with Reduced Texture Operations

With the removal of texture operations, the total system boot time is reduced to approximately 2.6 seconds. This achievement demonstrates an even more streamlined boot process for the entire system.

Overall, the optimization efforts, along with the option to remove texture operations from the EVS App, have led to significant improvements in both the launch time of the EVS system and the overall boot time of the entire system on your hardware development board. These enhancements contribute to a more responsive and efficient user experience, allowing users to access and utilize the EVS functionality more quickly and effectively.

Display Sharing — EVS Priority and Mechanism

The integration of exterior cameras in vehicles has transformed the way drivers navigate their surroundings. From parallel parking to navigating tight spaces, these cameras offer valuable assistance. However, the challenge arises when determining how to seamlessly switch between the main display, which often serves multiple functions, and the exterior view provided by EVS. The solution lies in prioritizing EVS for display sharing.

EVS Priority over Main Display

The EVS application is designed to have priority over the main display. This means that when certain conditions are met, EVS can take control of the main display to show its content. The main display is the screen usually used for various functions, like entertainment, navigation, and other infotainment features.

Grabbing the Display

Whenever there’s a need to display images from an exterior camera (such as the rearview camera), the EVS application can “grab” or take control of the main display. This allows the camera images to be shown prominently to the driver, providing important visual information about the vehicle’s surroundings.

Example Scenario — Reverse Gear

One specific scenario where this display-sharing mechanism is used is when the vehicle’s reverse gear is selected. When the driver shifts the transmission into reverse, the EVS application can immediately take control of the main display to show the live feed from the rearview camera. This is crucial for assisting the driver in safely maneuvering the vehicle while reversing.

No Simultaneous Content Display

Importantly, there is no mechanism in place to allow both the EVS application and the Android operating system to display content simultaneously on the main display. In other words, only one of them can be active and show content at any given time.

In short, the concept of display sharing in this context involves the Exterior View System (EVS) having priority over the main display in the vehicle. EVS can take control of the main display whenever there’s a need to show images from an exterior camera, such as the rearview camera. This mechanism ensures that the driver receives timely and relevant visual information for safe driving. Additionally, it’s important to note that only one of the applications (EVS or Android) can display content on the main screen at a time; they do not operate simultaneously.

Conclusion

The Exterior View System (EVS) stands as a remarkable advancement in automotive technology, addressing the critical issue of swift camera activation during vehicle ignition. By employing a self-contained application with minimal dependencies on the Android operating system, EVS ensures that drivers have access to real-time camera images within a mere two seconds of starting the ignition. This breakthrough architecture, prioritized display sharing, and synchronized activation make EVS a game-changer in enhancing road safety and driver convenience.

As the automotive industry continues to evolve, innovations like the Exterior View System pave the way for a safer and more efficient driving experience. With EVS leading the charge, we can look forward to a future where technology seamlessly integrates with our everyday journeys, ensuring a smoother and more secure ride for all.