The Sandbox: the AVSandbox knowledge hub

The validation and verification of autonomous systems poses a range of complex challenges. Whilst physical testing continues to be useful, the proliferation of conditions and variables means that new virtual methods are needed. There is an urgent demand for a comprehensive understanding of ADAS, connected & autonomous vehicles behaviours and fault detection algorithms in which we should be pulling together various existing validation & verification methods to provide a complete, fail-safe solution.

To date, two specific validation & verification techniques have been developed:

• Automatic worst case search
• Reachability analysis

Given the high risks involved, for safety critical systems, validation & verification must take place at component, function (sub-system) and system level with various iterations at each.

Here at Claytex, we have been working on solving these challenges for the past few years by combining the right simulation technology with tools we created to build a comprehensive and scalable solution which focuses on key areas that can be defined as modules as highlighted in the figure below.

Our Simulation Manager for rFpro is the pivotal part of our solution as it encapsulates all the different functions that are needed to configure, test, run and capture necessary results to validate ADAS features and autonomous vehicles behaviours. Our solutions are built around rFpro which is the virtual environment that provides high fidelity digital twins for the development and testing of autonomous vehicles, ADAS and vehicle dynamics.

The Simulation Manager handles all these tasks through an easy to use graphical interface but can also be fully automated to run multiple tests.  In addition to providing support for open standards such as OpenSCENARIO we make it easy to pull in pre-defined scenarios and then apply them to different locations with different weather conditions and at different times of day.  This enables the ADAS or AV system to be tested across a wide range of conditions which can’t be achieved through physical testing.

Aligning with the fundamental procedures of ADAS and AVs we will explore the different modules within our solution.

Control Systems

Advancing to higher levels of automation brings unprecedented challenges. To overcome these problems, the set of functionalities are growing in terms of algorithms complexity and the required hardware. The risk associated with testing the implemented solutions in the real world is high, expensive and time consuming. That is why virtual simulation tools for automotive testing are becoming more and more popular.

Original Equipment Manufacturers (OEMs) use these tools to create simulations that support the complete sense, compute, act loop. Production software is tested against simulated sensor data and the consequences of the decisions and actions are generated in response to the given software commands. This gives OEMs the ability to optimize the design of vehicles before any physical prototypes are produced. Early optimization brings reduced costs and less time delays as many more tests can be run in the virtual environment. Early in the design process this is realised using a Software-in-the-Loop approach with the control software running within the simulation environment. During the validation phase a Hardware-in-the-Loop approach is usually used which requires the simulation environment to run in real-time. Using rFpro as the simulation environment within our solutions enables both of these approaches.

Vehicle Dynamics/Physics Model

The requirements for the vehicle physics model vary significantly depending on the type of system that is being tested, the scenario being simulated and stage of the project.  For example, when simulating a routine driving scenario along a highway then the detailed dynamics are not usually that important.  However, when simulating an emergency braking scenario, the vehicle dynamics become much more important. In addition we shouldn’t only be thinking about the vehicle dynamics but with today’s complex vehicles we should really be using vehicle physics models that can include all the vehicle systems: mechanical, electrical, control, etc. Our solutions include a built-in vehicle model and we also provide integrations with a wide range of vehicle dynamics tools including Dymola, Simulink, CarMaker and many more.

Sensor Suite

Our sensor models include generic models and detailed, device specific, models of real sensors. Through our involvement with the Streetwise project (a CCAV and Innovate UK funded project led by FIVE AI) we have carried out an extensive testing and validation process to develop accurate, physics based, models of real LIDAR and RADAR sensors. Our latest models of Velodyne and Ouster sensors include weather effects and our multi-path radar models will also incorporate similar effects. We do offer a full range of sensor models including cameras, ultrasound, and GPS sensors in which we will focus on in later blogs and you can see some examples of these models here.

Scenario Managenent

The development process proposed in the ISO 26262 standard is based upon multiple V-models and defines activities and work products for each process step. In many of these process steps, scenario-based approaches can be applied to achieve the defined work products for the development of automated driving functions. To accomplish the work products of different process steps, scenarios must focus on various aspects like a human understandable notation or a description via state variables. This leads to contradictory requirements regarding the level of detail and way of notation for the representation of scenarios.

Scenario management is pivotal to testing any autonomous system. We are involved in the D-Risk project, which is a collaborative R&D project, funded by CCAV and Innovate UK, aimed at developing a comprehensive set of scenarios with innovative approaches to the management, navigation, analysis and testing of the scenarios. To fully understand the full structure of scenarios, we divided this module into 3 Modules:

1. Traffic

To bring scenarios to life, we need traffic and pedestrians to move around our virtual world.  There are many ways we can do this, ranging from time-based control through ego-synchronised actions and by coupling to specialist traffic simulation tools such as SUMO.

To enable the generation of new traffic scenarios, we have created a scenario definition tool that supports the OpenDRIVEstandard and provides a way to define the route that the ego and traffic vehicles should drive through the scene in which pedestrians and cyclists could be included.

 2. Scenario Database

Whether you are working on an ADAS feature or fully autonomous vehicle, the scenario database needs to provide a comprehensive suite of tests that are appropriate to the system under test. These tests must include the regulatory tests and it is increasingly important that real world edge cases are also covered.

Here at Claytex, we have a long history of supporting open standards such as Modelica and FMI. For the scenario definition we are providing support for OpenSCENARIO standard as well as our own in-house format. Through our involvement in the D-RISK project we are working with our project partners to develop a comprehensive database of edge cases together with test automation and assessment methods.

3. Scenario Editor

The Scenario Editor is used to define where and how dynamic objects, such as vehicles and pedestrians move on the road network. Scenarios can be created for vehicle dynamics simulation and traffic simulation or a combination of both. Scenarios can be as simple as following a predefined road profile with route definitions or they can be complex traffic situations with many vehicles, pedestrians on complex road networks.

Using the OpenDRIVE map provided by rFpro with their digital twins we can define the route that the ego, traffic and pedestrians should follow through the scene. This ability to define new scenarios is in addition to the support for the OpenSCENARIO standard to be able to read in existing scenarios.

Assessment and Validation

This module focuses on the different performance metrics needed to assess and validate the behaviour of the system under test in different scenarios. We capture a wide range of performance metrics during the simulation to enable the development of performance ratings (KPI) that are appropriate for the system and scenario. For example, a lane keeping algorithm needs different KPI to an automated emergency braking system or fully autonomous vehicle. By collecting a wide range of metrics such as accelerations, velocities, and distance between objects, we can adjust the KPI in appropriate ways and provide an extensible plugin architecture so that you can include your own metrics into the assessment.

In Conclusion

ADAS/AV system development incorporates multidisciplinary knowledge from many different aspects as highlighted above. These different components will be dissected and discussed individually in further blog posts in the coming weeks and months related to our solutions and the reasoning behind each step taken.

Written by Amina Hamoud – Project Engineer

Please get in touch if you have any questions or have got a topic in mind that you would like us to write about. You can submit your questions / topics via: Tech Blog Questions / Topic Suggestion