The Sandbox: the AVSandbox knowledge hub

Introduction

The safety of an autonomous vehicle is paramount, but the testing systems needed to ensure fully safe system performance are in their infancy. Meeting new standards such as SOTIF (Safety Of The Intended Functionality) with the increasing complexity of control systems that are developed to maximise an AV’s ODD (Operational Design Domain), requires extensive testing to prove the capability.

While creating a test scenario is not easy, automating the creation and testing of a vehicle is even more challenging. Choosing which scenarios should be run and their order is critical in mapping the performance of the AV across every variable, which is what I’ll explain in this blog post.

The Simulation Cycle

The Simulation Cycle is the process of creating, running and analysing results from a scenario. In the cycle, the scenarios are designed and created by the Test Manager and run in Simulation. Test automation using the Simulation Cycle can be conducted with Claytex’s Simulation Manager for rFpro.

Scenario Generation

Scenario generation is the process of taking the variable set from the Test Manager and creating the test case in simulation. Each simulation includes:

• Ego Vehicle with the system under test i.e. the controller
• The Road Network
• Road Features (Pedestrian Crossing, Traffic Lights, etc.)
• Traffic (Vehicles and Pedestrians)
• Weather

The scenario can be defined using OpenScenario or the more simple mroute alternative used by Claytex. It is also possible to include performance monitoring inside the scenario so that specific state requirements can be watched.

Simulation

The Simulation block repeats the Vehicle Control Cycle (Simulation control for vehicle automation) that is executed every step of the simulation until the scenario is complete. During or after the simulation, the performance of the controller is measured against targets set by the test manager.

It is critical the simulation is deterministic so that tests are repeatable and diagnosable in the event of a failure. In addition, physics based sensors with accurate noise (Lidar modelling with weather) and an environment with real world properties and accurate dynamic models for all objects in the scene are required for real world correlation.

Feedback

Rewards and penalties are accumulated by the ego vehicle’s control system in navigating the scenario and are fed in to the Test Manager. Feedback is essential in understanding the performance of an AV; however, special notice should be taken of the control system trying to maximise the rewards with unintended behaviour, such as not moving, in order to avoid penalties. Ensuring the agent behaves as desired is a key requirement and the subject of SOTIF. By inspecting the results of a test, unintended behaviour can be identified and fixed by modifying the reward criteria, such as introducing a penalty while the agent has not reached its destination state. Of course this can introduce new issues, highlighting the reason for extensive testing.

A starting point for creating rewards for autonomous vehicle systems could be to look at the Responsibility-Sensitive Safetymodel introduced by Mobileye. Building on this, tiers could be introduced to stop large safety infractions occurring or being chosen over multiple smaller penalties. For example, if the vehicle were stuck in traffic and building up a penalty for being stationary, it should not try to force its way forward by crashing in to the vehicles in front.

Test Manager

The Test Manager is responsible for choosing which parameters to focus on, the order they are tested in and how best to identify the ODD boundaries to high accuracy. For example, the Test Manager may prioritise testing for operating in extra urban areas, while not prioritising edge cases and operating extra safe in the rare occasions when these occur. This would however require the Test Manager to understand the rarity of each scenario occurring in the real world.

The methodologies listed in the table below are some approaches for testing the ODD. Many test parameters are dependant on other parameters, for instance the rate of rainfall is irrelevant if there are no clouds in the sky. This can be used to reduce the number of test cases required for concise testing.

Name	Methodology	Notes
Sensitivity	Tests each parameter individually	Quickly covers the test space
Random	Chooses values randomly	Easy to Implement
Worst Case	Create a scenario where the actor is most likely to fail	Identifies where the agent performs worse in previous tests and configures the scenario accordingly
Correlation	Test the Actors and environment to accurately recreate another testing domain (SiL, HiL, ViL, Real World)	Required for cross domain application
Adaptive	Use Performance Indicators to create a scenario to specifically target weaknesses of previous test cases	Targets weaknesses to find failures much quicker than other methods. Hardest to Implement
Gap	Create tests as different as possible to all the previously conducted tests	Only possible after first order tests complete
Safety	Conducting test cases created as the result of regulations such as SOTIF	Different techniques such as STPA and FMEA can be used

Example

Take for example an overtaking scenario with three vehicles with the states shown below. By repeating the scenario with different start conditions, the parameter range in which an overtake can occur can be defined.

Vehicle	X Velocity (m/s) t = 0	X Position (m) t = 0	X Position (m) t = 0
Ego	Control system (5m/s)	0	20 (now just overtaken safely)
T0	0	10	0
T1	-10, -20, -30	20, 50, 100	60, 20, -20

The ego vehicle requires four seconds to cleanly overtake T0 at 5m/s and leave a 10 meter gap. This means vehicle T1 must be more than 20 + 4v meters away. This relationship can be estimated by conducting tests varying the velocity and initial distance of T1.

Using the sensitivity method the test cases are depicted on the graph below. The dark blue crosses require 2^n tests and all crosses require 3^n test cases. As the number of parameters (n) increases the number of test cases gets large very quickly. Therefore, test automation and a test manager are essential to test efficiently and quickly determine any points of failure the ego vehicle control system has. Testing across the coverage map ensures there are no pockets of specific scenarios where a manoeuvre is not possible where scenarios around it are.

The worst case method is designed to conduct testing on scenarios where a manoeuvre is only just possible, as shown by the dashed line. This would help define the boundary between the scenarios in which the vehicle would and wouldn’t overtake and test if the scenarios match what is possible. By combining the sensitivity method with the worst case method, the Test Manager can increase its confidence in the vehicle only overtaking in the range of scenarios where an overtake is possible.

In summary, by carefully testing select scenarios, the behaviour of an autonomous vehicle can be extrapolated to any scenario with high confidence.

Written by Rob Smith – Project Engineer

Please get in touch if you have any questions or have got a topic in mind that you would like us to write about. You can submit your questions / topics via: Tech Blog Questions / Topic Suggestion