The Sandbox knowledge hub discusses many of the crucial issues affecting the development, engineering, use and regulation of Autonomous Vehicles.
Scroll to explore
Automated Testing Methodologies for Autonomous Vehicles
The safety of an autonomous vehicle is paramount, but the testing systems needed to ensure fully safe system performance are in their infancy. Meeting new standards such as SOTIF (Safety Of The Intended Functionality) with the increasing complexity of control systems that are developed to maximise an AV’s ODD (Operational Design Domain), requires extensive testing to prove the capability.
While creating a test scenario is not easy, automating the creation and testing of a vehicle is even more challenging. Choosing which scenarios should be run and their order is critical in mapping the performance of the AV across every variable, which is what I’ll explain in this blog post.
The Simulation Cycle
The Simulation Cycle is the process of creating, running and analysing results from a scenario. In the cycle, the scenarios are designed and created by the Test Manager and run in Simulation. Test automation using the Simulation Cycle can be conducted with Claytex’s Simulation Manager for rFpro.
The Simulation Cycle
Scenario generation is the process of taking the variable set from the Test Manager and creating the test case in simulation. Each simulation includes:
Ego Vehicle with the system under test i.e. the controller
The Road Network
Road Features (Pedestrian Crossing, Traffic Lights, etc.)
Traffic (Vehicles and Pedestrians)
The scenario can be defined using OpenScenario or the more simple mroute alternative used by Claytex. It is also possible to include performance monitoring inside the scenario so that specific state requirements can be watched.
The Simulation block repeats the Vehicle Control Cycle (Simulation control for vehicle automation) that is executed every step of the simulation until the scenario is complete. During or after the simulation, the performance of the controller is measured against targets set by the test manager.
It is critical the simulation is deterministic so that tests are repeatable and diagnosable in the event of a failure. In addition, physics based sensors with accurate noise (Lidar modelling with weather) and an environment with real world properties and accurate dynamic models for all objects in the scene are required for real world correlation.
Rewards and penalties are accumulated by the ego vehicle’s control system in navigating the scenario and are fed in to the Test Manager. Feedback is essential in understanding the performance of an AV; however, special notice should be taken of the control system trying to maximise the rewards with unintended behaviour, such as not moving, in order to avoid penalties. Ensuring the agent behaves as desired is a key requirement and the subject of SOTIF. By inspecting the results of a test, unintended behaviour can be identified and fixed by modifying the reward criteria, such as introducing a penalty while the agent has not reached its destination state. Of course this can introduce new issues, highlighting the reason for extensive testing.
A starting point for creating rewards for autonomous vehicle systems could be to look at the Responsibility-Sensitive Safetymodel introduced by Mobileye. Building on this, tiers could be introduced to stop large safety infractions occurring or being chosen over multiple smaller penalties. For example, if the vehicle were stuck in traffic and building up a penalty for being stationary, it should not try to force its way forward by crashing in to the vehicles in front.
The Test Manager is responsible for choosing which parameters to focus on, the order they are tested in and how best to identify the ODD boundaries to high accuracy. For example, the Test Manager may prioritise testing for operating in extra urban areas, while not prioritising edge cases and operating extra safe in the rare occasions when these occur. This would however require the Test Manager to understand the rarity of each scenario occurring in the real world.
The methodologies listed in the table below are some approaches for testing the ODD. Many test parameters are dependant on other parameters, for instance the rate of rainfall is irrelevant if there are no clouds in the sky. This can be used to reduce the number of test cases required for concise testing.
Tests each parameter individually
Quickly covers the test space
Chooses values randomly
Easy to Implement
Create a scenario where the actor is most likely to fail
Identifies where the agent performs worse in previous tests and configures the scenario accordingly
Test the Actors and environment to accurately recreate another testing domain (SiL, HiL, ViL, Real World)
Required for cross domain application
Use Performance Indicators to create a scenario to specifically target weaknesses of previous test cases
Targets weaknesses to find failures much quicker than other methods. Hardest to Implement
Create tests as different as possible to all the previously conducted tests
Only possible after first order tests complete
Conducting test cases created as the result of regulations such as SOTIF
Different techniques such as STPA and FMEA can be used
Take for example an overtaking scenario with three vehicles with the states shown below. By repeating the scenario with different start conditions, the parameter range in which an overtake can occur can be defined.
X Velocity (m/s)
t = 0
X Position (m)
t = 0
X Position (m)
t = 0
Control system (5m/s)
20 (now just overtaken safely)
-10, -20, -30
20, 50, 100
60, 20, -20
The ego vehicle requires four seconds to cleanly overtake T0 at 5m/s and leave a 10 meter gap. This means vehicle T1 must be more than 20 + 4v meters away. This relationship can be estimated by conducting tests varying the velocity and initial distance of T1.
Using the sensitivity method the test cases are depicted on the graph below. The dark blue crosses require 2^n tests and all crosses require 3^n test cases. As the number of parameters (n) increases the number of test cases gets large very quickly. Therefore, test automation and a test manager are essential to test efficiently and quickly determine any points of failure the ego vehicle control system has. Testing across the coverage map ensures there are no pockets of specific scenarios where a manoeuvre is not possible where scenarios around it are.
The worst case method is designed to conduct testing on scenarios where a manoeuvre is only just possible, as shown by the dashed line. This would help define the boundary between the scenarios in which the vehicle would and wouldn’t overtake and test if the scenarios match what is possible. By combining the sensitivity method with the worst case method, the Test Manager can increase its confidence in the vehicle only overtaking in the range of scenarios where an overtake is possible.
Coverage map for scenarios when attempting an overtake manoeuvre
In summary, by carefully testing select scenarios, the behaviour of an autonomous vehicle can be extrapolated to any scenario with high confidence.
Written by Rob Smith – Project Engineer
Please get in touch if you have any questions or have got a topic in mind that you would like us to write about. You can submit your questions / topics via: Tech Blog Questions / Topic Suggestion
The AVSandbox Knowledge Hub
Discover more about what makes AVSandbox unique. Explore our AVSandbox knowledge hub and find out about the issues, challenges and exciting developments that are behind the growth of the market for autonomous vehicles and advanced driver assistance systems.
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.