The Sandbox: the AVSandbox knowledge hub

One of the biggest challenges for autonomous vehicles (AV) is how to argue their safety case. The first step towards a state of informed safety is establishing the capability of an automated driving system (ADS) by defining its operational design domain (ODD).

What is an ODD?

“ODD is an operating condition under which a given driving automation system or feature thereof is specifically designed to function, including, but not limited to, environmental, geographical, and time of day restrictions, and/or the requisite presence or absence of certain traffic or roadway characteristics.” SAE J3016

The operational design domain (ODD) can be used to restrict where the automated driving system (ADS) is valid and thus confine the scope of the safety case, as well as the verification. In order to ensure this, use cases are needed to provide a strategy for a collection of operating conditions (OCs) and make sure that it is still within the scope of the ODD. The ODD defines the functional boundary of the system and modelling it with the structure below makes it modular and generalizable across potential use cases.

To complete the safety case, there is a need to ensure that the ADS will not exit its ODD. The ODD taxonomy defined in PAS 1883 helps specify and implement minimum safety requirements using sets of ODD attributes. PAS 1883 has three high level attributes as shown in the figure below.

• Scenery which is everything that is geo-stationary
• Environment which is everything that is atmospheric
• Dynamic Elements which are all the movable objects

Figure 1. ODD Taxonomy as per PAS 1883

The definition of an ODD can be presented in two formats: tabular format or as textual format (See PAS 1883 | BSI (bsigroup.com)).

Scenario Factor Analysis

In a recent project, we created an ODD for an autonomous baggage tug where the ODD was created from the point-of-view of the airport operator and is defined based on the airside operating environment. It is important to realize that the ODD does not include the use case of the vehicle, rather, the ODD provides the boundary conditions within which the vehicle should be able to operate. Therefore, within this project, we also outlined the use case for the autonomous baggage tug.

To be able to test if an autonomous vehicle (AV) is capable of operating safely within its ODD, we first need to be able to define the set of scenarios that we can simulate. Currently this is a manual process for an engineer to analyse the ODD and use their knowledge of the vehicle use case and intended operation to define a set of scenarios such as an overtake manoeuvre or right turn at a junction. The scenarios on their own though are not enough to enable adaptive testing of the AV; we need additional information. We have started to define a framework that bridges this gap between the ODD and what is required to define the scenarios and enable adaptive testing.

One element of this framework is called Scenario Factor Analysis which is the process of deciding the most useful test case to simulate based on a combination of the risk, frequency and test coverage of test parameters individually, and as part of a scenario. The risk and frequency of each parameter is determined manually and is related to the ODD.

Factor	Description	How to determine value
Risk	The probability and severity of failing requirements. How much damage could be caused to people and property	Calculated by an engineer using risk analysis
Coverage	How often a discrete parameter is chosen or the distribution of a continuous parameter	Using the parameters values from previous test cases.
Frequency	The likelihood of a test case occurring when the AV is in operation	Calculated by an engineer using statistical analysis of the ODD

Figure 2. Scenario Factor Description Table

The coverage factor is intended to ensure the adaptive testing covers the whole ODD without leaving big gaps in testing. The fewer tests that can be completed, the more important this factor is and the less chance the testing has to explore high risk or high frequency test cases. If there is time to conduct more testing, we can select test cases that can dive deeper in to the worst case scenarios and evaluate more parameter variations. The exact formula for creating new test cases is especially important.

Many test parameters are dependent on other parameters, for instance the rate of rainfall is irrelevant if there are no clouds in the sky. This can be used to reduce the number of test cases required for concise testing. As the number of tests increases, the probability of a failure in the AV is inversely correlated to the number of tests conducted. When using test automation to prove performance, human performance can be used as a benchmark.

Proposed Test Process

While test cases are often created using the ODD, there is currently no standard procedure for digitizing the ODD or using the ODD to create test cases automatically. With a digital ODD, adaptive testing can use information about the operating domain to generate the optimal next test case. Depicted in figure 3, the goal of this step is to identify short, common, risky and untested scenarios from within the vehicle ODD, such as a vehicle overtake manoeuvre or right turn at a traffic light. These are the primary tests of concern and passing them will result in obtaining the highest confidence in the AV in the shortest amount of time.

The process and adaptive test manager have been implemented as a proof-of-concept within our autonomous vehicle simulator and further exploration of our framework and findings will be disseminated in future blog posts.

Written by Amina Hamoud – Project Engineer

Please get in touch if you have any questions or have got a topic in mind that you would like us to write about. You can submit your questions / topics via: Tech Blog Questions / Topic Suggestion