The Sandbox: the AVSandbox knowledge hub

The Sandbox knowledge hub discusses many of the crucial issues affecting the development, engineering, use and regulation of Autonomous Vehicles.

AVSandbox | Autonomous Vehicle Simulation

Scroll to explore

The Sandbox | Knowledge Hub | AVSandbox

Verification & Validation of a Simulation Model

Simulation is the way forward to autonomous driving because it is the only way to deliver the level of testing needed to create reliable systems (“(31) Is Simulation the Way Forward to Fully Autonomous Driving? – YouTube”). Therefore, it has become necessary to have traceable, robust Verification and Validation (V & V) processes in place to support the evidence that the tests are providing. However, there are no well established ways to validate a model, and the V & V process is often tailor-made. That leads to huge differences depending on the developers themselves, the safety culture and the environment they operate in. In this blog post we will describe the term and theory of V & V (“A bit of theory“). Then, the steps that need to be taken to validate a model will be laid out (“Steps required to validate a model“). Finally, we will illustrate via the validation of a Radar Antenna conceptual model (“Example“). Of course, V & V is a vast and complex subject and we will get to cover more of it in future blog posts.

A bit of theory

Basics of  V & V

Verification : “Ensuring that the computer program of the computerised model and its implementation are correct.” Validation : “Substantiation that a computerised model within its domain of applicability possesses.” – Sargent (2011). The validity of a model is always defined in respect to a certain purpose and a set of experimental conditions:

  • What are the questions that the model has been developed to answer ? Validity is done in regards to each of those questions.
  • Identify the model’s output variable. Define the required acceptable range of accuracy for each variable.
  • Use means and variances for random variables.

It is important to note that it is often too costly to prove absolute validity over the domain of application, so we are looking for sufficient confidence.

Approaches and time frames

There are four approaches on how to lead a validation process:

  • The developer decides on the validity.
  • The user is part of the decision.
  • External validation (independent verification and validation (IV&V) for large scale only).
  • Scoring model (Balci (1989), Gass (1993), and Gass and Joel (1987), rare and unpopular).

There are also two possible time frames:

  • Validation conducted along with development (better).
  • After the development. (One conclusion made is that performing a complete IV&V effort after the simulation model has been completely developed, is both extremely costly and time consuming, especially for what is obtained). It is usually better to only validate the validation work done by the team.

Validation techniques

  • Animation to see what is going on, along with graphical / visual validation.
  • Comparison with other Models.
  • Degenerate test (look for nulls, infinites, redundancies, incoherencies, ..).
  • Event Validity (compare events with real life event, number, magnitude, ..).
  • Idiot proof value.
  • Expert’s opinions of the output
  • Historical Data Validation
  • Parameter influence on the output (does increasing a increase b? In reasonable proportions?).
  • Multistage Validation: “Naylor and Finger (1967) proposed combining the three historical methods of rationalism, empiricism, and positive economics into a multistage process of validation. This validation method consists of (1) developing the model’s assumptions on theory, observations, and general knowledge, (2) validating the model’s assumptions where possible by empirically testing them, and (3) comparing (testing) the input-output relationships of the model to the real system.”

Defining the paradigm

A V & V process starts by defining the paradigm of the problem:

  • The problem entity: A description of the real world that we are trying to simulate.
  • The conceptual model:  The theoretical model of this system.
  • The computerised model: The actual model (code) that implements the conceptual model.

These interact as such:

Figure 1: Simplified version of the modelling process (Sargent, 2011), (Junjie Liu, 2011)

The process therefore relies on three main steps:

  • Conceptual Model Validation
  • Computerised model verification
  • Operational Validation

Steps required to validate a model

Conceptual Model & Conceptual Model Validation

According to Robert G. Sargent – Verification and Validation of Simulation Models, conceptual model validity is determining that:

  • (1) the theories and assumptions underlying the conceptual model are correct.
  • (2) the model’s representation of the problem entity and the model’s structure, logic, and mathematical and causal relationships are “reasonable” for the intended purpose of the model.

There are many definitions of a conceptual model:

  • Translating modelling requirements into detailed simulation specifications (and associated simulation design) which fully satisfy requirements ”.
  • The agreement between the simulation developer and the user about what the simulation will do ”.
  • An abstraction of real world that serves as a frame of reference for federation development by documenting simulation-neutral views of important entities and their key actions and interactions ”.

In this document, we will use S. Robinson’s definition (Robinson, 2006): “ The conceptual model is a non-software specific description of the simulation model that is to be developed, describing objectives, inputs, outputs, content, assumptions and simplifications of the model ”. Specifically, the validation of a conceptual model will depend on how well those underlined words are defined and described.

Computerised Model & Computerised Model Verification

The Computerised model is the actual algorithms we will use to put into practice the conceptual model. For example, we rely on the ray tracing technique to build the radar model (Zhengqing Yun, 2015). The verification step; that is making sure that, once the theory is reasonable, the code does what it should be doing; is then the daily work of a software engineer:

  • Error free simulation; Correct programming of the language; time-flow mechanism properly implemented, ..
  • Static testing:
    • Structured walkthrough
    • Correctness proofs
    • Structure properties of the program
  • Dynamic testing:
    • Happens during execution
    • Input output relationship
    • Internal consistency check
    • Reprogramming critical components to determine if the same results are obtained.

Operational Model Validation

In this step we want to make sure that the behaviour of our model corresponds to the  problem entity. It is what we think about when we talk about validation. For this step we can rely on Erwin, Roth Tobias J., Dirndorfer Alois, Knoll who use 6 steps to validate their sensor model:

  • (1) Carrying out real test drives on the basis of a previously defined test maneuver catalog, Logging experimental parameters and recording perception sensor and position reference sensor data during the test drive.
  • (2) Automated generation of the virtual test scenario data for the simulation toolchain based on the recorded position reference sensor data.
  • (3) Parameterisation of the sensor models according to the parameters of the real test drive setup and sensor equipment properties.
  • (4) Execution of virtual test drives in VTD on the basis of the virtual test scenario data generated beforehand.
  • (5) Recording synthetically generated predictive sensor data while running a virtual test scenario in VTD.
  • (6) Comparison of real and synthetically generated sensor data by means of MATLAB.

Figure 2: Analysis and validation of perception sensor models in an integrated vehicle and environment simulation – Erwin, Roth Tobias J., Dirndorfer Alois, Knoll

Example: Conceptual Model

Let’s illustrate with the structure of the conceptual model validation of an antenna beam pattern. After defining the paradigm model, we need to clearly lay out its objectives, inputs, outputs, content, assumptions and content. More details on the antenna is given here : Modelling of Phased Array Antenna – Claytex.

The objectives

The aim of the antenna model is to create a power distribution around the antenna, for emitting and transmitting, as well as an illuminated area (azimuth + elevation) where the wave will be emitted towards. Because we are using rays to model the wave, each ray will be assigned a power coefficient upon leaving and arriving to the antenna, according to a beam pattern.

The inputs and outputs

Our modelled antenna is a phased array with an adaptable number of array elements and spacing that allow for control over the beam pattern. The frequency of emission will obviously impact the pattern as well. Finally, the orientation of the antenna in azimuth and elevation to the referential will define the illuminated area.

The content

The details of the model and formulas used are given here (Modelling of Phased Array Antenna – Claytex.). The formulas vary given the definition of the parameters, as explained by Peter Delos, 2020: “There are some variations in equations used across literature depending on how parameters were defined in the linear array“. The validity and context of these formula must be justified! Are they valid under this circumstances (frequency, weather, antenna state, ..)?

The assumptions and simplifications

  • This model assumes a perfect spacing of the antenna arrays. In reality there might be slight delta spacing that will impact the beam pattern.
  • There are main lobes of higher orders when a main beam is shifted more than 30 degrees.
  • The state of the arrays is not considered here; aging and damaging will impact the pattern.

It is important to note that each assumption is also a point of improvement for the model. Thereby, an aging function can be added relying on different studies on the subject (“On the Effect of Aging on Digital Sensors” – CSEE Department LTCI, CNRS, Télécom ParisTech).

What next

In order to carry on the validation of the antenna beam pattern used in the radar, we will then have to verify the computerised model. That is when we will test our code against theory with data like this:

The Sandbox | Knowledge Hub | AVSandbox

Figure 3: Verification step of the antenna pattern

Finally, the operational validation will consist of comparing the data from the computerised model against field test data (see Operational Model Validation). In this author’s opinion, the most important thing to do is to structure your V & V process and make sure all steps are justified, even the most obvious ones. It will also help you write a detailed document that is understandably more and more required in the industry.

Written by: Geoffroy Heurtier – Project Engineer

Please get in touch if you have any questions or have got a topic in mind that you would like us to write about. You can submit your questions / topics via: Tech Blog Questions / Topic Suggestion

The AVSandbox Knowledge Hub

Discover more about what makes AVSandbox unique. Explore our AVSandbox knowledge hub and find out about the issues, challenges and exciting developments that are behind the growth of the market for autonomous vehicles and advanced driver assistance systems.

Go to Top