The Sandbox: the AVSandbox knowledge hub

In the recent year, when discussing with businesses in the CAV sector, we can clearly observe a great variety of projects, with very different goals emerging. This makes the process of validating these systems a challenging task due to the different requirements to validate each system.

To face these ever-changing demands, Claytex AV simulation solution offers a variety of tools for engineers to set up and test anything: from sensor hardware in the loop to full autonomous driving. These features combined with the Sensor Realistic Simulation with rFpro allow engineers to extract virtually any data from cameras, sensors, and vehicle telemetry data from simulation. This data is crucial for the validation of your system.

To make use of this data, our solution contains a mechanism of Custom Metric Calculators. These allow any sort of calculation to be done using data from simulation and then defining if a simulation test has passed or failed. As this is done via coding, engineers have the freedom to code their own method of validation, and it will be run automatically by Simulation Manager.

To demonstrate custom metrics into a practical way, in this blog article, we will simulate the creation of a test metric calculator. This will define if an autonomous airport service truck is stopping accurately at a target point for loading and unloading.

Video 1. Scenario to be simulated

Creating your own custom metric

Although the coding will not be covered in this blog, let’s start by having a very brief overview on how a custom metric calculator works. In this case, we will use .NET to create a .dll file, that will be loaded and run by Simulation Manager. In our project, the reference to MetricCalculatorInterface.dll should be added. This should give you everything required to start writing your own customer metric.

This reference gives you access to several methods and tools, however the most important is the calculate() function. This function will be called once a simulation is complete and will contain all data collected during the simulation.

For more technical details about how to create a Custom Metric Calculator, please refer to the Simulation Manager help or contact us.

The Calculate function

This is where all the magic happens. In this function, you will be able to write the code that defines if a test will be passed or failed.

For the stopping vehicle example, one possible approach is to find in which part of the simulation the vehicle was not moving. In other words, its velocity was 0.

We can get this information from the telemetry data. You are also able to visualize this data and other simulation data in the Simulation Manager built-In analysis tools.

As show in Figure 1, you can see the vehicle was not moving from simulation time 18 to 22.7 seconds. Having this information, you can now look back into the telemetry and find its position (XY) at that given time.

Figure 2. Vehicle X and Y position over time

Assuming the position of the vehicle is also important, you can also extract the heading data and confirm the angle is as expected.

Figure 3. Vehicle heading angle over time

The graphics above confirm what we expect, the position (XY) and heading are constants while the vehicle was stopped. This can be used in code to confirm that the data is valid.

Once you have this information, you can then compare if the vehicle XY position and heading match the coordinates. The target position and heading will be provided by stop events in your scenario. If the stop was not accurate enough, the metric calculator can return false and inform Simulation Manager that the test has failed.

Custom Metric Parameters

Custom metrics also have an embedded mechanism for defining custom variables. This gives you extra freedom to have different parameters for each scenario. That prevents your metric calculator’s form having hard-coded values in your code. In our example, we can use these variables to allow the user to define a tolerance for the stop position distance and angle, as well as value to ignore possible small noises in the telemetry data.

Figure 4. User interface for selecting metric calculator and parameters

Conclusion

Claytex’ AV simulation solution has a flexible and programmable mechanism that allows you to define your own test metrics. This gives you full control on how your simulation will be validated. These can be created by you or one of our engineers.

Simulation Manager can run as many metrics calculators as necessary. A summary of all passed/failed metrics will be provided for each test via our user interface.

Written by: Felipe Ferrary – Senior Software Engineer at Claytex

Please get in touch if you have any questions or have got a topic in mind that you would like us to write about. You can submit your questions / topics via: Tech Blog Questions / Topic Suggestion