The Sandbox: the AVSandbox knowledge hub

Thanks to the hard work of rFpro, ultrarealistic light rendering is now available in the AVSandbox toolkit.

This is light rendering by raytracing: the computational technique of modelling light as a series of rays that bounces around a scene, and delivers rich lit textures to the objects in it.

With this technique, the light behaves far closer to how it is observed in real life. Reflection, refraction and scattering are represented more precisely and specific phenomena such as soft shadows and motion blur are also enabled.

(image credit)

By using raytracing in our product, you will be able to train and test your computer vision stack with far more realistic and accurate imagery than before, all generated from a computer rather than real driving!

A science

There are many interesting nuances to this science of rendering. The way light scatters from objects is continuous, while we are restricted to calculating it as a series of discrete rays; but with a large enough amount of sampling and techniques such as path tracing, where the likeliest path of light is considered before it is calculated, it becomes accurate enough to be called realistic.

rFpro has optimised their raytracing to run fast despite the intensity of the task, for example by not calculating paths of light that are highly unlikely to affect the image at the camera. This means that while the raytracing cannot be done real-time, it is still quite rapid in our synchro-step mode, where all of the vehicle and sensor calculations are allowed to finish before the next timestep is advanced to.

 

Examples

Observe this drive through our Coventry map with raytraced light:

Notice in particular, the quality of reflections in puddles and car windows.

This is the scenario behind it, but using rasterisation instead- still perfectly detailed for applications such as real-time simulating, but without the same convincing lighting and shadow complexity. Most other simulation engines for self-driving cars operate at this or a lesser definition.

There is no doubt that one is more realistic than the other!

Here is another example of a raytraced scene. Notice how the bus window is both reflective and transparent, and the realistic feel of the buildings.

 

Device effects

Apart from modelling the behaviour of the light in the scene, visual distortions borne out from the camera itself are also included. These are things such as shutter size and lens shape, which impacts the final projection of light unto an image, and shutter speed and rolling shutter effects, which leads to motion blur and deformation.

With a long exposure time (low shutter speed), you can see the effects of motion blur, like this bus moving past at 110km/h with 0.1 second exposure:

And rolling shutter distortion, where different rows of pixels in the camera are written to at different times – witness the same bus appearing to lean backwards, with a fast shutter speed but a 0.1 rolling shutter factor:

Finally, lens distortions are also included. In this case, the bus has instant shutter but is being recorded with an equisolid lens:

This combination of detailed light modelling and device effects provides the next level of image generation for testing your perception stack.

Keep an eye out for our release announcements.

Please get in touch if you have any questions or have got a topic in mind that you would like us to write about. You can submit your questions / topics via: Tech Blog Questions / Topic Suggestion