The Sandbox: the AVSandbox knowledge hub
The Sandbox knowledge hub discusses many of the crucial issues affecting the development, engineering, use and regulation of Autonomous Vehicles.
Scroll to explore
Ultra photorealism in AVSandbox
Thanks to the hard work of rFpro, ultrarealistic light rendering is now available in the AVSandbox toolkit.
This is light rendering by raytracing: the computational technique of modelling light as a series of rays that bounces around a scene, and delivers rich lit textures to the objects in it.
With this technique, the light behaves far closer to how it is observed in real life. Reflection, refraction and scattering are represented more precisely and specific phenomena such as soft shadows and motion blur are also enabled.
(image credit)
By using raytracing in our product, you will be able to train and test your computer vision stack with far more realistic and accurate imagery than before, all generated from a computer rather than real driving!
A science
There are many interesting nuances to this science of rendering. The way light scatters from objects is continuous, while we are restricted to calculating it as a series of discrete rays; but with a large enough amount of sampling and techniques such as path tracing, where the likeliest path of light is considered before it is calculated, it becomes accurate enough to be called realistic.
rFpro has optimised their raytracing to run fast despite the intensity of the task, for example by not calculating paths of light that are highly unlikely to affect the image at the camera. This means that while the raytracing cannot be done real-time, it is still quite rapid in our synchro-step mode, where all of the vehicle and sensor calculations are allowed to finish before the next timestep is advanced to.
Examples
Observe this drive through our Coventry map with raytraced light:
Notice in particular, the quality of reflections in puddles and car windows.
This is the scenario behind it, but using rasterisation instead- still perfectly detailed for applications such as real-time simulating, but without the same convincing lighting and shadow complexity. Most other simulation engines for self-driving cars operate at this or a lesser definition.
There is no doubt that one is more realistic than the other!
Here is another example of a raytraced scene. Notice how the bus window is both reflective and transparent, and the realistic feel of the buildings.
Device effects
Apart from modelling the behaviour of the light in the scene, visual distortions borne out from the camera itself are also included. These are things such as shutter size and lens shape, which impacts the final projection of light unto an image, and shutter speed and rolling shutter effects, which leads to motion blur and deformation.
With a long exposure time (low shutter speed), you can see the effects of motion blur, like this bus moving past at 110km/h with 0.1 second exposure:
And rolling shutter distortion, where different rows of pixels in the camera are written to at different times – witness the same bus appearing to lean backwards, with a fast shutter speed but a 0.1 rolling shutter factor:
Finally, lens distortions are also included. In this case, the bus has instant shutter but is being recorded with an equisolid lens:
This combination of detailed light modelling and device effects provides the next level of image generation for testing your perception stack.
Keep an eye out for our release announcements.
Please get in touch if you have any questions or have got a topic in mind that you would like us to write about. You can submit your questions / topics via: Tech Blog Questions / Topic Suggestion
Ultra photorealism in AVSandbox
Thanks to the hard work of rFpro, ultrarealistic light rendering is now available in the AVSandbox toolkit. This is ...
Tackling High Development Costs: How AVSandbox Can Accelerate Your Autonomous Vehicle Deployment
Reducing costs of autonomous vehicle development without compromising AV Safety The development and successful deployment of autonomous vehicles is ...
Determinist Traffic Simulation
Introduction In my previous blog deterministic scenario simulation, I detailed why we define our simulator deterministic and what is ...