A team of engineers from Duke University has unlocked a previously unrecognised 3D imaging capability of modern cameras by repurposing existing components.
The research demonstrates how image stabilisation and focus modules in modern digital cameras could be harnessed to achieve 3D imaging capabilities without additional hardware.
The team demonstrated the concept in a laboratory experiment using a small deformable mirror. The purpose of the experiment was to extract depth-of-field information from a single shot image without suffering any trade-offs in image quality.
When integrated into commercial cameras and other optical technologies, this visualisation technique could improve core functions, like image stabilisation, and increase the speed of autofocus, which would enhance the quality of photographs.
‘Real scenes are in three dimensions and they’re normally captured by taking multiple images focused at various distances,’ said Patrick Llull, Duke Imaging and Spectroscopy Program (DISP), Duke University. ‘A variety of single-shot approaches to improve the speed and quality of 3D image capture has been proposed over the past decades. Each approach, however, suffers from permanent degradations in 2D image quality and/or hardware complexity.’
The research team, led by David Brady, a professor at Duke, was able to overcome these hurdles, developing an adaptive system that can accurately extract 3D data while maintaining the ability to capture a full-resolution 2D image without a dramatic system change, such as switching out a lens.
Brady and his team have presented their findings in Optica, the open access journal from The Optical Society.
Modern digital cameras, especially those with video capabilities, are frequently equipped with modules that take the jitter out of recordings. They do this by measuring the inertia or motion of the camera and compensate by rapidly moving the lens – making multiple adjustments per second – in the module. This same hardware can also change the image capture process, recording additional information about the scene. With proper software and processing, this additional information can unlock the third dimension.
The camera is programmed to perform three functions simultaneously: sweeping through the focus range with the sensor, collecting light over a set period of time in a process called integration, and activating the stabilisation module.
As the optical stabilisation is engaged, it wobbles the lens to move the image relative to a fixed point. This, in conjunction with a focal sweep of the sensor, integrates that information into a single measurement in a way that preserves image details while granting each focus position a different optical response. The images that would have otherwise been acquired at various focal settings are directly encoded into this measurement based on where they reside in the depth of field.
For the paper, the researchers used a comparatively long exposure time to compensate for the set-up of the equipment. To emulate the workings of a camera, a beam splitter was necessary to control the deformable lens. This extra step sacrifices about 75 per cent of the light received. ‘When translated to a fully integrated camera without a beam splitter, this light loss will not be an issue and much faster exposure times will be possible,’ noted Llull.
The researchers then process a single exposure taken with the camera and obtain a data cube, including the all-focused 2D image as well as a depth map. This depth map data, in effect, describes the focus position of each pixel of the image. Since this information is already encoded into the single measurement, it’s possible to construct a depth map for the entire scene.
The final step is to process the image and depth map with a commercial 3D graphics engine. The resulting image can be used to determine the optimal focal setting for subsequent full-resolution 2D shots, as an autofocus algorithm does, but from only one image. Additionally, synthetic refocusing may be used on the resulting 3D imagery to display the scene as viewed at different depths by a human.
Though only performed in laboratory settings with surrogate technologies, the researchers believe the techniques they employed could be applied to basic consumer products. The result would be a more efficient auto-focusing process, as well as the added third dimension to traditional photography.
Further information:
Duke Imaging and Spectroscopy Program (DISP), Duke University
‘Image translation for single-shot focal tomography’, Patrick Llull et al., Optica, 2, 9, 822 (2015)
