Caption: Fraunhofer IIS demonstrated Lucid Vision Labs' Phoenix camera being used to inspect carbon fibres at Control 2018
Demand for polarised imaging in industry is on the rise, thanks to Sony’s recent release of a set of relatively low-cost polarised monochrome and colour imaging sensors, the IMX250MZR and IMX250MYR.
Built upon the Sony Pregius five-megapixel IMX250 CMOS sensor, the IMX250MZR/MYR incorporates a layer of nanowire micro-polariser arrays above the sensor’s photodiodes. Each array in the layer comprises four different angled polarisers – 0°, 45°, 90° and 135° – that are placed over one of four pixels in a 2 x 2 block on the sensor. An array of micro lenses is also positioned on top of the polariser layer of the sensor to reduce crosstalk between pixels.
By linearly interpolating the light passing through the four polarisers, a single intensity pixel value with an associated angle of linear polarisation (AoLP) and degree of linear polarisation (DoLP) can be calculated. These values can then be used to produce false colour images, which can reveal scratches, surface defects, and even the internal stresses of materials that would otherwise be undetectable using conventional imaging techniques.
Arnaud Destruels, European visual communication product manager at Sony Europe, said that despite it still being early days for the new sensor, Sony is already seeing strong demand for the technology, and expects this to continue in the future. ‘Intelligent transport systems and security will be key, but similarly, factory automation is also showing strong demand,’ he said.
Sony forecasts that the sensor’s uptake around the globe will mirror existing technology trends, with a strong element of factory automation in Asia, and robotics and intelligent transport systems in Europe.
Removing glare
Reflections or glare can make it difficult for machine vision systems to pick up defects, especially on shiny or transparent materials. These effects are caused by polarisation, a fundamental property of light that describes the direction in which the electric field of light oscillates. While the majority of illumination sources emit unpolarised light, reflections from non-metallic or semi-transparent surfaces, such as glass, plastics, or water, can cause some of it to become polarised in a particular direction – depending on the surface characteristics of the material. This can also happen when light passes from one medium to another.
Polarisation filters can be used, but the equipment set-up can be complex, often requiring multiple filters oriented to different directions of polarisation. Companies have also positioned micro-polarising filters on top of off-the-shelf image sensors, but this is expensive to manufacture.
Polarised imaging is suited to the inspection of carbon fibres because the fibres polarise light parallel to their direction. Credit: Fraunhofer IIS
Not only is the new Sony sensor more practical and more advanced than alternative polarised imaging solutions, Sony is able to produce the sensors at a much lower cost. This presents camera manufacturers with an opportunity. ‘With the significantly lower cost of Sony’s sensor, more companies will now be able to buy polarisation cameras and try them out, to see how they can augment their existing industrial inspection applications,’ remarked Jenson Chang, product marketing manager at Canadian company Lucid Vision Labs.
Lucid Vision was among the first to deliver a camera incorporating Sony’s new monochrome sensor to the machine vision market. The Phoenix polarisation 5.0 megapixel camera is supplied in a 24 x 24mm form factor, and can capture up to 24 frames per second via a GigE vision interface.
‘Most of the time they [Lucid Vision customers] are combining polarisation with other imaging technologies,’ Chang said. ‘They can use high-performance standard cameras to inspect most of what they need, then polarisation is just used to improve their accuracy in certain areas – so they don’t really need a high frame rate.’ Customers are instead requesting higher resolution, he added. However, this is limited by the sensor technology itself, rather than the camera.
Lucid Vision has plans to incorporate the IMX250MZR into other camera platforms in the future. In the meantime, the firm intends to release another version of the Phoenix camera featuring the IMX250MYR colour version of Sony’s polarised sensor in the coming months.
Teledyne Dalsa is another imaging firm that has identified the potentials of Sony’s polarised sensor, now offering the Genie Nano M2450 area scan polarisation camera. The model joins the company’s Piranha4 Polarization line scan camera, which was released towards the end of last year. This camera is based on a custom CMOS sensor that – similar to the Sony sensor, uses nanowire micro-polariser filters – but positioned in quad-linear architecture. The Piranha4 camera offers a maximum line rate of 70kHz and can output independent images of 0°, 90° and 135° polarisation states, as well as an unfiltered channel, which captures the total intensity, equivalent to a conventional image.
Teledyne Dalsa’s Genie Nano M2450 polarised model offers up to 35 frames per second via GigE Vision
The Genie Nano area camera incorporates Sony’s monochrome sensor and comes in a 21.2 x 29 x 44mm format. Offering an augmented 35 frames per second via GigE Vision, the camera’s higher frame rate can be attributed to Teledyne Dalsa’s TurboDrive technology – a mode of operation that speeds up the line and frame rates of GigE Vision cameras beyond the nominal link capacity of Gigabit Ethernet without affecting image quality.
Manufacturer Flir also plans to release a camera incorporating Sony’s new sensor. In addition to being available in a GigE Vision format, the upcoming Blackfly S polarised camera will also be offered with a faster USB3 interface, which will enable frame rates of up to 75fps.
Jennifer Yeung, director of marketing for Flir’s Integrated Imaging Solutions division, commented that this faster format will be especially useful for non-industrial polarised imaging applications, such as unmanned aerial surveillance (UAS). This will complement the IMX250MZR’s ability to adjust polarisation parameters quickly and automatically when compensating for changing lighting conditions, movement and orientation.
Lastly, Sony Europe’s own Image Sensing Solutions division will be unveiling a new polarised camera incorporating the IMX250MZR at Vision 2018 in Stuttgart, Germany, this November.
Polarisation in practice
Potential uses of polarised imaging include inspecting products wrapped in film or plastic through packaging, or inspecting shiny cylindrical or curved objects – both instances where reflections are in abundance.
‘We’ve seen a high demand for technology detecting surface defects – not just in metal, but also for glass and clear plastics, as these are hard to detect with standard imaging cameras,’ said Chang. ‘Even when inspecting opaque materials there are still flaws that are hard to see – especially small ones – with standard imaging cameras. However, these become much more obvious with polarised imaging.’
Surface defects can be difficult to identify on mobile phone cases using conventional imaging (above). Polarised imaging (below) can therefore be used to help identify scratches and imperfections. Credit: Lucid Vision Labs
Polarised imaging can be used to determine the internal stress, birefringence, surface roughness, film thickness and alloy composition of materials. According to Chang, the edges of objects can also be detected using the technique, as the corners of an object exhibit different polarisation properties than its flat surfaces. This makes polarised imaging a viable tool for object detection and segmentation – an application that has sparked interest in some of Lucid Vision’s customers.
Chang described how the company has also received a lot of queries about whether polarisation can be used to improve existing 3D imaging methods, such as stereo imaging or time of flight. This is indeed the case, according to MIT researchers who, in 2015, demonstrated that by exploiting the properties of polarisation, the resolution of conventional 3D imaging devices such as the Microsoft Kinect – a time-of-flight camera – can be increased by as much as 1,000 times.
At Control 2018 in Stuttgart, the Fraunhofer Institute for Integrated Circuits IIS showed that Lucid Vision’s Phoenix camera could be used to inspect carbon fibres. Polarised imaging is particularly suited to this application because one of the unique properties of carbon fibres is that they polarise incidental unpolarised light parallel to the direction of the fibre.
The Phoenix camera – in combination with Fraunhofer IIS’ own Polka imaging software – can therefore be used to capture the angle of linear polarisation, calculating the orientation and position of carbon fibres in real time and displaying them in false colour. This information is particularly important when producing parts made from carbon fibre reinforced polymers, as their tensile strength is dependent on the direction of the fibres. Manufacturers can therefore use polarised imaging as a quality control method to minimise the safety risks associated with using such parts.
While it’s still too early to say which market sectors will see the greatest uptake of polarised imaging as it becomes more affordable, Chang predicted that the increasing availability of the format – as a result of Sony’s new sensor – will certainly lead to new applications arising that were previously unachievable using conventional imaging methods: ‘There are a lot of academic papers on the possibilities of what polarisation can be used for, but no one has been able to realise these applications because of the previous high costs associated with this technique.’
Flir’s Yeung noted that polarimetry can be used in surveillance for detecting man-made objects that would otherwise be difficult to identify using traditional visible or thermal imagery. ‘Camouflaged vehicles or structures will still reflect polarised light oriented parallel to the surface that stands out clearly in AoLP mode,’ she explained.
Yeung added that by decluttering images through the removal of unwanted glare and reflections, training deep learning systems can be simplified using polarised imaging. This is particularly effective in the high-glare environments encountered by autonomous vehicles, for example.
The convenience of reducing reflections also extends to traffic monitoring applications, where polarisation cameras could be used to see through car windshields in bright sunlight to identify drivers who aren’t wearing seatbelts or who are using mobile phones.
The high resolution and low-read noise of the new IMX250MZR sensor could also enable a wide field of view to be analysed with standard microscopy equipment when studying healthy or diseased tissues in medicine which, according to Yeung, can be differentiated by the polarising properties and optical activity of their biological compounds.
Knowledge is power
Even though the low price of Sony’s sensor will make polarised imaging far more appealing to potential end users, as it stands, the industry’s experience with polarisation is currently limited, according to Chang, which may hinder the technology’s widespread adoption.
‘The biggest potential inhibitor for polarised imaging to take off is going to be knowledge, as this is a relatively new technology in the industrial market,’ he said.
‘Most are familiar with using a polarising filter in front of a camera to reduce glare from objects like metal, which is a very simple application of polarisation. For the more advanced polarised imaging applications, however, knowledge is limited to a niche market, and adoption in the industrial inspection market is at a very early stage. A lot of people, therefore, have to spend time learning how to use the technology to discover its potential.’
For polarised imaging, the configuration of the lighting surrounding the application is important; for example, the position of the light source, the type of light – polarised or unpolarised – and its colour must all be controlled carefully. ‘Whereas with a standard imaging camera this can usually be figured out manually by the customer by experimenting with different options, with polarised imaging they have to learn a lot more about how the properties of light work,’ said Chang.
Companies such as Lucid Vision are therefore producing educational material, such as tech briefs and whitepapers, to help teach industry members about polarised imaging. ‘We also have support teams and application engineers that can answer customer questions and go out to sites to help people understand how they can use this new technology,’ Chang concluded.