Machine vision is based largely on imaging in black and white. Introducing colour brings benefits, but inspection also becomes a whole lot more difficult – there’s three times the amount of data to process for a start and maintaining colour consistency is also a challenge. How we humans perceive colour is subjective – the shade of a colour will differ depending on who is observing it – and the same problem exists with imaging hardware. Different camera sensors will detect subtly differing RGB values and colour temperature from lighting will vary.
‘The benefit of imaging in colour is that you split the visible spectrum up into three areas enabling you to see defects that you might not have seen with monochrome,’ comments Mark Butler, group manager, product management and marketing at Teledyne Dalsa. Frayed strands of copper on printed circuit boards (PCBs) called ‘whiskers’ can be picked out against the green background of the board with a colour sensor, for instance. ‘The problem with monochrome is that you can image a blue and red checkerboard and if the intensities are the same it would appear completely grey with no distinction between the colours,’ he continues.
PCB inspection has only relatively recently implemented colour imaging, largely because the speed of colour line scan cameras wasn’t fast enough up until the latest generation of cameras.
Where the difficulties lay, however, is with building and maintaining a consistent colour inspection system. Dr Rudolf Krampe, managing director of Isra Surface Vision, notes that the lighting unit must be extremely stable with regards to intensity and colour stability. Even small changes can cause detection of false defects, he says.
‘One big constraint in developing a colour inspection system is the consistency of the setup,’ states Arnaud Lina, processing team manager for analysis tools at Matrox Imaging. ‘One of our future focuses is how to offer system developers software tools to improve colour consistency in their setup.’ Of course, building a consistent colour inspection system in the first place is preferable, but Matrox Imaging will provide software tools to fine-tune colour consistency in the image.
Print inspection is a classic application that needs high-fidelity colour imaging. Teledyne Dalsa provides line scan colour cameras for various applications, including prism-based cameras typically used in print inspection. ‘The advantage of a prism-based camera is that there is a common optical axis, so the red, green and blue will be imaging the same line of the object at the exact same time,’ explains Butler.
Tri-linear and bilinear line scan colour cameras, on the other hand, use three (or two for bilinear cameras) linear arrays fabricated on one die. The arrays are spaced apart from one another so that the red, green and blue are captured simultaneously, but at slightly different points on a moving object. The camera or frame grabber will compensate for the spatial separation, but if there’s too much vibration the fields of view for the different colours won’t match. In print inspection, for example, the paper can flutter up and down, which can cause aberrations in the image. Certain food inspection tasks can also exhibit heavy vibration and would benefit from a prism-based colour camera with a common optical axis.
However, tri-linear sensors have improved over the last 10 years or so – Teledyne Dalsa’s current tri-linear colour line scan cameras provide two-pixel spacing between arrays, which makes them closer to prism-based cameras and more applicable to print and food inspection.
Teledyne Dalsa’s Spyder bilinear line scan colour cameras have one line as green and the other as alternating red and blue. Bilinear cameras require less bandwidth than a tri-linear sensor, but they resign some colour fidelity
Tri-linear and prism-based cameras give accurate colour measurements for red, green and blue, with prism-based cameras providing higher colour fidelity, because of the common optical axis. Bilinear line scan colour cameras provide accurate colour measurements for only two of the three colours, meaning one colour per pixel has to be interpolated when the image is processed. Teledyne Dalsa’s Spyder bilinear line scan colour cameras have one line as green and the other as alternating red and blue. ‘Dual line scan cameras lowered the overall cost compared to tri-linear versions; they require less bandwidth than a tri-linear sensor, but you resign some colour fidelity as you’re always missing one of the colours,’ Butler says. Bilinear cameras that use Bayer filters placed in front of the sensor use even lower bandwidth, but also with lower colour fidelity, as two colours have to be interpolated for every pixel.
Colour matching
Typically, the aim for a colour inspection system is to determine if a measured colour matches a sample colour. ‘A colour line scan camera cannot measure the colour, but can evaluate the colour in comparison with a pre-defined set of colours,’ explains Dr Krampe at Isra Vision. Isra Vision’s ColorScan3D system carries out a comparison within the colour space and is used for applications such as packaging inspection.
In print inspection, the perfect image and the target image are typically aligned so that there is a pixel to pixel value comparison. ‘The key is for the system to reach the same decision as a human when distinguishing between two colours,’ comments Lina of Matrox Imaging. Image processing software calculates the distance between a measured and a sample colour based on the colour space and the distance formula.
A colour space is a means of representing a colour using three basic components, be that RGB, HSI or CIELAB. ‘There are many ways to represent a colour and it might be more convenient to use hue, saturation and intensity [HSI] rather than red, green and blue,’ explains Lina. ‘When, as humans, we speak about colour, it is easier to do so in terms of hue and intensity of the colour rather than the proportions of red, green and blue. This is same on the computational side and some colour spaces are more convenient than others in certain circumstances to represent and to compare colours.’
‘HSI is much more reliable in terms of matching colour under varying illumination conditions,’ states Paul Wilson at Scorpion Vision. ‘It is a more robust model for colour measurements than RGB. An additional benefit of HSI is that it’s much faster to compute than RGB values.’
Scorpion Vision’s colour matching tool identifies HSI values at the level of each pixel to provide an accurate colour measurement. The company also offers a colour matching app based on the full Scorpion Vision software package.
To determine between very different colours only requires a simple colour space and distance formula. However, distinguishing between subtly different colour shades requires a more comprehensive colour space like CIELAB, which is a perceptual colour space, and statistical distances. ‘When the colour shades are very close together, the quality of the setup and good colour space selection are the two important factors in making an accurate classification,’ states Lina.
Colour imaging is used to ascertain ripeness and grade product quality of fruits and vegetables. Image courtesy of Teledyne Dalsa
According to Dr Wolfgang Eckstein, managing director at MVTec, algorithms used for colour classification tend to be fairly basic, mostly two sliders specifying the hue and saturation. For many inspection tasks, the software needed has to be more advanced – organic products like food have a complex distribution of colour shades where the basic colour analysis methods quickly reach their limits.
One advanced colour classification algorithm included in MVTec’s Halcon software is Gaussian Mixture Models (GMM), in which each of the feature spaces is covered by multiple Gaussian distributions. GMM can accurately cover the feature space even if it’s a combination of multiple colours. Multilayer Perceptron (MLP), also called neural networks, and Support Vector Machine (SVM) are other robust colour classifiers.
The disadvantage with more advanced algorithms is that their standard implementation is relatively slow. However, Dr Eckstein says that once the system is trained, its internal state can be converted to lookup tables greatly speeding up the classification. ‘On a standard computer you can process 200-300 images per second, which is faster than colour cameras typically operate, so speed is no longer a constraint,’ he says.
The bandwidth required for colour imaging is higher than monochrome, but a lot of the processing, Dr Eckstein notes, can be undertaken on a graphics card, thereby offloading work from the CPU. GPUs can be used for Bayer encoding to generate the RGB image, colour space transformations, such as RGB to HSI, and even colour classification. ‘What comes back to the CPU is a ready-labelled image indicating the areas of interest,’ he says. ‘It speeds up the processing by taking the load off the CPU. You can’t use GPUs for everything, but colour imaging is one area where it definitely does speed up processing.’
Imaging outside the visible
One of the trends taking place is to break up the spectrum into more bands. ‘People are asking for a monochrome line or an NIR line along with the RGB lines,’ comments Butler. In the food industry, for instance, bruising in fruit and vegetables can be seen clearly under near-infrared illumination around 850nm. Lensing, though, which is complicated as it is with colour imaging, is made even more difficult when imaging in the visible and NIR spectrum and adds to the cost of the overall system.
As the cost of image processing comes down, a lot more system integrators are using colour to enhance an image or to show the operator a defect in colour. ‘We’ve had examples of companies buying both colour and monochrome cameras. They’ll do all the processing and defect detection with the monochrome camera and then take a snapshot of the defect in colour for the operator to view,’ says Butler.
Butler adds that there is still a requirement for higher speed. Teledyne Dalsa will be releasing a high-speed bilinear line scan colour camera soon. Its current tri-linear cameras can reach up to 35kHz line rate, while its bilinear line scan cameras go up to 70kHz.
There is growth in colour imaging in general, according to Lina, but that growth is relatively slow due to the complexities of achieving good colour consistency. In addition, colour images are typically eight bits, much lower detail than human perception and we can discern between colours much more accurately than machine vision.
The reduction in cost will aid the uptake of colour imaging though, and companies will switch to it to gain an advantage over their competitors. It can reach a tipping point, where companies in a particular sector will all convert to colour imaging to remain competitive. This will take time, but the benefits of the additional information gleaned from colour imaging means more and more inspection systems will be employing it in the future.
