Queen Victoria was just half way through her reign, and Britain was still at the height of the industrial revolution, when James Clerk Maxwell took the first colour photograph in 1861. And 70 years have passed since John Logie Baird gave the first colour TV demonstration in pre-wartime London. Yet, despite the many advances in technology over the last century, it has taken years for manufacturers to wholeheartedly embrace colour inspection on the factory floor.
Not surprisingly, inspecting the colour of a product is essential in food and textile manufacturing, electronics, and the semiconductor industry (see panel on page 21). But it comes at a cost. For monochromatic applications, each pixel measures the intensity of light falling at a single point on the image. Colour cameras measure the intensity for three different colours at each separate point – tripling the amount of data that needs to be captured. This complicates every step from the data capture to the computer processing, meaning that greyscale techniques were still the preferred technology, despite the limitations.
‘The imaging industry has pushed the use of inspection on monochrome images,’ says Arnaud Lina, MIL processing group manager, Matrox Imaging. ‘These images are simpler to acquire. The expertise for processing monochrome images is large, but very little expertise or software packages existed to process colour images. The greyscale methods were more widespread, better understood, better performing, cheaper and less computationally expensive. All those factors made grey-scale the attractive solution for the industry.’ It’s only recently that machine vision manufacturers have overcome these hurdles to produce cost-effective and easy-to-use solutions.
In some instances, carefully chosen lighting and a monochromatic camera will suffice without the added expense of full-colour systems. For example, red lighting will highlight red features while leaving green or blue features as dark patches in the image. Simpler greyscale software tools can then be used to make suitable measurements.
‘A lot of people misunderstand and think that if you’re looking at colour, you need colour inspection,’ says Mark Williamson, sales director of Firstsight Vision. ‘It is important if you need to differentiate colours, but you don’t need it for simple measurements.’
Data capture
For the more complicated applications, machine vision manufacturers have been making steady improvements during recent years. One hurdle has been to achieve good resolution for colour cameras at an affordable price, but recent improvements in CCD resolution are providing a solution.
In the past, area scan colour cameras used three separate CCD chips to capture separately the red, green and blue components of each pixel. These would then be recombined into a full-colour image.
A cheaper solution is to use only one chip, in which each pixel is sensitive to only one colour, which alternates across the image. Since each pixel will be surrounded by other pixels measuring different colours, the camera can try to fill in the missing colours by looking at the response of its nearest neighbours. But this reduces the colour resolution of the image to roughly half the number of actual pixels, making the colour definition less precise than a threechip camera with the same number of pixels.
This used to limit the use of single-chip cameras, but new Megapixel chips are making it possible to achieve high colour resolution without needing to dedicate a sensor to each colour. ‘We’re tending to see a shift to single chip colour cameras as they are much cheaper to build,’ says Williamson.
Colour inspection also presents problems for image capture using line scan cameras. Line scan cameras contain a row of many thousands of pixels that scan objects on a moving conveyor to build up an image of each object line by line. Either the single row of pixels has to capture each colour separately, which slows down the cameras’ performance, or the cameras use three rows of pixels, which is costly.
Basler’s colour Sprint camera.
While tri-linear scan cameras are still popular for high-end applications, some manufacturers are now producing bi-linear cameras, where one row of pixels captures just one colour, and the other row contains alternating pixels that capture the two other primary colours.
‘The colour performance is not as good as three-chip CCD cameras,’ says Xing-Fei He, senior product manager of line scan cameras at Dalsa. ‘But it’s good enough to recognise the individual boxes, for example, when sorting recycling. It’s very good at these applications.’
Since the conveyors often move very fast, the volume of captured data is enormous. ‘Getting data from the camera to the grabber has always been an issue,’ says Williamson. ‘It’s only with PCI express that we’ve been able to handle this volume of data.’
However, according to Williamson, since the advent of high-speed digital interfaces such as Gigabit Ethernet and FireWire, the transfer of data is no longer an issue for area scan cameras.
Once the data has been captured, it needs to be analysed by a computer, and this, too, is complicated by the extra volume of data. ‘For smart cameras in particular it’s a big overhead,’ says Williamson. ‘You need three times the processing power, and it can be a bottleneck.’ Constant improvements in silicon technology should ease this problem.
Colour analysis
FPGA technology could also help to ease the problem, by providing very quick preprocessing before the data has left the camera or frame grabber. Euresys’ Grablink Quickpack ColorScan frame grabber includes an FPGA with many features to ease colour inspection applications.
For example, as an object moves underneath a line scan camera, each row of pixels will see a feature on the object at a slightly different time, and a computer must then piece together the pixels’ different colour readings into an RG image. Euresys’ solution helps ease the computer’s task by performing this ‘scan-delay compensation’ on the frame grabber using an FPGA, which is much more efficient at processing than the computer’s CPU. It also corrects distortions and balances the different colours.
Even following the pre-processing of data, the greater volume means that analysis algorithms still need careful consideration. ‘The algorithms are more complex than traditional processes,’ says Dalsa’s He.
Firstsight Vision’s Williamson agrees: ‘You are effectively working with three times the data, so it is much more complicated. The key is to make the interface simple and hide the complexity from the user.’
He says that Dalsa’s IPD inspect software tool is a good example of this. Using this tool, the user defines in which three colours they are most interested. The tool then creates three sets of monochromatic images mapping the intensity of each of these colours.
A normal colour sensor would only provide a binary image (black where the colour is valid, and white where it is not), but it wouldn’t show the intensity. iNspect gives a map of all of the different shades of that colour, says Williamson. This allows users to take advantage of typical monochromatic tools, such as edge finding or blob analysis, on full-colour images.
Machine vision software libraries can also ease the task of colour inspection, by providing ready-written code for many common tasks. Euresys’ Open eVision suite includes the EasyColor library for colour analysis. These include algorithms that can translate RGB colour data into a description of the colour’s intensity, hue or saturation, which are important parameters for some applications. It can also help with tasks such as segmenting an image according to colour, correcting colour balance, and colour filtering.
Colour analysis can be used to count different objects, such as these sweets. Courtesy of Matrox Imaging.
Many of the current software offerings aim to improve the ease of use of the systems, allowing users to benefit from colour inspection without extensive machine vision expertise. ‘Matrox has developed software tools that are integrated into the Matrox Imaging Library (MIL), a general-purpose imaging and vision library. The software tools include colour-matching tools, colourdistance tools, and colour projection’ says Matrox Imaging’s Arnaud Lina.
‘The integration of colour tools inside a general purpose imaging and vision library allows a developer to add colour image acquisition and processing to an imaging application using a simple uniform programming interface. It allows users to focus on the integration of the colour inspection hardware and software rather than on developing the software algorithms.’
Matrox’s tools include statistical analyses of colour; colour recognition and learning, and the ability to work in different colour spaces that more closely resemble human perception. Flexibility is the key: ‘We offer various strategies depending on the nature of the problem,’ says Lina.
The future
Other small innovations will also ease the maintenance of colour inspection equipment. Dalsa has recently replaced the colour filters on its sensors with dyes normally used on car lights, making them more durable with time. ‘Users don’t want to see the colours fading within a few months,’ says Dalsa’s He.
Technology rarely stays stationary, and it seems that just as machine vision suppliers seem finally to have cracked colour inspection, they may be moving into new territories that could further confuse the data processing. As if three colours (red, green and blue) were not enough, most manufacturers are now looking into adding fourth and fifth components – UV and nearinfrared wavelengths.
‘Pure colour information will probably be extended to the near-infrared,’ says Joachim Linkemann, vision components product manager at Basler. ‘Most failures such as bruises in fruit and vegetables are mainly visible in NIR. Colour cameras will probably be developed with a NIR pixel in addition to RGB.’ This could use the same pixel technology, but with additional filters to allow the infrared waves to reach the desired pixels.
Xing-Fei He, from Dalsa, agrees that cameras will probably be sensitive to a broader range of wavelengths in the future: ‘Customers will also move to ultraviolet inspection. As electrical components become smaller and smaller, we will need to use shorter wavelengths to detect them. I can see it becoming an important market trend.’
Until then, manufacturers can at least benefit from the many other developments in colour inspection technology that are making it easier to use and more costeffective than ever before. ‘We will see more and more customers moving to colour inspection to distinguish different products,’ predicts He.
Colour analysis tools can be used to separate different features. Image courtesy of Matrox Imaging.
The most obvious applications of colour inspection to the production line would appear to be to aesthetic: to detect faults in label printing, for example, or to ensure that a piece of clothing matches a specific hue. These applications are indeed very important, but often colour can be used to measure features other than beauty.
‘In the food industry, the colour of a product is often an indication of its quality,’ says Arnaud Lina, MIL processing group manager, Matrox Imaging. ‘Colour processing can be used for quality assessment of a single product, such as checking the redness of tomatoes, or of a mixture, such as measuring the proportion of various spices in a package.’
In the textile industry UV inspection can find impurities in fabric that would be invisible to the naked eye.
In the semiconductor and electronics industry, colour can be used to distinguish different components on a circuit board. ‘Even though components aren’t necessarily colour-coded any more, colour is still another aspect that can be used for identification,’ says Xing-Fei He, senior product manager of line scan cameras at Dalsa.
Colour can also be important in medical applications. Some companies are now producing imaging machines for hospitals that can automatically analyse samples from tumours and moles to determine whether a patient has a malignant form of skin cancer. ‘A very experienced doctor would be able to diagnose cancer from a sample, but they need a lot of experience. A machine gives a more reliable result – it wouldn’t have a bad day,’ says Joachim Linkemann, the vision components product manager at Basler. ‘Without colour analysis, the tumours would be very difficult to identify,’ he adds.