Our reliance on plastic is putting enormous strain on our oceans and rivers. And it’s incredibly hard to get rid of once it’s there – 12 million tonnes of plastic a year enters the marine environment, according to a Eunomia report in 2016.
On Henderson Island, an uninhabited atoll in the South Pacific, plastic has been found from Russia, the USA, Europe, South America, Japan and China. Despite no people, plastic was washed up on the shores of this 14-square-mile island at a rate of 3,500 pieces a day, reported National Geographic in 2017, carried by the South Pacific gyre ocean current.
What’s more worrying is that Henderson is a Unesco World Heritage site due to its biological diversity. Unesco describes it as ‘one of the world’s best remaining examples of an elevated coral atoll ecosystem’.
Cleaning the oceans remains a difficult proposition, but one that groups are working on, notably the Ocean Cleanup project, which caught its first plastic in the Great Pacific Garbage Patch last year. More than a ‘patch’, it is the largest accumulation of ocean plastic in the world. It is between Hawaii and California and covers 1.6 million square kilometres, roughly three times the size of France, where it’s estimated that 1.8 trillion pieces of plastic, weighing about 80,000 tonnes are churning about.
The Ocean Cleanup team has built a floating structure that creates an artificial shoreline. By applying drag so that the parachute-shaped structure moves slower than the current, plastic washes up and accumulates over time. The floating system can catch pieces of rubbish millimetres in size up to large items like abandoned fishing nets. The team’s models have suggested that a full-scale rollout could catch half of the plastic in the patch in five years.
Hyperspectral imaging, especially working in the shortwave infrared (SWIR), has an important role to play in reducing plastic pollution, thanks to its ability to distinguish between different types of plastic. The technology is being investigated as a potential way to identify plastics in natural ecosystems, while also showing promise for use in recycling plants to sort plastics. A lot of emphasis is put on preventing plastics from entering rivers and oceans in the first place, through better product and packaging design, avoiding single-use plastics, and improved waste handling and recycling, which is where vision technology comes in.
Researchers at Texas A&M University-Corpus Christi are working on image processing algorithms to differentiate between different plastics based on SWIR hyperspectral data. The idea is to eventually use this technology to search for plastic pollution in the environment. Professor Ruby Mehrubeoglu, who is leading the work, said that, when combined with an effective classification algorithm, SWIR hyperspectral imaging has ‘the potential to assist with not only detection and identification of plastics, but also management of plastic pollution in the long run’.
Her team have so far been working in a controlled environment to classify plastics. Different polymers will have different chemical fingerprints when viewed through a SWIR hyperspectral camera, meaning the technology can differentiate between material that would be indistinguishable under visible light. One benefit of hyperspectral imaging is that it combines both spectral and spatial information. It then requires advanced image analytics including classification and segmentation algorithms to make sense of the data, which is what the Texas A&M team are working on.
The researchers are using a SWIR spectral imager from Surface Optics. Most of Surface Optics’ work is for the military; it’s been providing SWIR video-rate spectral imagers for the military for almost 10 years. ‘It’s really valuable for identifying energetic materials or any material that would indicate explosives,’ explained Austin Van Sickle, the firm’s technical sales lead.
‘The imagers can do this for trace amounts outside, in “the theatre”, as it’s called in the military,’ he continued. ‘This is the same process as looking for plastics in the environment.’
Van Sickle said that a hyperspectral camera can easily be attached to a drone to survey a river looking for plastics. However, most cameras tend to be scanning systems, which means they have to be looking straight down and scan across a stretch of land in a very ordered way. There’s nothing wrong with doing this, but it is time consuming. The advantage with using Surface Optics’ spectral cameras, according to Van Sickle, is that they can pan around the environment.
‘For the military, nothing is ordered; they want real-time data,’ he said. Surface Optics provides a staring system, which is a video-rate imager operating at 30fps that can pan around similar to a video camera.
Professor Mehrubeoglu and her team are now trying to fine-tune the classification algorithms for robust segmentation, so that the hyperspectral data can be interpreted reliably.
Demo using Imec's Snapscan camera, based on a line scan sensor that is scanned on an internal translation stage. Credit: Imec
One of the challenges with using SWIR hyperspectral imaging to seek plastic in rivers is that water is opaque under SWIR light. Nevertheless, there are approaches to finding plastics in and around water using data captured in SWIR wavelengths, Mehrubeoglu said. ‘Plastics on land, near the shore or floating on the water surface, are much easier to detect and identify than those that are small and may be submerged in water,’ she said.
Another challenge is that plastics deteriorate over time, and their physical attributes and spectral signatures also change, Mehrubeoglu continued. ‘Accuracy in plastics characterisation is a challenge when there exists plastic debris of varying state, shape, texture, colour and size. Improving accuracy is an ongoing research goal, particularly on the post-data acquisition side,’ she said.
Mehrubeoglu added that real-time applications for environmental surveying will need better coupling of hardware and software. ‘By finding effective bands that increase contrast among a variety of plastics in the scene, image processing techniques can also be applied and combined with spectral analysis to perform semantic image segmentation and image mapping of plastics,’ she said. ‘Currently, hardware exists to collect useful SWIR hyperspectral data in outdoor settings using natural light. The next steps are improving identification and segmentation algorithms to properly label plastics based on their class or type. Machine learning will be a promising solution for this application.’
Waste sorting
Spotting plastic pollution in the environment is one thing, but hyperspectral imaging also promises to improve plastic sorting in recycling plants. ‘There are a lot of research groups working on feasibility studies for sorting plastics with hyperspectral imaging. Now companies are picking this up and starting to do the application research,’ explained Wouter Charle, hyperspectral imaging expert at Belgian R&D institute Imec.
Imec provides two types of hyperspectral camera: its Snapscan technology has a line scan sensor that is scanned on an internal translation stage behind the lens. The sensor has 100 spectral bands in the SWIR range, or 150 bands in the VNIR range. The high-resolution spatial image makes it a good choice for lab setups and application research.
Imec also offers its Snapshot technology, which is based on a mosaic pattern of 3 x 3 or 4 x 4 filters covering the imager. The camera achieves video-rate spectral imaging at the cost of some of the spatial and spectral resolution. Snapshot VIS, RedNIR and NIR cameras are available with 16 or 25 spectral filters with real-time acquisition; the SWIR version is available with nine filters, with 16 bands coming soon. Snapshot cameras would typically be used for industrial implementation.
‘Plastic sorting might use either a push-broom scanning camera, imaging line by line and using the motion of the conveyor to do the scan,’ explained Charle. ‘The other would be an area scan camera like our Snapshot camera, which behaves similar to a regular machine vision camera.’
He added: ‘We have feedback from industrial partners that, because this is a snapshot camera, it can give a complete field of view quickly, rather than waiting to build up a scan.’ A robot bin picking application, for example, can classify objects in the full field of view and identify a new location for the robot arm to move to. Waste sorting is often done on a conveyor belt with many people around the conveyor picking and removing material. There will be occlusions from material lying on top of items, so you need to reacquire images regularly to automate this process, Charle explained.
Imec's SWIR Snapshot camera is available with nine filters
Imec has a few feasibility studies with industrial partners where it has shown that the snapshot SWIR camera with nine bands gives sufficient information to classify around 20 different types of fabric – polyester, cotton, wool, linen and others. ‘That was enough to have a high success rate and reliability in classification and sorting,’ Charle said. ‘A human would only have a 66 per cent accuracy. With this study, we have shown that with just nine spectral channels in the shortwave infrared – carefully positioned to measure the required spectral features – you can realise this sorting application.’
Sorting all plastics is a bit tougher, according to Charle, because the dye colour of the plastic will have an impact on the success of the classification. Black plastic, from which electronic items are often made, is more difficult to classify than white plastic because it does not reflect a lot of light. Hyperspectral imaging depends on the light reflected from the sample, so dark plastic won’t give a good signal. ‘Hyperspectral imaging needs a lot of light, so that’s still a challenge,’ he said.
‘To be completely independent of dye colour you would need to move to the mid-wave infrared, but this is a different imaging technology and is not price effective at the moment,’ Charle added.
‘There are still challenges, but for instance in fabric sorting we have clear success,’ he said. ‘For bright plastics, then hyperspectral imaging is a no-brainer – we have this as a demo; you can easily recognise the different plastics with a nine-band sensor.’
In terms of lighting for SWIR hyperspectral imaging, Charle said that the best source is still a halogen lamp. There are SWIR LEDs available, but they tend to be only narrow-band versions that don’t cover the SWIR range fully. In the visible spectrum there are wide-band LEDs available, which is good for machine vision and hyperspectral in the visual range. Charle said that there’s still a way to go, however, to provide wide-band SWIR LEDs.
There are other opportunities in recycling, and Imec has worked on paper sorting. Recycling plants pay per kilo for paper or cardboard, but the quality of it depends on its moisture content – too moist and it can’t be recycled effectively, but it will weigh a lot more. Moisture content is usually determined by manual sampling, but suppliers might put moist paper on the bottom of the batch and good paper on the top. ‘There is a clear absorption valley in the SWIR range caused by water, so a SWIR camera can be used to do a quality inspection on the incoming paper and measure the water content,’ explained Charle.
Each of Imec’s SWIR hyperspectral sensors undergo a characterisation stage, where the sensor is tested to see how it responds to light for each nanometre in the SWIR range, 1,000 to 1,700nm. In this way, a spectral sensitivity model is built up for the camera. The camera is also validated once the bandpass filter has been integrated. The true spectral information is captured by the system and given as a digital reading. The characterisation data is used to interpret the digital reading, to convert it back to true irradiance and reflectance spectral information.
‘Our goal is to go as close as possible to the end-user; we don’t want the user to worry about how to go from raw data to good hyperspectral data,’ Charle said. ‘But when the application starts, the customer will need to tune the system.’
Imec has a support team of application experts and can help with feasibility studies.
‘There needs to be a further industrialisation step now to improve uptake of hyperspectral imaging,’ Charle noted. ‘It’s a new technology. We expect that customers will firstly explore the potential of hyperspectral imaging by doing application research, after which they will bring this to OEMs and start to make systems in volume.
‘SWIR imagers are still niche,’ he continued. ‘When there can be a volume application attached to it, we will push to convince sensor manufacturers to go to higher volumes, which will drive prices down and make this technology more available.’
Charle said that eventually Imec wants to integrate SWIR sensors on its CMOS processes, similar to what it has done with its chips on CMOS in the visible and NIR range. ‘This is a long way ahead,’ he said. ‘People have to start using the technology. If we can make it available in a system then these applications will come, and then this market pull will follow.’
