According to the World Health Organisation, around 10 per cent of medical products circulating in low- and middle-income countries are either substandard or falsified.
This means that people are taking medicines that not only fail to treat or prevent disease, but which can also cause serious illness or even death – in addition to being a waste of money for the individuals and health systems purchasing them.
Therefore, as an anti-counterfeiting measure, different regions around the world, including China, South Korea, Saudi Arabia, and most recently Europe – the EU’s Falsified Medicines Directive came into effect in February – have brought in regulations that require every unit in a batch of pharmaceutical products to have its own unique identification number.
This enables a pharmacy at the end of the supply chain to scan a barcode on a product’s packaging and check its identification number against the UK Ministry of Health’s database, to see whether that product is genuine. This not only ensures patient safety, but also makes sure that all pharmaceutical goods being sold in a country are taxed appropriately – governments cannot tax counterfeit products.
Scaling up serialisation
The new regulations have led to a dramatic ramp up of serialisation in the pharmaceutical industry, which Irish vision integrator Crest Solutions says has changed the paradigm of how machine vision is purchased and integrated by pharmaceutical multinationals.
‘When the regulations were first announced, senior management teams of global pharmaceutical firms recognised that if they didn’t take action to become compliant with such regulations before their introduction, then their firm’s products would not be allowed on the market,’ said Conor O’Kelly, responsible for industry research and technology insight at Crest Solutions. ‘These firms, therefore, began looking for solutions that would enable the mass-serialisation of their products on a global scale – with machine vision being one of these, having previously been used for optical character recognition (OCR) for decades.’
While firms could achieve standardisation across their factories in terms of machine vision hardware – they could just purchase all their serialisation cameras from a single OEM – when it came to installing and supporting these systems, they were faced with a patchwork of integrators offering different solutions.
‘This is because traditionally machine vision integrators have tended not to scale – they normally stay in their own country serving local plants, which are the main source of their revenue,’ said O’Kelly. ‘Global pharmaceutical firms were, therefore, looking for machine vision integrators that were willing to scale up and provide a multi-site, multinational, multilingual, standard offering across their different factories in a number of time zones.’
This was a fundamental change in the pharmaceutical industry, according to O’Kelly, as some integrators stepped up to the mark, while others decided not to. Crest Solutions was one such integrator that decided to scale, and as a result saw revenue increase tenfold from 2012 to 2018.
‘Tier 1 pharmaceutical manufacturers, who have sites all over the world, now have a commonality of support across their geographies in a way that they haven’t been able to have previously,’ O’Kelly said.
‘Not only can global engineering directors now receive standardised job files, algorithms and documentation across all their sites, but they are now able to benchmark for the first time too – determining what the good/bad levels of productivity are and use them to assess and improve their sites.’
Responsive recruitment
The increase in serialisation across the pharmaceutical industry has also led to integrators such as Crest Solutions having to hire more employees with specific expertise than they would have had to previously, in order to address the new demands presented by serialisation.
‘Due to the vital role that serialisation now plays in the pharmaceutical industry, if there are hardware issues or a serialisation system has not been set-up correctly for a batch, then this [pharmaceutical] product cannot be released to the market,’ O’Kelly said. ‘This has led to pharmaceutical manufacturers asking us to permanently embed engineers that now remain on-site and provide support to not just the serialisation cameras of a pharmaceutical company, but also the other machine vision cameras involved in their production lines.’
In addition, O’Kelly said, serialisation requires large amounts of data to be handled – for example, a batch of half a million products requires half a million unique identification numbers to be communicated between a company’s printers, cameras and the rest of the healthcare industry. ‘This creates a huge data challenge, and as machine vision integrators we’ve really had to expand our data management capabilities,’ he said. ‘We’re now employing software engineers and data scientists to work alongside our vision and project engineers.’
Crest Solutions installs a range of camera systems for inspecting labels on pharmaceutical products
Having so much data, however, does enable individual products to be tracked throughout the entire pharmaceutical supply chain, meaning that if anything happens that results in certain products having to be recalled, pharmaceutical firms know exactly where each of those products is in the supply chain. This allows them to make sure it’s just the affected products being recalled – rather than having to recall an entire batch, for example.
Time to diversify
While the requirement of serialisation has caused a ramp up in demand for machine vision in the pharmaceutical industry over the past three to four years, according to O’Kelly this demand has been mostly limited to serialisation cameras.
This is because companies’ budgets have been solely directed towards making themselves regulation-compliant, establishing proper serialisation capability in their production lines.
‘For the first time in around three or four years, however, we’re starting to see the machine vision budget be replenished for certain organisations, as serialisation isn’t dominating their in-process quality control spending in the way that it has been for the last few years,’ he said. ‘Engineering managers are now getting back to addressing the kind of topics that they previously knew were there, but that they just didn’t have the budget for in the past few years.
‘So I’d definitely say that custom vision applications for in-process quality control are now coming back.’
Detecting reflections
One topic that has been on the minds of pharmaceutical firms is making the tracking information – such as serial numbers and data matrix codes – on pharmaceutical products more discrete, to avoid confusion when customers read the information on pharmaceutical packaging.
‘The pharmaceutical industry has had this idea for a while, but hasn’t been able to implement it up until now as they haven’t been able to find the right technology to do it,’ said Stéphane Clauss, senior business development manager at Sony Image Sensing Solutions (ISS) Europe.
This could soon change, however, thanks to the introduction of Sony’s IMX250MZR/MYR polarised sensor last year, which comprises a CMOS image sensor equipped with a layer of nanowire micro-polariser arrays above the sensor’s photodiodes. Each array in the layer has four different angled polarisers – 0°, 45°, 90° and 135° – that are placed over one of four pixels in a 2 x 2 block on the sensor. The sensor is more practical and more advanced than earlier polarised imaging solutions, such as a single-directional polarising filter positioned in front of a camera.
Results from Crest Solutions' label inspection system
Sony ISS Europe has incorporated the sensors into a five-megapixel camera, the XCG-CP510, for which, in May, it announced a dedicated software development kit (SDK) designed to cut time for developing polarised imaging applications from 6 to 24 months to around 6 to 12 weeks. With the new camera, pharmaceutical firms are able to track inscriptions on pharmaceutical products the same colour as their packaging, making it less visible to the human eye, but still perceivable using polarised imaging.
‘Polarised technology can be used to discern the difference in polarisation between the light reflected off the ink and the light reflected off the packaging material, which provides a good contrast that enables the inscription to be read with a high accuracy,’ Clauss said. ‘In addition to enabling the tracking and tracing of products, this technique makes the pharmaceutical products more difficult to counterfeit.’
Polarised imaging can also be used to remove unwanted reflections in machine vision applications. ‘On certain types of plastic packaging and blister packs, the reflected light can be quite strong, making it difficult to discern OCR and data matrix codes,’ said Clauss. ‘Taking the example of a blister pack, the plastic shape surrounding the pill is rounded, meaning the light reflected by this plastic part cannot be removed by one linear polarising filter alone – in contrast to a flat surface, a rounded surface polarises reflected light in many different directions.’
The four different angled polarisers of the IMX250MZR/MYR sensor enable a much larger portion of polarised light to be removed from images containing reflections, making them suitable for imaging rounded packaging. The technology can also be used to look through transparent packaging when looking at pills inside, ensuring what is seen by the camera is a pill, not a reflection off the packaging.
According to Clauss, new polarised imaging technology is gaining a lot of traction in pharmaceutical applications. ‘From time to time polarised imaging has previously been used in the pharmaceutical industry. This is why our technology has now been well adopted, because the technology is already well understood by this industry,’ he said. ‘What we have provided is a solution that is much more flexible than the previously used cameras with a fixed polarised filter in front of them, which were only able to remove one type of reflection.’
New technologies, new challenges
Another vision technique that is starting to take hold in the pharmaceutical industry is hyperspectral imaging, which, while having been introduced to the industry three to four years ago, is now being adopted thanks to its lower cost.
‘Hyperspectral imaging enables pharmaceutical companies to analyse the chemical composition of every single product on their production line with real-time data streaming, which prevents them from having to sample products at regular intervals and take them to a lab for analysis,’ said Mahmoud AbdelAziz, founder and CEO of Egyptian vision firm DevisionX. ‘The hyperspectral cameras can also be linked to an alarm system to alert staff when a contaminated or incorrect product is on the production line.’
According to AbdelAziz, despite the technology seeing higher levels of adoption in pharmaceutical applications compared to three or four years ago, there are still two main challenges preventing its full adoption in the industry. In addition to the relatively high cost of hyperspectral imaging compared to standard vision systems, which is still deterring potential end users, there is a lack of chemical data available from early adopters in the pharmaceutical industry, which could be used to build more capable analysis software.
‘We are working on developing hyperspectral imaging software libraries related to pharmaceuticals. However, this still requires collaboration with customers, which will involve further testing and data collection,’ AbdelAziz said.
DevisionX specialises in the application of deep learning, a technology which it says is also starting to make its way into the pharmaceutical industry.
Sony's five-megapixel polarisation camera has uses in pharmaceutical inspection applications
‘Deep learning is becoming more advanced and easier to implement,’ said AbdelAziz. ‘It enables the development of flexible vision systems that can be adapted quickly to the inspection task. Models are trained using multiple datasets that can increase the accuracy of inspection processes.’
He explained that deep learning models can be designed to inspect different types of pharmaceutical product. However, while these models are quick to develop and implement, similar to hyperspectral imaging they depend on the right data being provided by customers, who AbdelAziz believes need to be more collaborative if deep learning is to become properly established in pharmaceutical inspection applications.
DevisionX is also involved with many applications where Arabic letters on pharmaceutical packaging have to be read. However, achieving this with vision technology is currently proving challenging.
While English optical character verification (OCV) has been developed over many years, and most companies can easily obtain OCV technology for English and other Latin languages, the same cannot be said for Arabic OCV, as well as other Middle Eastern languages and some Asian languages, confirmed AbdelAziz. ‘This makes it much harder for companies based in these countries to automate their production lines, as they are not able to automatically read and verify the text on the labels of pharmaceutical products,’ he said.
In order to develop OCV for these languages, according to AbdelAziz, they simply need to be introduced to established English or Latin language OCV technologies. This would benefit other sectors, as well as pharmaceuticals. ‘There is a gap here that needed to be filled, in addition to regulations that should be applied to it – especially as this would affect the whole pharmaceutical supply chain,’ he said.
‘This can then be applied to the whole packaging and printing industry across different sectors.’
Countering contamination
MVT's software is able to track tablets between multiple images and determine whether they are acceptable for distribution
One of the most dreaded situations that can take place in pharmaceuticals is cross contamination – for example a tablet of a different shape or size becoming mixed in with a batch of tablets currently being packed.
‘Cross contamination in the pharmaceutical industry is just about the worst thing that can possibly occur; entire plants would need to be shut down if this happened,’ confirmed Brian Castelino, managing director at UK firm Machine Vision Technology (MVT). He added that because of its severity, cross contamination can also lead to drastic fines that have the potential to cripple pharmaceutical businesses.
MVT was approached by another UK firm, Pharma Packaging Systems (PPS), after PPS wanted to create an inspection solution for its existing range of tablet counting and packaging machines. The solution would need to check for any broken or partially formed tablets, in addition to any rogue tablets from another batch – preventing cross contamination.
The tablets are fed from a hopper onto vibrating stainless-steel trays that separate them and guide them down a number of channels. As the tablets leave the end of the channels, they pass through high-speed infrared optical sensors, which count them in free fall as they fall into bottles.
The proposed inspection solution involved placing a number of cameras above the final feed tray to inspect the tablets just before they were counted and bottled. The challenge with this, however, was that, because of the vibrations of the trays, the tablets were often touching and moving irregularly, meaning they could be in any possible orientation when presented to the cameras. Another challenge was that, thanks to the bright and reflective finish of the stainless-steel feed trays, the tablets would often reflect in the channels of the tray, meaning the vision system would have to differentiate between the tablet and its reflection.
Tablet tracking
In the solution developed by PPS and MVT, each camera images six channels of tablets, with each tablet being inspected typically ten or more times as it passes from top to bottom through the camera’s field of view. At any time each individual tablet could appear to be either ‘good’ or ‘bad’ depending on the way it is presented to the camera.
‘This was particularly challenging because while in one image a tablet might be stood on its end – and from our system’s perspective have “bad” dimensions – in the next image it might have corrected itself and therefore have “good” dimensions, so we needed to make sure each tablet can be tracked individually through each image,’ said Castelino.
MVT therefore spent more than a year developing sophisticated software that was able to track individual tablets in such a way.
‘As the tablets leave the tray and get counted into a bottle, the vision system outputs the tablet inspection result,’ said Castelino. ‘Once the inspection returns a “good” result for a tablet, it is locked-in as good, even if it then bounces onto an edge and so appears to be “bad” in subsequent inspections.’ The tablet must be deemed ‘good’ in at least one image, otherwise it is rejected.
‘The vision system has to find every tablet in each image, predict where it will be in the next image and match its new position and orientation in the new image to be able to identify it as the same tablet and so individually track it, building up its good/bad history.’
The software is also able to account for the vibratory feeders starting and stopping several times a minute, in addition to the varying feed rates used to maximise the throughput into each bottle.
Since the software’s release, using MVTec Software’s Halcon suite of vision tools, MVT has gradually updated the software with better ways of analysing tablets in pharmaceutical packaging applications, particularly using colour information. This enables even two-toned tablets to be tracked between images using the software. The software is therefore now able to check the quality of tablets, and check for broken or cross-contaminated tablets, by measuring their length, width, area and colour.
The latest machine developed by PPS and MVT is designed to reject individual tablets before they are put into bottles, rather than the entire bottles themselves.
This system can inspect up to 10,000 tablets per minute, but is scalable to suit any application, according to PPS.
