Surgery has evolved tremendously over the past few decades. One particularly interesting development has been the emergence of microsurgery. This development is especially meaningful due to the use of surgical microscopes to magnify the operating field and allow for higher precision. Combined with new types of cameras, they truly “make the invisible visible.”
This approach is bound to revolutionise medical treatment. Imagine surgeons being able to make decisions based on realtime information about tissues’ chemical composition at a molecular level, ‘seeing’ the oxygenation of blood flowing through arteries and vessels, or distinguishing in vivo (in living tissue) between healthy and anomalous tissue, such as tumour cells.
It is a glimpse at a future that is much closer than one might think. One important driver is the development of spectral imaging technology directly compatible with current medical practice.
The idea of using hyperspectral cameras in surgical practice is not new. Many studies on this topic have already been conducted, but several issues have prevented the technology from being integrated into hospitals’ everyday workflows.
Hyperspectral imaging (HSI) solutions’ form factor is one important hurdle. They are typically quite bulky and thus incompatible with an operating theatre’s already crammed sterile field. Another limitation has been the lack of video-rate hyperspectral cameras that can acquire real-time data, which is essential to capturing the motion and dynamics in a surgical scene. Companies, such as my own, needed to respond to this.
Contrary to conventional methods of building a hyperspectral camera, Imec’s approach does not use precision optics between the lens and the image sensor to select and diffract the light. Instead, it leverages thin-film spectral filters deposited directly on top of image sensors. This translates into robust yet compact devices, very similar to standard machine vision chips and cameras. And since they build on the same wafer-level CMOS process used to create microchips, they are mass-manufacturable, cost-efficient, and highly customisable to specific customer and application needs.
Imec has an extensive track record in the domain of HSI research and prototyping – with a portfolio that comes in two flavours, depending on spectral filters’ patterns. A striped pattern on top of each row of pixels enables high-resolution hyperspectral imaging, comparable to classic line-scanning push broom cameras – yet more compact, faster, and easier to use. They make up Imec’s ‘snapscan’ series of hyperspectral sensors.
Figure 1: A variety of Imec hyperspectral sensors, with spectral filters deposited directly on top of image sensors, in a mosaic or striped pattern.
A mosaic pattern on top of a group of 3x3, 4x4, or 5x5 pixels comes with a somewhat lower resolution but enables video-rate spectral imaging for real-time data acquisition – even in motion. Dubbed ‘snapshot,’ the cameras come in several flavours. Each variant covers a specific spectral resolution and range – from the visual (VIS), over the near-infrared (NIR), to the short-wave infrared (SWIR) ranges.
Case study: Using HSI for the in vivo identification of low-grade gliomas
Low-grade gliomas are a diverse group of slow-growing brain tumours that often arise in young, otherwise healthy patients. While typically benign in origin, studies have shown that lowgrade gliomas can expand by 4 to 5mm a year and come with the risk of malignant transformation. Early surgical resection is thus a muchfavoured treatment option – although in vivo detection of low-grade gliomas and retrieving their exact demarcations is extremely hard, even with the aid of surgical microscopes.
Giving surgeons the proper tools to detect these tumours in vivo would make for an important breakthrough. Hyperspectral imaging technology shows potential to do just that – as demonstrated in a recent research project featuring Imec’s snapscan VNIR 150 hyperspectral camera.
Figure 2: Thanks to its small form factor, and its compatibility with standard C-mount optics, Imec’s snapscan can easily be mounted on a standard surgical microscope.
Thanks to its small form factor (10 x 7 x 6.5cm, and weighing 645g), and its compatibility with standard C-mount optics, Imec’s snapscan can easily be mounted on a standard surgical microscope. It makes for a compact set-up that can be incorporated smoothly into hospitals’ stringent clinical workflows, contrary to the bulky systems used in previous studies.
What’s more, the set-up generates accurate clinical data that can be fed into a deep learning neural network. This can convert the data into actionable knowledge, allowing surgeons to discriminate between healthy and anomalous tissue. As such, it is an important first step to accommodating the in vivo detection of intrinsic brain tumours – such as lowgrade gliomas.
While intraoperative use of the set-up is premature, the approach has so far been validated using a clinical dataset of six patients at Belgium’s Leuven University Hospital. Imec aims to further this project by integrating its snapshot technology, which accommodates video-rate spectral imaging. This would allow researchers to explore the realtime detection of low-grade gliomas in surgical practice.
Making it easier to experiment with hyperspectral imaging technology
A growing number of users are investigating how hyperspectral imaging technology can enhance their product and/or service offerings – or how it can be used in (surgical) practice. For some, it is clear from the start which spectral range best suits their needs. For others, however, it is a journey that involves testing several camera options. While these camera options are available, fusing their data into one dataset from which conclusions can be drawn is a proven challenge.
This is why Imec recently expanded its snapshot portfolio with a multi-sensor system that covers both the VIS and RedNIR ranges, complemented with a high-resolution RGB sensor. It is specifically aimed at companies and research groups engaging in HSI application development.
Imec’s new hyperspectral camera comes with three sensors, integrated into a single housing, and equipped with a standard F-mount lens. It will allow partners to flexibly assess the pros and cons of different spectral resolutions and ranges without needing to invest in a myriad of devices or duplicate experiments.
Figure 3: The world’s first VIS & RedNIR spectral camera system, complemented with hi-res RGB imaging, for data acquisition at video rate.
Inside the camera, the light is directed to three channels. Two of them are equipped with an Imec off-the-shelf two-megapixel sensor – covering the VIS and RedNIR spectral ranges. A third channel sports a high-resolution RGB sensor. As such, the device covers 30 bands in the 460 to 870nm range, complemented by a true-colour, five-megapixel image – all working in sync at video rate speed for real-time data acquisition (even of dynamic scenes).
The new camera particularly lends itself to supporting use cases that involve an uncontrollably dynamic scene. Examples include assisted (micro) surgery, environmental monitoring, anomaly detection, automotive vision, precision agriculture, and crop inspection – just to name a few.
So far, the underlying complexity has prevented the industry from building a fully functional multi-sensor hyperspectral camera. Imec believes this is where its expertise is most valuable, exploring and prototyping the required components and helping the industry overcome all related hurdles. Thanks to the system’s flexibility and the software that comes with it, experimenting with hyperspectral imaging technology has never been easier. It allows partners that are still in doubt about which sensor best suits their needs to focus on their core business, that is in application discovery and development.
As a next step, Imec plans to share its learnings and a complete development kit with camera builders and systems companies to support the system’s further commercialisation.
Wouter Charle is the Program Manager of Imec’s spectral imaging on-chip activities. He has a physics and software engineering background and started his career in the domain of 3D machine vision. Charle joined Imec in 2014.
