An eye tracking and gesture recognition system for pilots has been developed by the Swiss institute CSEM.
The computer vision system has been tested by ten pilots in a Lufthansa cockpit simulator, and was built under the EU project Peggasus – short for Pilot Eye Gaze and Gesture tracking for Avionics Systems using Unobtrusive Solutions.
The dashboard-mounted camera can track a pilot’s gaze and recognise their hand gestures in real time inside the cockpit. It gives pilots feedback to improve their situational awareness when executing complicated tasks.
“The Peggasus system addresses the unique challenges encountered in aeronautical context situations, such as extreme variations in light intensity,” said Andrea Dunbar, the consortium coordinator at CSEM. “It improves pilots’ situational awareness, and consequently decision-making, and makes it easier for the flight crew to work with aircraft controls.”
The algorithm pipeline for eye tracking combines analytical and data-driven approaches, operating in real time at 60 frames per second with minimum latency while achieving eye-tracking accuracy of better than 1°.
To enable hand-gesture recognition, a deep neural network model was developed and incorporated into the final system.
The research was coordinated by CSEM and financed by the EU; other consortium members include ETH Zurich, Serma Ingénierie and Swiss International Air Lines. CSEM developed the vision system and machine-learning algorithms for eye tracking and hand gesture recognition; Serma Ingénierie integrated the hardware into a cockpit demonstrator for testing; ETH Zurich designed and carried out the testing in a simulator study; and Swiss International Air Lines provided insight on use cases and ideas for future training applications.
The pilots that tested the vision system rated it better than head-mounted eye-tracking systems in terms of comfort and low distraction.
“Based on many incident and accident analyses, the highest degree of flight safety can be obtained from an optimised human-machine interface,” said Gilad Scherpf, the head of aviation training development at Lufthansa Group. “Any steps towards improving such an interface will be of great value to Europe’s avionics industry. What’s more, these interfaces can improve both flight operations and pilot training. Computer-assisted technology like the one used in Peggasus can be very effective in modern airliners, especially since air-safety levels will need to be maintained or increased as the degree of automation increases.”
