As the drive to introduce and expand Internet of Things (IoT) applications gathers pace, a growing number of companies and organisations around the world are considering how evolving digital camera technology can be applied in urban settings, particularly in the so-called smart cities of the future. In what ways could such digital security solutions feed into IoT and smart-city applications, what sort of imaging hardware and software would be needed to make such systems work, and what trends and innovations can we expect in this sector over the coming years?
Managing video flows
Although the concept of the Internet of Things is very broad, and potentially relates to any device that can be connected to and accessed from the Internet, there is a growing recognition that imaging technology could have a valuable role to play in many applications. Jukka Nurminen, Adjunct Professor in the Department of Computer Science at Aalto University in Finland, believes that the potential applications of digital surveillance technology could fall easily into one of two categories: those employing cameras in the possession of a single owner, such as a company, and those combining data from multiple cameras and camera owners, which he believes would provide ‘much more opportunities’. Although many of the possible use cases are already in place – for example, those focused on following the movement of people in shopping centres – Nurminen also believes that video streams captured during these activities could be used for ‘many other solutions’.
‘I think the number of possible use cases is immense and there are lot of opportunities for smart ideas,’ he said.
Professor Andrea Zanella of the Human Inspired Technologies Research Centre at the University of Padova in Italy agreed that surveillance cameras can form an important part of IoT, particularly where video flows are shared through the Internet and consumed by different applications, for example when traffic camera images can be publicly accessed through a webpage. That said, he believes that the full potential of the IoT paradigm is really expressed when IoT nodes can achieve what he described as ‘synergic benefits’ from their respective functionalities.
‘In a smart city setting, for example, surveillance cameras may provide traffic information that can help the smart management of the traffic light systems. In the case of accidents, sensors in the vehicles involved in the event may activate the surrounding surveillance cameras in order to provide a real-time video stream of the scene to rescue teams,’ he said.
‘Similarly, it is conceivable to deploy cheap acoustic sensors that can recognise unusual noises such as gun shots, screams and crashes, and that can communicate with smart rotating cameras to provide a video recording of the area where the sound was revealed,’ he added.
AI-powered applications
Elsewhere, Ionut Mirel, co-founder at Canadian image processing outfit Phi Algorithm Solutions, remarked that the use of digital security camera systems for person tracking is currently widespread. In fact, he even implied that such applications are already dated, with several pioneering companies in the field now focusing their attention on person recognition based on facial features and walking patterns.
‘Traffic surveillance from both fixed cameras, as well as from pedestrian smartphones or vehicles themselves, can also provide improved traffic handling. In addition, surveillance can migrate from dedicated systems to any device with a sensor – optical or any other type – connected to cloud-based apps,’ he said.
Gal Oppenheimer, senior product manager at San Francisco-based company Built.io, which assists enterprises to deploy and integrate a variety of apps across mobile, web and IoT systems, agreed that the ready availability of cheap and high-power digital hardware means that organisations are no longer bound solely to the sometimes costly solutions provided by major players in the industry. One interesting example he cites is Acusense, an AI-powered application programming interface (API) tool that enables users to upload video data and generate rich information, before using it to ‘make actionable decisions without significant monitoring needs’. Another innovative example has recently been unveiled by embedded vision company Movidius, which has teamed up with Chinese IoT outfit Hikvision to create a range of smart cameras that employ deep neural network techniques to carry out highly accurate video analytics.
‘We anticipate seeing a reasonably sized proof of concept by 2018,’ Oppenheimer said. ‘In the same way that inexpensive chips and sensors can be easily connected to provide a blueprint of a whole city or downtown, low-cost video cameras will provide the ability to cover entire downtown regions.’
In his view, major advances in analytics and cloud computing will soon allow such video streams to be processed and stored in real-time, an important development because, even with low-cost cloud solutions, he pointed out it is generally too costly for most cities to store video footage for hundreds of cameras 24/7.
‘We anticipate cities storing video for short periods and building rich analytic databases to capture data over time as needed alongside smaller portions of the video footage. For example, you can have software auto-monitor and track your videos. If it suspects a crime occurred, that portion of video and the video from nearby cameras could be auto-stored at the same time to be analysed and reviewed by security or police,’ Oppenheimer added.
Pedestrian tracking technology is already widespread in security imaging applications (Credit: Phi Algorithm Solutions)
Imaging technology
As far as the imaging hardware needed to carry out such applications is concerned, Mirel at Phi Algorithm Solutions observed that the standard requirements are relatively straightforward, and generally include a camera sensor with a resolution of 4K or higher, coupled with high efficiency video coding (HEVC) capabilities, autofocus and low light compensation. He also noted that AI capabilities have now become the industry norm for both mobile camera and fixed camera scenarios.
However, although sourcing and installation of such kit is a fairly simple undertaking, Mirel stressed the interconnection and analysis of data streams from the millions of sensors required for large-scale IoT and smart city applications calls for ‘tremendous bandwidth and stable communication channels’, either via city-wide unified wi-fi systems or mobile data networks – or through ‘special beam-forming architectures, allowing for seamless data sharing’.
‘Centralising the information for analysis and dispatch also requires tremendous server and computing capacity,’ he said.
Although Nurminen agreed existing camera hardware is ‘probably already good enough’, he argued that software systems still need to be improved to make it easy to detect the relevant content from the videos at high enough speeds, especially given the fact that images of large numbers of people are often captured.
‘Application programming interfaces to make it possible to combine and include third-party innovations on top of the video streams would also be needed, as well as the appropriate business models,’ he said.
Zanella also pointed to a recent tendency to equip digital security cameras with advanced signal-processing functionalities, largely in an effort to adhere to the increasingly popular edge computing paradigm that seeks to move complexity to the edge of networks in order to reduce the delay and the traffic that needs to be handled by the transport network. For example, cameras may automatically perform face recognition functions by extracting specific features from captured images and checking their correspondence with those stored in a database.
Another trend is to interconnect multiple cameras through a data network and control them in a co-ordinated manner. In such a way, Zanella explained that rotating cameras can, for example, be co-ordinated to follow a subject that moves across the coverage area of different cameras, or be primed to focus on a specific direction in order to offer different points of view of the same area.
‘Cameras should be able to self-determine which scenes are of interest and need to be reported to a control centre. This is essential, since without automatic scene detection algorithms, the deployment of a large number of surveillance cameras will just result in a flood of images that cannot be processed in real time by human operators,’ he said.
‘However, the automatic detection of scenes of interest is a formidable task, considering that the concept of interest is per se quite fuzzy. In general, we can expect that the automatic classification of the scenes requires advanced image processing algorithms to discriminate among different objects in the scene and machine learning algorithms to assess the interest of a scene,’ Zanella added.
Future trends
Looking ahead, Nurminen predicted that security will remain an important market application for digital camera hardware and software, with other potential applications including the dynamic planning of public transportation systems, as well as shopping centre layouts. Oppenheimer thinks that we will continue to see improvements in low-cost imaging lenses, chips, and whole components with lightweight built-in processing abilities, which he believes will be combined with new cloud-based software solutions that can connect directly to the installed devices.
‘I do not anticipate most traditional technologies competing in the new space as there’s likely too much legacy infrastructure and the software is not designed for the proper use cases of the future,’ he said.
Instead, he expects the industry to benefit from the ‘same industry progressions we’re expecting for all technology spaces over the next few years’ – including a proliferation of new cameras providing an ‘unimaginable amount of rich data’.
‘Initially, we anticipate this being used in a silo, not because it should be, but because most cities will be slow to adopt and integrate their various sensors into one smart dashboard. By 2019 we will start seeing cities integrating these into a smart dashboard and using technologies such as IBM Watson to provide predictive analytics,’ Oppenheimer said.
‘For example, cities will be able to use cameras and other sensor data such as traffic patterns and water drainage to anticipate flooded streets or accidents on a rainy day. Using the improvements in security software, they’ll be able to apply this information for health services, civic planning, safety services, and more,’ he added.
Further into the future, Zanella said it is also possible to conceive of teams of mobile cameras carried aloft by flying drones that are capable of self co-ordinating their activities, perhaps for patrolling a wide area like the suburban sprawls surrounding cities or even following events of interest, such as strikes, political rallies or open-air concerts.’
Despite the wide range of potential benefits of using digital security camera technology in smart city and IoT settings, a growing number of observers are also keen to stress the need to manage the potential downsides. In particular, there are now growing calls to address the inevitable privacy concerns emerging as a result of the increasingly large scale of collection, storage and analysis of video data.
Ionut Mirel, co-founder of Phi Algorithm Solutions, stressed that the security of data and privacy are big issues in this field – and are likely to require both legal frameworks and ‘algorithmic advances in cryptography’. Jukka Nurminen, Adjunct Professor at Aalto University in Finland, agreed that privacy issues remain a ‘big concern and a roadblock’ for many interesting security applications and, in helping to mitigate such concerns, he argued that the benefits of emerging applications should be very high to offset lost privacy.
‘The situation here is different from Facebook and other such things, where you as a user agree to give up part of your privacy, while you cannot do that when just walking in the city. Maybe some strong mechanisms to hide the personalities of the people would be needed but, as with most autonomisation mechanisms, they might be possible to break,’ he said.
In Nurminen’s view, one way to solve such privacy issues would be to devise digital camera hardware possessing tamperproof capabilities to make people anonymous in the video streams, a feature he believes would ‘naturally limit applications, but help increase acceptance’.
