The Khronos Group and the EMVA are to explore software standards for embedded vision. Khronos’ Neil Trevett and EMVA’s Chris Yates explain the work
What challenges do embedded vision developers face?
Trevett: The embedded industry is building smart devices and applications that use a variety of cameras and sensors to understand their surroundings. While the diversity and capabilities of sensors are evolving, and there are established hardware interfaces to streamline system integration, there are no widely adopted, open standard, cross-vendor APIs to ease software integration of new sensors. For many companies, the cost and time needed to integrate new cameras and sensors into systems has become a significant bottleneck that is becoming more critical as firms seek to preserve their software investment across diverse platforms and hardware generations.
Yates: The scope of vision technology is increasing, and new devices and sensors are creating significant commercial opportunities. However, while vision technology and components are readily available, embedded systems rely on tight integration to realise the ultimate benefits. Without widely used open standards for control and connectivity, manufacturers must spend additional effort each time a new sensor or processor is designed into a system, slowing development cycles and reducing the time available for creating value-add features. This becomes more apparent as the trend for complex multi-sensor systems accelerate, such as combinations of lidar and vision, or non-visible imaging.
What does the initiative hope to achieve?
Yates: There are two aspects where I hope our joint initiative will be successful: primarily in defining the scope for an embedded camera API, which can then be developed and used independently of any particular industry vertical or sector. This would be a tremendous benefit to the entire vision market and enable rapid product development and innovation, culminating in significant end-user value. An effective open standard would also provide a benefit to sensor suppliers, by eliminating the need for every individual vendor to define and maintain their own API, with associated development and support costs, while time-to-market for new cameras and sensors can be reduced.
Secondly, I hope that the model of cross-industry collaboration stimulates an open discussion and identifies key requirements through information exchange, while opening opportunities for new contacts and opportunities between participants. For the EMVA this partnership with the Khronos Group matches our desire to increase cooperation and standardisation across the vision industry as a route to market growth.
Trevett: Ultimately, the goal is to expand the commercial opportunities for all companies making and using advanced cameras and sensors by removing industry friction slowing their adoption and deployment. Khronos’ interest stems from our work to develop open standards for embedded compute acceleration including vision processing and inferencing, often used to process sensor streams. Machine vision applications often need precise programmatic control over the sensor - for example the lens, iris, exposure and ISP of a camera and sensor assembly - to gather the data needed for downstream processing. If this level of control can only be achieved through proprietary APIs, then integrating a new sensor can cause significant software rework. So, we would like to see an effective, widely adopted camera API that can work well alongside our acceleration APIs.
How will it work?
Trevett: EMVA and Khronos are working to create a safe space for the industry to explore if there is an opportunity and a need to build software API interoperability standards for cameras and sensors. We are not jumping into designing API right away. We are using the Khronos Exploratory Group process to bring together a wide industry quorum to explore whether we can, in fact, generate a consensus on what camera API standards would most benefit the industry. What standardisation activities result from the exploratory group, if any, will depend on the results of this group discussion – there is no predetermined outcome. The exploratory group will start meeting at the end of March and is open to all at no cost – we invite any organisations to join if they are interested.
Yates: Any organisation that is willing to sign a simple NDA can participate in these meetings and share their expertise, whether they are a member of the Khronos Group or the EMVA; and to protect the IP of all participants, the meetings will explicitly exclude any detailed design discussions. Our goal will be to create an agreed scope of work which will outline the standardisation activities that can best address the fragmentation of embedded sensor control APIs. From this starting point, we can work on any actual standards development using established processes and IP frameworks, such as those used by both Khronos and EMVA in their existing standards development activities.
EMVA and Khronos represent complementary groups in the overall vision technology sector. By bringing together as many parties as possible from across multiple industries we will create a much more representative quorum for the exploratory group.
The Embedded Camera API Exploratory Group process
How long will any activities take?
Trevett: An exploratory group typically goes through three phases. Firstly, all members are invited to present on relevant public technologies or projects, pain points that they are experiencing, and requirements they feel are key - so that all participants are aware of the current industry situation. Then the group brainstorms on potential directions to fill any identified standardisation gaps. Then, finally, the group triages any proposals and attempts to generate consensus on standardisation activities that would gain industry participation and produce a scope of work document. This would typically take around six months, maybe less depending on how the discussions go.
Yates: Beyond the creation of an effective statement of work, there is obviously a significant effort to generate a first consistent API specification. We expect this phase to take around 18 months to reach a v1.0 specification, but the actual timelines will of course depend on the scope and complexity of the work. While the overall project may take several years to reach wide-scale deployment in the field, it is important that we achieve a consensus in the scope of work in a reasonable timeframe to crystallise the benefit to industry.
Who would you like to be involved?
Yates: While many sectors already make heavy use of embedded vision, this joint initiative is truly open, and will benefit from participants from all sectors, as well as companies providing the component technologies.
Trevett: The automotive industry is one of the largest users of advanced sensors, and there is an increasing number of smart devices such as drones, industrial robots, security devices that will benefit from advanced sensor APIs. But for the exploratory group, we warmly invite camera, sensor and silicon vendors, and software developers, from any market segment that has an interest in state-of-the-art sensor processing applications.
Why now?
Trevett: A number of companies are asking for a solution to streamline use of cameras and sensors in their software stacks, and we see a growing willingness to consider cross-vendor standards, so we think this may be the right time. A significant number of Khronos members have expressed interest in participating in the exploratory group including: All3D, Almalence, AMD, Apertus, the Augmented Reality for Enterprise Alliance, Arm, Cadence, Codeplay, Collabora, EA, Facebook, Google, Holochip, HP, Huawei, LunarG, Mobica, Nvidia, Oculus, Oppo, Qualcomm, RedHat, Texas Instruments, Ultraleap, and Valve.
Yates: We are also seeing a number of EMVA members and major machine vision companies such as Allied Vision, Adimec, Lucid Labs, Matrix Vision, Basler, Baumer, China Daheng, MVTec, and Stemmer Imaging signing up. We also have interest from significant open-source projects. I think it is important to remember that standards development is a long-term and iterative activity, but by starting this initiative now, we have the best chance to provide a firm foundation for future development at a time when the market is active and innovating, and I would encourage as many companies to participate as possible.
Neil Trevett is president of the Khronos Group and VP of developer ecosystem at Nvidia; Chris Yates is president of EMVA and director of Vision Ventures. For more information and to join the Embedded Camera API Exploratory Group please go to https://www.khronos.org/embedded-camera or https://www.emva.org/standards-technology/embedded-camera-api/
