Greg Blackman reports from the Association for Advancing Automation's latest digital show on the advances being made using AI for industrial inspection
Andrew Ng positioned his phone above the cracked screen of an old mobile phone, took a picture, and an AI model running in the cloud picked out the crack. He did the same with a scratch on the back of the phone.
The results popped up instantaneously, and while the demonstration was a simple one – and Ng an expert in AI, previously leading the Google Brain research team, a former vice president of Baidu, a professor at Stanford University, and now CEO of industrial AI firm Landing AI – the point Ng was making is that a simple proof-of-concept study with just a mobile phone camera is a good place to start experimenting with AI in industrial inspection.
Start small and start quickly was one of his messages in his presentation during the Association for Advancing Automation's Automate Forward digital show last week. He said a common mistake with AI was making a huge investment in hardware without planning for the software, and spending years building an IT infrastructure before even looking at a neural network. Start with small projects and get AI running on one production line before scaling up.
He trained the neural network for his demo with 40 images of cracks and 30 images of scratches. The model was running in the cloud, but he said that all practical deployments Landing AI deals with are edge-based.
The other central message of his talk was to take care of the data. He said a lot of emphasis is put on the machine learning model, but that building the model accounts for only five to ten per cent of the workload. The majority – Ng said around 80 per cent of time invested – is in dealing with data.
He gave an example of a project Landing AI had worked on using an AI model to spot defects in steel. The model achieved 76 per cent accuracy in finding defects; the goal was 90 per cent accuracy. Using Landing AI's analysis tool, the steel producer was able to improve the accuracy of its model by 16 per cent, by identifying inconsistencies in data labelling.
The consistency of how the images are labelled will have a big effect on the accuracy of the model, Ng said. This is particularly important in manufacturing, where images of defects are relatively scarce.
The key to having clean data is to use the expertise of the quality control production staff to label the data, Ng advised. Then, after training the model, a tool like Landing AI's LandingLens can be used to identify any classes of data where the algorithm falls short in terms of accuracy. These images can then be revisited and relabelled to iron out ambiguities in how the defects are labelled.
Ng added that deploying a model in production is really only the halfway point in working with AI. The model needs to be retrained regularly, and it's critical to flow data back for continuous learning. He said that AI models can drift far from reality if left to their own devices, and that a QC inspector is needed to keep the model on track.
Don't expect consumer neural networks to work in manufacturing, Ng said. He also advised that, in terms of camera choice, a basic test is whether a QC inspector can look at a picture and pick out the defect. If the answer is yes, then it's reasonable to assume an AI system could do the same.
Getting a grip on simulation
Automate Forward also heard from Dieter Fox, senior director of robotics research at Nvidia, who spoke about Nvidia's Isaac Sim robotics and AI simulator. Nvidia is using physics simulation through Isaac Sim to assess the stability of grasping points of a robotic gripper. It can randomly generate thousands of grasp points of a 3D object in parallel to see which ones are successful.
This can be done for many objects – Nvidia did this for 8,872 objects from ShapeNet with 2,000 grasps per object based on the Franka Panda gripper, a total of 17.7m grasps to assess gripper stability. The input to the neural network is an observed point cloud and the output is a possible way of grasping the object, determined through physics simulation
The neural network, trained on a database of 17.7m simulated grasps, can then be used on real-world depth data to instruct a robot arm to pick up real objects it has not seen before – the model had a 88 per cent success rate.
Nvidia is working with BMW on a neural network that will learn to pick up boxes of unknown size and from random locations to stack on a pallet. The idea is to minimise the amount of manual design and tuning of such a robot palletiser, because the robot doesn't need specific CAD models of the objects it is grasping.
In simulation, many random scenes can be generated of different size boxes and positions. The observed point cloud is simulated and used to train a neural network to segment the point cloud into different boxes, and to suggest a grasp point for each box. This is then deployed in the real world.
The task of placing the box on a target pallet is simulated too. The system can therefore stack boxes onto pallets without knowing the dimensions of the box beforehand.
The palletising work has only been done in simulation so far, but Fox said the team is close to trialling it in the real world.
He added that there is a lot of work being done by the research community investigating how to reduce the labelling effort for training neural networks, and one way to do this is to train in simulation.
Pent-up demand
Demand for automation couldn't be higher at the moment, after the disruption of last year – Mike Cicco, president and CEO of Fanuc America, described the levels of incoming orders as 'incredible' during Automate Forward, while Patrick McDermott, president North America for B&R Industrial Automation, noted that it's now a logistical and planning challenge to deliver on pent-up demand.
Areas where improvements could be made to increase uptake of automation include things like remote access for equipment, making it easier for robots and other machines to communicate, and, as noted by Christine Boles, VP of the Internet of Things group at Intel, de-risking automation and proving return on investment.
There's also the question of workforce development, which Fanuc is heavily involved in, working with 1,300 education providers – schools, colleges, universities – to develop the skills of young people. Cicco also said that the existing labour force needs an injection of new skills to tackle IoT projects.
Specifically on the automotive sector, one of the hardest hit by disruption caused by the pandemic, Cicco said he is 'bullish' about the state of that industry. Automotive production is cyclical, but Cicco expects it to remain strong over the coming years because there is a shift to new electric vehicles and new concept vehicles.
He said that automotive makers are now not just making combustion engines, but also batteries and electric drivetrains, and that with new production processes comes a need for automation. This also trickles down into the tier community, he noted.
Overall, Cicco added that automation is now propagating to smaller users, and that there is significant growth in new – and smaller – users of automation equipment because the technology has become a lot more accessible.
Next year's Automate show will be held in Detroit, USA from 6 to 9 June 2022.
