Human-AI collaboration in the industrial sector

Artificial intelligence (AI) is serving an increasingly significant role in many workplaces — one that continues to grow as new developments in AI and surrounding technologies become available. This leads us to think about how such technologies can be and should be used to empower humans. In this article, Tomokazu Murakami from the Research & Development Group of Hitachi, Ltd. sat down with Didier Stricker, Head of the Augmented Reality Research Department at the German Research Center for Artificial Intelligence (DFKI) to discuss the use of AI in the workplace, its benefits and challenges, human-AI collaboration and its impact on workers.

‘AI for workers’ today and the future for ‘AI in workplaces’

Didier：AI will enter many domains and help workers. Some of the work we are doing suggests that the combination of augmented reality (AR) and AI is very powerful because it's not just about a machine always doing the same things. It's about supporting the user with systems that help at the right time within the workflow.

Initially, systems were very rigid, but now we need flexible systems that understand when workers need support and provide it in a way that is not just performing one step after another. We can have systems that better sense actions and interact with a person based on the surroundings and, with AI, understand workflows and support users in a totally different way. It also can help address concerns over focus, fatigue, ergonomics and many different areas.

Tomokazu：AI is a powerful tool for industry. There has been rapid growth of the technology, intensive research and significant results. So, now is a good time to apply AI technology to industry. Hitachi has many customers in industry, and we are focusing on developing human worker recognition technologies and other AI technologies.

Currently, AI is applied primarily to things like helping with instructions but in the near future it will be applied to more fields and have more applications, such as efficiency and safety improvements.

Didier：I see three main areas in which AI will be utilized for human workers in the future. First, there is worker health, especially in factories. A lot of people have injuries or are at risk of work injuries. So, health AI can help. There also should be personalization. People are different, so instead of fixed safety guidance, we need flexible models that can target the needs of a specific person. For example, by sensing actions and considering age or injury histories, we can develop better personal profiles and improved time and work management.

Another area is skills and training. Because of AI, simple tasks will be automated, so we need to train people to do more complex tasks. And the last area is human-robot collaboration. We will have robotics systems that need a communication interface so the robots understand what a person is doing and wants to accomplish, so they can work together.

Tomokazu：In the future, we should focus on humans and human-AI interaction. This includes worker safety, security and health, as well as quality of life and quality of work environment. Also, with advanced sensing technology, we will be able to measure human factors more deeply and better understand human abilities and internal states.

In the future, AI and data analytics also may be able to assist workers by enhancing creativity. Currently, AI is good at searching for optimum points in complex data. The AI of the future may be combined with data analytics to go beyond finding optimum points to create new ideas and support human creativity.

Human-AI collaboration in future workplaces

Tomokazu：We should be careful about the way we use AI to support humans. We should care about efficiency but also safety, security, and worker satisfaction and engagement. How we use AI also can lead to ethical issues. AI is very powerful, so it is important to control it.

Speaking of AI ethics, Hitachi has released principles guiding the ethical use of AI in its Social Innovation Business. These include aspects of AI usage such as safety, privacy, fairness, equality and prevention of discrimination, transparency, explainability and accountability, as well as security and compliance.

Didier：I agree — we must find a balance between workers’ autonomy and their having time to learn by doing. If we support someone too much, then learning is reduced. The challenge will be achieving teamwork between AI and workers that empowers workers and supports their skills development. Worker motivation and empowerment should be central.

If we use a lot of sensors and observe someone all the time, a lot of ethical questions arise. First, there is a question of data protection, and there is another regarding ethics. On data protection, in the BIONIC system we developed, we comply with the GDPR, or General Data Protection Regulation, rules in Europe.

The ethics question takes us back to the discussion about how to use systems. Humans should be at the center, supported by the system. We also have to be careful of unintended consequences. Sometimes, we have good intentions and want to support the user in some way, but in the longer term, it has the opposite effect. So, we need to design systems correctly and not reduce the human to a biomechanical system.

Key enabler technologies in the future workplace

Didier：I see two types of technology at work. One will be workplace optical systems. Such systems are configurable and enable us to analyze the scene and understand different actions.

The other system will be body sensor networks. Sensors on the body can improve our understanding of kinematic and other physiological information. The next form of smart devices will be smart clothes. The sensors disappear and aren’t noticeable, but they provide information on demand to support and better understand someone’s current state and working actions.

We work a lot on human-object interaction. We want to understand, for example, how people grasp and manipulate an object. First, using the camera, we capture one hand grasping an object. We reconstruct the kinematics of the hand, including the joints, but also the shapes of the hand and the object to obtain a full 3D reconstruction. We identify, in real time, the contact points of one hand grasping the object. Our goal is to capture the action with very high precision. We can use this to understand workflows, such as the objects being used and how they are grasped during different workflows.

Another use is training with a virtual reality (VR) system. For example, we can create a VR version of a process we want people to learn. Everything is done in VR, but when it comes to manipulation skills, we provide a real tool so a person can actually work with it in the virtual environment.

For the body sensor network, we want to go from kinematic to the “kinetic” to understand the forces exerted on the body or different joints during work. For this, we will use insoles to measure the vertical forces. We can then use a biomechanical simulation system to better understand the forces’ effects on the body.

Other future work pertains to physiological states, looking at eye tracking and sensors to measure pulse and heart rate, gaining information about physiological states.

Tomokazu：Sensor devices will be much improved in the future. We will get better smart clothes and other more sophisticated devices. The sensors we provide to workers should be comfortable and convenient.

Also, a better understanding of internal states may lead to realizing better quality of life and satisfaction. So, we need devices and recognition technology that enable us to use things like heart rate or body temperature, tones of voice, small movements, or where someone is looking so we can better estimate internal states. We also should collect data from many people, tag and annotate the information, and develop good databases.

Smartphones, smart glasses and wearable devices - what next?

Didier：We are very close to having small glasses that provide an overlay in 3D with real-time information. Workers will have a screen with them all the time, on demand, giving them the information they need. These glasses also enable workers to be hands-free.

Another area involves combining sensors and other technologies. Currently, sensors are passive; they measure what the person is doing. But other applications will require combining a body sensor network with, for example, a lightweight exoskeleton for the back and shoulders. From the sensor, we can learn the forces exerted on the body, and then we can compute the best support to provide via the exoskeleton. Sensors also can be combined with exoskeletons or haptic feedback to empower workers so they know when to use additional support or respond to fatigue.

Tomokazu：Exoskeletons are a very interesting possibility. They will not only be very helpful for factory workers who must lift and manipulate heavy loads, but also very useful in the healthcare field, helping move patients safely and smoothly. All of these technologies should be small, easy to use and not annoying. I expect, in the future, that these kinds of devices would be much more developed.

At first, the price for such technologies will be quite high, making them difficult to provide for every worker. But as they begin to be manufactured on a larger scale, ultimately perhaps they can be provided to every worker.

Didier：Currently, we are also doing a lot of work on AI for edge devices. A body sensor network should work for at least eight hours, but the computation is quite heavy. With a full body sensor network, many sensors are sending a lot of data into a small processing unit. A system must integrate this data, perform motion recognition and give feedback. So, this unit must be very energy efficient. We started working on deep learning and AI on each device to address this issue. So, edge AI and power-efficient AI are key technologies for compressing huge networks and making them more efficient for the body sensor network and intelligent objects.

We also are working on a continual learning system. Currently, we collect and annotate data, train our network, and upload it. But everybody is different, so although we collect data from 40 or 50 people, there might be a worker with a musculoskeletal disorder or injury. We need a deep learning system that can adapt to that data on the fly and personalize the system for this person.

Tomokazu：Yes, edge computing is very important. We make models and train them in the cloud, often using significant computing resources. But, for more adaptability or flexibility, it may be good to train models at the edge. And, in the future, learning and also executing directly on the edge will be important.

Didier：Up until now with body sensor networks, we have kinematic information but not metric. For example, we know the physical joint, but we do not know its position in motion. Similarly, when someone is walking, we can capture the motion and approximate the distance walked, but we do not know the person’s position in space. We can achieve good estimates, but they are prone to error because small errors accumulate quickly. We need to be extremely precise for applications like robotics. For example, for a human and a robot to work together, we need absolute positions to exchange tools or carry something.

A camera can be very precise if we do proper image detection, which helps improve measurements, but a camera alone in 3D is not precise enough. However, using one in combination with inertial measurement units (IMUs) and schematic information will be very helpful. We see these, especially in human-robot collaboration, as technologies we need.

Tomokazu：Skill education is an important area where this use of AI will be critical. We collect a lot of important skill information from veteran workers, so it is useful to share that information with novices using AI.

Another area would be to create more general efficiency within businesses. Important feedback areas are task management and status updates on manufacturing lines. For example, we could dynamically evaluate tasks to develop better assignments. That might include factors like daily progress, workers’ preferences, and their previous performance and skills. Providing such information and applying good task management inputs to the total work environment would enable the sharing of this feedback with workers to improve the overall efficiency of manufacturing systems.

The key technical challenges in applying AI for human workers

Didier：One challenge is system efficiency. We face problems when making systems cost-efficient and energy-efficient. Another issue is recycling the systems. With electronics in smart textiles, we must think about their lifecycles and plan for recycling them. We also have to consider the need to change, remove or update the electronics.

Cost is another issue. Currently, good AI systems and devices are expensive. Those costs are going to come down in the next two or three years, but it will remain a challenge.

Another challenge will be adaptation, continual learning and personalization. Personalization in systems should be more robust. For instance, when a person is doing well and doesn’t need support, we need to present appropriate information to enable new learning. Resolving human-AI issues will be important to avoid disillusionment with AI technology.

Tomokazu：The important thing is human-AI collaboration. There are still many technical challenges with edge computing, vision recognition and other technologies, but the most important factor is understanding human-AI collaboration. The key point is not only efficiency but also understanding human motivation and satisfaction, understanding both human and AI factors, and thereby creating good human-AI systems. With such technology, we can change our society for the better.

As AI and supporting technologies become more common and continue to develop, they promise many workplace benefits — for companies and workers alike. However, key issues remain around human-AI collaboration and ensuring the technologies truly support and empower workers. Technological developments will solve many issues, whereas others depend on good planning and system design with a keen focus on human-centric factors. Hitachi is committed to powering good through the ethical use of AI in its Social Innovation initiative. Our focus is on understanding both human and AI factors and creating harmonized human-AI systems which can contribute to social goodness. To find out more, please visit the Industrial AI Blog as we continue to share developments.

Profiles

(As at the time of publication)

Didier STRICKER, Ph.D.

Scientific Director and Head of the Augmented Reality Research Dept., Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI)
Professor of Computer Science, Technische Universität Kaiserslautern

Prof. Dr Stricker studied electrical engineering at the Technical University of Grenoble as well as at the TH Karlsruhe and graduated at both universities. In 2002 he received his doctorate at the Technical University of Darmstadt for: "Computer Vision-based Calibration and Tracking Methods for Augmented Reality Applications".

He works as an expert for various European and national research organizations. His scientific focus is virtual and augmented reality, computer vision, human-computer interaction and user recognition.

MURAKAMI Tomokazu, Ph.D.

Head of the Intelligent Vision Research Department,
Center for Technology Innovation – Artificial Intelligence,
Research & Development Group, Hitachi, Ltd.

Tomokazu Murakami joined the Central Research Laboratory of Hitachi, Ltd. in 1998 after completing his Master of Information and Communication Engineering. Since then, he has been engaged in R&D of image processing, video coding and image recognition techniques. In his current position as the head of the research for intelligent vision, his team is developing new technologies and solutions for video surveillance, image-based inspection system, multi-modal manufacturing support system and so on.

He received his doctoral degree in Information Science and Technology from The University of Tokyo in 2012. He has over 40 patents and has authored 20 technical publications, and is a member of The Institute of Electronic, Information, and Communication Engineers (IEICE), The Institute of Image Information and Television Engineers (ITE), and The Virtual Reality Society of Japan (VRSJ).