Technologies Underpinning Generative AIResearch & Development

Column : Hitachi’s Research into How the Metaverse, AI, and Robotics Can Assist Frontline Workers

Generative AI and Frontline Workers

The world’s labor problems are a topic of active debate, one that incorporates wellbeing and ethical considerations while also being informed by the visions and policies of different governments. A goal of Society 5.0, for example, is to use artificial intelligence (AI), the Internet of Things (IoT), robotics, and other innovations to provide efficient and flexible working practices where humans and technology co-exist in harmony. Similarly, Industrie 5.0 builds on the Industrie 4.0 concept of autonomous industrial processes in which systems communicate with one another via networks in real time, but includes an additional focus on co-existence and harmony between technology and humans. In either case, the debate addresses the question of how best to use AI, robotics, and other advanced technologies to support human labor in pursuit of a human-centric society.

Against this background, the emergence of generative AI is driving major changes in working practices for office workers. Released in November 2022, ChatGPT^* is improving productivity, having demonstrated a capability not only for office automation (OA) tasks such as drafting documents or performing translations, but also for creative work such as programming. In the work of frontline workers, on the other hand, which is said to constitute 80% of gross domestic product (GDP) and requires domain- or workplace-specific experience and knowledge, the use of generative AI has remained limited due to the serious consequences that could occur should an AI make incorrect judgements. As it pursues its Social Innovation Business, Hitachi recognizes that the resolution of societal challenges will require support for the workers engaged in manufacturing or services who represent the frontline of the workplace.

Hitachi believes that the workers most in need of support are those who support society by putting both brains and brawn to work at the front lines of a variety of different industries¹⁾. Examples include the production and maintenance of rolling stock and assembly workers in manufacturing. Currently, these frontline workers have been placed in a difficult position by global labor shortages. Harsh working conditions are one factor behind this, as is a lack of flexibility in how people work, as exemplified by nighttime work. Other issues are the trend for low wages and the time it takes to pass on the skills needed in these workplaces.

Metaverse and Multimodal AI Technologies Transforming the Workplace

Work is being done to improve the productivity of frontline workers by means of an industrial metaverse that replicates workplaces in cyberspace. In one example, Siemens and NVIDIA are simulating production line modifications prior to installation on a cyberspace replica of an industrial plant, using it for training. Hitachi, meanwhile, is working on an enhanced workplace metaverse that also incorporates AI, using it to overcome issues with human operation and maintenance work²⁾. Another example involving plant installation management tracks relevant documents and data such as work logs by linking them to the corresponding plant asset in cyberspace. The task of constructing the metaverse is simplified by using an AI to determine locations in the physical plant and extract metadata from the stored data. Use of the metaverse as a tool for data collection provides intuitive access to things like work schedules and logs by placing them in the context of the overall plant. Doing so provides new ways of working, enhancing human senses while also fostering communication between departments by enabling people in different physical locations to view the same workplace, assets, or information when talking to one another.

Using AI to transform the data in this metaverse into knowledge should provide better support for human work. There has been growing interest over recent years in AI agents that utilize large language models (LLMs) to assist humans’ work. These agents can use LLMs to generate draft work processes, visit web sites to find the best choice of hotels for a trip, or suggest desirable products on electronic commerce sites. However, while this technology works with the structured information found in places like the web, it is necessary in the case of manufacturing or infrastructure workplaces to first give structure to real-world data, thereby transforming it into a form that an AI can understand. This is why Hitachi is using multimodal AI to convert diverse workplace data into natural language and building AI that can advise or support the work of humans based on an understanding of the workplace. Figure 2 shows one such prototype. When performing pump maintenance, the worker uses a tablet or other device to collect audio and video of the pump. This is uploaded to an AI assistant that identifies the pump concerned from the images and ascertains whether it has any problems from the sounds it makes³⁾. The AI assistant can explain the pump sounds in natural language, identify causes, and propose an action plan with reference to past instances.

In this way, rather than being separate items of data, the different forms of data acquired through the digital transformation (DX) of the workplace are analyzed in an integrated way that also incorporates documents and images, thereby transforming it into knowledge. Not only does this provide intuitive support for the work, but it should also help to enhance worker learning and enhance their senses.

Figure 2—Multimodal AI for Supporting Work of Inexperienced Workers

Towards a Society where Humans Collaborate with AI Robots

In addition to AI support for humans, improving the productivity and safety of frontline workers increasingly involves robots taking over a share of the work. Autonomous mobile robots (AMRs) are being adopted in warehouses and serving robots are being used in bars and restaurants. Recent years have also seen the use of drones for tasks like delivery and infrastructure inspection, and there have been extensive trials of the use of four-armed robots for inspection work in factories, construction sites, and other industrial plants. These have primarily been aimed at reducing on-site labor requirements and workloads. In terms of assisting human growth, however, they can be said to still be in the development phase. Meanwhile, to consider what form work by humans and robots should take, a FY2022 United Nations project entitled “A future with AI” surveyed 254 young people in 35 nations as to whether, in a selection of industries, they would prefer work to be done by humans only, by AI robot, or by both working together⁴⁾.

Of the 10 industries included in the survey (factory work, delivery, surveillance, grading students, housework, driving cars, drug discovery, surgery, sales, and education), the ones that elicited the highest scores for AI robots alone or robots and humans working together were factory work, delivery, and surveillance. In contrast, the industries that received the highest proportion of responses wanting humans only were surgery, sales, and education.

For all the industries other than delivery, robots and humans working together scored more highly than AI robots alone. Past work on collaborative robots by Hitachi has included development of the EMIEW⁵⁾ as a service robot with autonomous mobility and functions for interacting with humans, and a collaborative tool for the repetitive task of making fine position alignment adjustments when installing elevators⁶⁾.

Looking at AI and robots used together, research into humanoid robots has accelerated since the emergence of LLMs. Robots are now better at operating autonomously in scenarios where they are instructed in and understand human language, and they are also making use of virtual environments for reinforcement learning. However, it is recognized that training in a virtual environment has its limits when it comes to use of touch and other forms of sensing. Accordingly, Hitachi is researching deep predictive learning techniques for generating actions in real time based on the actual surroundings⁷⁾. Hitachi is also working to simplify the process of teaching robots to perform human work by combining this with imitation learning⁸⁾ and on other capabilities such as the autonomous performance of complex tasks.

In the future, there is potential for using virtual environments such as the metaverse to enhance spoken and other interaction with humans and for building up workplace know-how in cyberspace and operating remotely at other sites. If use of the human senses in the metaverse can be improved, there is also the prospect of robots performing tasks collaboratively and remotely, including those that bring them into contact with people.

With workplace conditions undergoing rapid changes, the work of frontline workers requires them to process diverse forms of information and to make decisions and act based on experience. While starting out with initiatives for reducing human workloads, efforts to support this work then progress to expanding the scope of skills and career development, as indicated in the roadmap shown in Figure 3. This is done using both AIs, which have an in-depth knowledge of the workplace, and the metaverse, where this knowledge is accumulated. This use of AI and the metaverse as an intermediary between people and robots should promote greater mutual understanding and enable new ways of working. While various laws and regulations apply in workplaces where a high level of safety is needed, Hitachi intends to continue developing technology for improving labor productivity while also contributing to human growth and wellbeing through the progressive reform of work processes.

• * See the list of “Trademarks.”

Figure 3—Roadmap for Use of AI and Robots to Support Frontline Workers

1. System Development and Maintenance Support Using Generative AI

To improve the productivity of large system maintenance and development, Hitachi is developing techniques for using generative AI at every stage of the system integration (SI) process, including project management, early-stage design, implementation, and testing.

One of the challenges has been that the quantity and size of detailed design documents and source code required for the development of large systems tend to exceed the capacity of generative AI to process all at once. This also leads to a loss of accuracy. Accordingly, Hitachi has developed a way to convert large amounts of input information into a form that generative AI can handle. The technique can prevent loss of accuracy when large detailed design documents and source code are used as input to make changes to source code, for example. This helps to improve maintenance and development productivity for large systems.

The technique has been incorporated into a framework for the use and development of generative AI that combines generative AI with the system development knowledge that Hitachi has built up over time. It has also been deployed on customer projects. In the future, Hitachi intends to add functionality and enhance the technique to improve development productivity.

［01］System Development and Maintenance Support Techniques Using Generative AI

2. Software Dependency Visualization Solution

Automotive control systems, which were once primarily hardware-based, are now mainly implemented in software. Meanwhile, the availability of over-the-air (OTA) software updates makes it easy for users to upgrade vehicle functions. However, systems are also becoming more complex due to the increasing number of legal and regulatory requirements that relate to automotive software and functions. As a result, more time is required to gather stakeholders to review and confirm applicable laws, regulations, and system dependencies.

To address this problem, Hitachi has developed a solution that significantly reduces the work involved in reviewing the laws, regulations, and systems affected when software is updated and the associated software dependencies. The solution leverages generative AI to efficiently extract and visualize complex dependencies involved in software update management, using existing data and the Software Bill of Materials (SBOM). This dependency information is then presented visually, facilitating easier analysis.

In the future, Hitachi plans to further deploy this solution to support frequent software updates, not only for software-defined vehicles (SDVs), but also for other products such as medical devices.

［02］Block Diagram of Dependency Visualization Solution for SoftwareDB: database
RAG: retrieval-augmented generation
UI: user interface

• * See the list of “Trademarks.”

3. Federated Learning toward Network-distributed AI

The rapid advances in AI require vast amounts of data for building generative AI and other deep learning-based AI. Federated learning provides a promising solution to the challenges of acquiring such data.

Federated learning is a technique designed to protect privacy while effectively leveraging enterprise data—highly confidential datasets distributed across multiple organizations. One significant challenge in deploying federated learning is that data imbalances between organizations can destabilize the learning process.

Hitachi has addressed this issue by adopting specialized training methods tailored to datasets with distinct properties. It is actively utilizing this approach in deployments of federated learning, including applications involving generative AI, a technology that has attracted attention in recent years. One illustrative example is the work to extract valuable insights by using federated learning for the text documents and other insurance-related data distributed across different branches and organizations.

［03］Diagram of Coupled Learning

4. Use of Generative AI for Faster Response to IT System Faults

IT system operation workloads are increasing. In addition to the ongoing shortage of personnel to staff the departments responsible, systems are also growing in size and complexity. In response, Hitachi has developed two techniques that use generative AI to support fault response, an activity that imposes a particularly heavy workload.

1. Automation of first response to events
   Hitachi has developed a generative AI assistant that can assess whether events that occur in an IT system are serious or not based on operating procedure documents. The assistant uses generative AI to interpret the (natural language) judgement criteria specified in operating procedures and automatically determine whether events fit the criteria. This significantly reduces the amount of work that goes into the initial response to the hundreds of events that occur every day.
2. Support for extracting knowledge from fault tickets Hitachi has developed a technique for generating knowledge (operational know-how) from the response records contained in fault tickets. The technique uses generative AI to extract important procedure and other information such as how to identify causes, the extent of impact, and how to recover from faults. This reduces the amount of effort needed to obtain knowledge and facilitates the efficient accumulation of operational know-how.

［04］Use of Generative AI to Support Response to IT System Faults

5. Automatic 3D Generation by AI Trained on Workplace Design Documents

Three-dimensional (3D) models are used in industry not only for design and maintenance, but also for applications such as the metaverse and digital twins. While the growth of these activities has brought an increase in the number of skilled engineers who can produce such 3D models, the trend toward shrinking workforces around the world is being accompanied by interest in the use of generative AI to improve the efficiency and lower the cost of 3D modelling.

In response, Hitachi has looked to the large portfolios of 3D models from design offices and various other sources to develop a technique for building generative AI models that can be trained on this data. The technique can auto-generate 3D models from a rough sketch provided by the user. By making it easy to produce 3D models from sketches, this frees working designers and engineers from the duplicated effort of drafting similar shapes and allows them to concentrate on testing their ideas and making improvements. Use of this automatic 3D drafting technique offers the potential for greater efficiency and automation at design and engineering offices while also accelerating efforts to create more sustainable and efficient production maintenance processes.

［05］Auto-generation of 3D ModelsViT: vision transformer
SDF: signed distance function

6. Agentic AIs for Hitachi Support 360

Improving the efficiency of customer support is one of the main applications for LLMs. Having the LLM automatically generate responses to customer inquiries based on manuals and other reference sources has the potential to reduce the workload of customer support staff.

In many cases, however, LLMs like ChatGPT^* lack the product information and other specialist knowledge that is needed to respond to such inquiries.

This has led to the use of a technique called RAG that can automatically extract the textual information relevant to an inquiry from manuals and other sources and supply this to the LLM along with the prompt itself. Unfortunately, RAG on its own is not always able to produce correct answers when inquiries are complex, creating the need for a more powerful technique.

It was to this end that Hitachi developed ReAct for Customer Support (CS), a technique that utilizes the LLM-agent-based Reasoning and Acting (ReAct) method but augments it based on problem-solving methods and requirements that are specific to customer support activities. The utility of the technique has been demonstrated by trial use in Hitachi’s own support service, Hitachi Support 360.

It is anticipated that the technique will be applied to other activities in the future, including those that involve responding to more complex inquiries.

• * See the list of “Trademarks.”

［06］How ReAct for CS is UsedNAT: network address translation
NAPT: network address port translation

7. Multimodal Foundation Model with Conversion between Different Types of Data

The shrinking size of the workforce over recent years has led to shortages of experienced workers. To address this problem and improve the efficiency of plant maintenance inspections, Hitachi supplies a solution that uses audio of machine operation and other time-series sensor data for remote monitoring and to provide advance warning of problems.

Previous systems have done no more than issue an alert if the degree of anomaly exceeded a threshold. To improve on this, Hitachi has developed a multimodal foundation model that can provide a textual explanation of detected abnormalities based on the characteristics of machine operation audio and other time-series sensor data. Not only does this provide workers with information that is easy to understand, but it also opens up the possibility of text-based searching of past problem instances, thereby providing appropriate information the next time maintenance work is needed. The foundation model can also generate simulated audio or time-series data based on conditions specified as text. This enables the performance of anomaly detection to be tested by simulating conditions for which actual data is difficult to acquire.

［07］Multimodal Foundation Model Featuring Conversion between Text and Audio or Time-series Data

8. Domain-specific LLMs Training Pipeline

While many companies are making use of LLMs, the lack of industry-, company-, and domain-specific knowledge in generic LLMs means that such knowledge needs to be input to LLMs if they are to be used for specialist tasks. While this has led to RAG being used to retrieve and feed such knowledge into LLMs, RAG can fail for a variety of reasons.

In response, Hitachi has developed a training pipeline for domain-specific LLMs that generates training data from business documents and trains a domain-specific LLM from a base open LLM. As a case study, it built an LLM for JP1, a suite of middleware and software for integrated operations management. The JP1-specific LLM scored 58% on the Certified JP1 Consultant examination, which was significantly better than the base LLM (43%) and an external proprietary LLM (47%). When the JP1-specific LLM was combined with RAG, it became the first LLM to pass the exam with a score of 70%. The technique is incorporated into the construction and operation service for domain-specific LLMs that Hitachi launched in October 2024. Further developments are planned to enable use of a wider range of data modalities and to enable handling of specialist tasks under limited computational and data resources.

［08］Domain-specific LLMs Training Pipeline* An example of training a JP1-specific LLM

9. Use of Reinforcement Learning for Cybersecurity with Generative AI Agents

Cyber-attacks have become increasingly severe over recent years, reaching levels in 2024 that were more than 10 times higher than in 2015. However, while cybersecurity continues to increase in sophistication, it is predicted that the shortage of people who can do the work will make it more difficult than ever to keep the infrastructure of society safe. The research described here involves work on resolving this problem through the development of a technique for using AI to automate the detection and response to cyber-attacks.

The technique works by creating reinforcement learning agents in a virtual IT infrastructure environment and assigning each of them with roles for (1) identifying vulnerabilities and (2) providing a defense so that, by building up experience in the virtual environment, they can automatically learn and search for superior methods. A technique was also developed that incorporates an LLM into the agents to enable use of knowledge for highly efficient method searching. This enables previously unknown vulnerabilities [role (1)] and methods for dealing with them [role (2)] to be identified without human intervention.

［09］Overview of AI Technique for Automating Cyber-attack Detection and ResponseSQL: structured query language
SSH: secure shell

10. Application of Generative AI to Materials, Biotech, and Healthcare Industries

Hitachi, Ltd. has developed a technique for using generative AI in materials development that automatically designs new high-performance material structures^*1. The technique is one of the services available in Hitachi’s materials development solution. With a view to expanding the use of the technique to a wider range of applications, Hitachi has extended its scope beyond low-molecular-weight compounds to also encompass proteins and is utilizing collaborative creation to verify its value.

One example is a generative AI developed for the drug discovery industry, which designs genetic modification patterns to enhance immune cell function, thereby reducing costs and shortening lead times for new cancer therapies. In experimental testing, the AI achieved functional improvements of up to 88%. Meanwhile, a collaborative creation project with academics from the University of Tokyo working in the biomaterials field^*2 tested the use of the technique for the design of biocompatible macromolecules (artificial proteins) with the goal of reducing the use of animal experimentation in performance testing. Hitachi intends to continue with AI development to promote innovation in the data-driven development of materials and to help create revolutionary biomaterials that are good for both people and the planet.

*1

Hitachi News Release, in Japanese.

*2

Hitachi News Release, in Japanese.

［10］Hitachi Generative AI for Automatic Molecular and Protein Design and Future PlansQoL: quality of life

11. Ideation Support System Using Generative AI

The use of generative AI continues to grow, powered by advances in LLMs. One such application is the use of chatbots as a means of developing new ideas. Unfortunately, this approach can fail to deliver the hoped for originality in cases where a high level of distinctiveness is called for, such as when companies are seeking to act in ways that differentiate them from their competitors.

In response, Hitachi has developed an ideation support system using generative AI with the goal of providing an efficient means of producing highly distinctive new concepts. This involves humans and generative AI working in a complementary manner, with the former providing inspiration that the latter then fleshes out into fully formed ideas. The system is based on cross-disciplinary search, a similarity search technique that minimizes dependency on particular domains. It uses a database of existing literature to present words as hints for prompting new thinking by the user. When the user selects a word, the generative AI uses it as the basis for generating draft text expressing a new idea. The system also features a function for collating large numbers of these draft ideas, generating new ideas that have been efficiently honed by a repeated cycle of first throwing out new ideas and then narrowing them down.

The technique will be commercialized as an ideation support service.

［11］Example Use of Ideation Support Service to Generate New Ideas

12. Knowledge Management for Manufacturing Using Generative AI

The process of knowledge management in manufacturing currently faces difficulties with information being siloed and shortages of experienced workers. To address these problems, Hitachi has developed a multi-agent generative AI solution that improves the efficiency of organizational knowledge management.

The externalization process of turning tacit knowledge into formal knowledge involves both extracting the latter from the former by having frontline workers and back-office staff work with a generative AI and by checking it against the documents that contain existing formal knowledge. This enables the knowledge to be formalized in a database format that best suits the type of work concerned. The combination process further extends the formal knowledge of individual agents by integrating the optimized databases with agents for each type of work. The internalization process then uses copiloting with these agents equipped with expanded formal knowledge as a means of efficiently passing the formal knowledge on to young workers engaged in the corresponding work, thereby helping them build up tacit knowledge through ongoing training. The socialization process has humans and agents working together to facilitate the formation of group tacit knowledge, thereby providing a pathway to the next round of formal knowledge acquisition. By encouraging the conversion of individuals’ tacit knowledge into formal knowledge and using generative AI agents to establish the socialization, externalization, combination, and internalization (SECI) model^* as a virtuous circle, use of this technology helps to turn information into a shared resource and enable the efficient training of skilled workers.

*

Ikujiro Nonaka, “The Knowledge-Creating Company” (1996)

［12］Knowledge Management for Manufacturing Using Generative AI* Prepared based on SECI model for knowledge management frameworks

13. Multimodal Technique for Collecting and Using Data for Robots that Use Generative AI for Operation

If workforce shortages are to be overcome, methods are needed that can automate tasks and reduce their labor requirements. However, as the places where frontline workers work and the tasks they perform are diverse and constantly changing, this calls for automation techniques that perform tasks precisely while also taking into account the surrounding conditions. Accordingly, Hitachi has developed a mobile manipulator that can learn workplace tasks and be taught autonomous robot operations, and a technique for using multimodal sensors.

To teach the mobile manipulator robot how to perform a task, the worker stands behind the robot and manipulates its movements while high-quality vision, force, and other multimodal sensor information is acquired from the performance of the task. As this is happening, weightings are automatically assigned to rank the importance of the environment and task sensor information at each instant. These are then used to choose which sensor information to use in real time. This makes it possible to simulate where the worker is directing their attention and how much force they are exerting as they perform the task, thereby improving the task success rate.

The technique will help to resolve labor shortages in the social infrastructure industry by accelerating the automation of workplace tasks and reducing their labor requirements.

［13］Mobile Manipulator Robot and Technique for Switching between Multimodal Sensor InformationRNN: recurrent neural network

14. Use of Generative AI to Explain Vehicle Camera Images

To improve the efficiency of automotive software development, Hitachi has developed a technique for using generative AI to produce text explanations of what is happening around a vehicle from its camera images^*.

Work on the development of software for applications such as advanced driver assistance systems (ADASs) or autonomous driving includes visually reviewing large amounts of video to identify the required scenarios. The accuracy of this work and the amount of time it takes are a problem. In the newly developed technique, a generative AI identifies traffic situations such as highways or intersections and automatically generates explanatory text such as “A pedestrian is walking across a crosswalk.” As these explanations are amenable to natural language search, it is possible to identify the required scenarios from video with a high level of accuracy, and to significantly reduce the time required to do so.

A service that uses this technique was launched in September 2024 and Hitachi is deploying it across applications such as the automotive industry, logistics, and manufacturing.

*

Patent pending. Information provided here about the patent application refers to the situation as of November 27, 2024. Patent and other details may differ from those at the time of writing due to factors such as legal action by third parties to have patents invalidated and the progress of patent application procedures.

［14］Use of Generative AI to Explain Vehicle Camera Images

15. Use of LLM for Automatic Construction of Asset Knowledge

The construction of specialized knowledge, such as asset fault causes, and automation solutions that use this knowledge play an important role in improving the reliability and efficiency of the design, planning, and execution of asset management operations and maintenance. The past practice of failure mode and effect analysis (FMEA) has had problems with the time required to construct asset knowledge and extend coverage. To improve performance, Hitachi has devised a technique for semi-automating the construction process, including the development of associated tools.

The technique uses an LLM to extract a breakdown of the asset’s functions, its failure modes, and other information from both the general knowledge on which the model has already been trained and specialized knowledge from documents on the topic concerned that are provided as input. The information is produced in a form that conforms to a defined asset knowledge ontology. When the technique was applied to an FMEA of aging gas circuit breakers used in substations it output five times as many items of knowledge as when the task was performed manually. It also achieved a recall of 0.66. Recall is a measure of the coverage ratio for valid items in the generated knowledge. By doing so, it is anticipated that the technique will reduce the amount of work required for knowledge construction to one-third what was required previously.

［15］Block Diagram of Automatic Construction of Asset Knowledge

16. Use of Generative AI for Structuring of Product Design Knowledge

Improving the efficiency of product development is crucial to maintaining competitiveness and profitability in the manufacturing industry. Over recent years, model-based systems engineering (MBSE) has been finding a place for itself in international standards as a method for defining products in system engineering terms. A lot of work is also going into its introduction in Japanese manufacturing.

Together with Hitachi Industry & Control Solutions, Ltd., Hitachi, Ltd. has used generative AI to implement the automatic generation of system models from design documents with the aim of improving the efficiency of system modeling, including MBSE. To ensure that this achieves high accuracy at extracting the model elements that the user wants, a technique has been developed for augmenting the generative AI prompt with an ontology that specifies the concepts to be extracted and the links between them, and also instances made up of a small number of practical examples. When tested using publicly available design documents, the amount of time taken to produce the system model was less than one-tenth what was required when performing the work manually and the technique achieved a model extraction accuracy (60% to 90%) that is adequate for practical use. In the future, Hitachi intends to use the technique to support the creation of system models and to help improve product development efficiency.

［16］Auto-generation of System Models by Generative AIXMI: XML metadata interchange