Hitachi Group’s Progress on Worksite Augmenting Metaverse for Industry

1. Introduction

The fall in the percentage of people of working age in Japan’s population is predicted to accelerate⁽¹⁾. If social infrastructure is to be maintained appropriately under such conditions, efficiency gains that enable this infrastructure to be managed by fewer people will be crucial. One example of what is needed would be the use of digital technology to boost productivity in the planning, preliminary training, and on-site execution of frontline work. It is with this goal that various companies are working on industrial applications for the metaverse. Hitachi has likewise been working on the research and development of a Worksite Augmenting metaverse for improving work efficiency. Drawing on its knowledge of research and development in artificial intelligence (AI) and extended reality (XR), it has been replicating workplaces in three dimensions (3D) and collecting data in various forms to provide people with an intuitive sense of how things relate to one another. In addition to the Research and Development Group, this work has also involved people from Hitachi’s industrial plant and railway businesses.

This article describes the progress of studies into using this Worksite Augmenting metaverse for managing construction work at plants as well as in rolling stock design and the administration of maintenance information. It also reports on the collaborative creation work being undertaken with Microsoft Corporation to facilitate a rapid roll out of the technology across different industries.

2. Application to Management of Plant Construction Work

2.1 Issues with Reaching Agreement in Construction Work Management

When installing equipment at an industrial plant, the various stakeholders and other interested parties meet to review progress and make changes as needed. This includes the designers, site supervisors and workers, and management. Unfortunately, it is not easy for these people to all be present at the work site on every occasion. Instead, reports are prepared for these stakeholders and time needs to be put aside for presenting them. By making this process of reaching agreement more efficient, the project can proceed in a way that enables decisions to be made more quickly at every step⁽²⁾.

2.2 PoC at Project to Relocate Full-size Mockup of Nuclear Power Plant

To resolve the issues raised above, Hitachi has developed a technique that uses 3D scanning to replicate a plant in a virtual space and rapidly collect its operational data, an AI technique that utilizes this collected data, and easy-to-use, browser-based visualization technology. Figure 1 shows a conceptual impression of a prototype that incorporates this new technology. Using a specially developed smartphone app, the location of workers at the plant can be determined (both indoors and outdoors) and their work records can be location-tagged when saved in the metaverse. Remote access is also available, with a browser being all that is needed to view work progress or the plant data collected by this app.

A proof-of-concept (PoC) project using a prototype of this technology took place over an approximate two-month period from July to August 2023. This involved the relocation of a mockup of a nuclear power plant that was being undertaken jointly by Hitachi, Ltd., Hitachi-GE Nuclear Energy, Ltd., and Hitachi Plant Construction, Ltd. The prototype system was used to hold the daily evening meeting that in the past had always taken place at the worksite. The PoC demonstrated that the system could be utilized to share information from the site with off-site participants and to reach agreement on decisions on the basis of this information, without having to use any special digital equipment such as virtual reality (VR) goggles or a high-spec personal computer (PC). The results confirmed that the system is an effective means of improving productivity, allowing for the timely issuing of drawings and for planning to reflect actual conditions at the site. It also helps to reduce the sort of problems that arise out of misunderstandings between departments such as rework or delays waiting for other work to be completed.

Figure 1—Use of Metaverse for Plants Prototype5W1H: when, where, who, what, why, and how, AI: artificial intelligence
* Construction of the mockup shown in the figure was funded by the Agency for Natural Resources and Energy. The figure shows how the prototype of the metaverse for plants is used. Featuring the convenient collection of plant data with 5W1H (Who, What, Where, When, Whom, and How) information attached and the use of AI to interpret it, this allows agreement to be reached more easily by providing everyone involved, whether present at the site or not, with an intuitive view of this information.

3. Application to Rolling Stock Maintenance

3.1 Issues with Information Sharing in Railway Industry

Railway industry assets such as track and rolling stock are managed by a number of different divisions. Unfortunately, given the different ways in which maintenance is handled and the different considerations that apply to the information needed for each type of work, the amount of time required for sharing or retrieving the information held by these different divisions poses a problem. This means there is scope for making work more productive through more efficient sharing of information between departments, allowing the information to be retrieved and understood more quickly.

3.2 Platform for Facilitating Information Sharing Using Rolling Stock Metaverse

To resolve the issues raised above, Hitachi has developed a prototype “rolling stock metaverse” that supports inter-departmental information sharing. It replicates rolling stock in a virtual space and enables all information associated with maintenance work to be held in this metaverse. Figure 2 shows how this prototype is used. Everyone can access the same metaverse via the cloud, including the rolling stock divisions (design and operational departments), traffic management and other functions, and suppliers. As shown in the center part of the figure, in addition to entering the metaverse as an avatar by means of VR, users can also use other devices such as a PC or tablet. The virtual space contains a railway car that has been replicated based on 3D computer-aided design (CAD) data. It holds location-linked information in the form of voice records made orally or documents such as design drawings, modification records, and inspection reports. One example is the note made during the inspection of an automatic door linking railway cars that is shown on the right of Figure 2. Based on this information, the person arranging for work to be done or the person doing the work can easily reference the information they need, either before or during the task itself. Using the system in this way provides users with a 3D view of design drawings and encourages the sharing of expertise or observations by location-tagging information when it is saved.

Based on this concept, Hitachi developed a prototype of a rolling stock metaverse in partnership with Tobu Railway Co., Ltd. using their Limited Express Train Spacia X^*1. This prototype was exhibited at the Hitachi Social Innovation Forum (HSIF) in September 2023 and at the Combined Exhibition of Advanced Technologies (CEATEC) in October 2023. This provided an opportunity to gather feedback from several hundred attendees, revealing demand for a range of applications, including the facilitation of inter-departmental information sharing and training exercises at workplaces that are difficult to replicate.

*1

Spacia X is a registered trademark of Tobu Railway Co., Ltd. in Japan.

Figure 2—Use of Rolling Stock Metaverse Prototype3D: three dimensionalPeople from different departments can enter the metaverse and view work details or other location-linked information.

4. Co-creation to Expedite Delivery

4.1 Issues with Technology Delivery

Deployment of the Worksite Augmenting metaverse described above requires the acquisition of 3D data from the workplace and the use of multiple IT systems. Unfortunately, many areas of social infrastructure face constraints on investment, making it difficult to bring in any new technologies or systems that involve major changes to existing IT systems. This is one reason why projects sometimes fail to proceed past PoC to actual implementation. This, then, calls for technologies and systems that impose few changes on the workplace.

4.2 Worksite Augmenting Metaverse Based on Microsoft Teams/Azure

To overcome the issues raised above, Hitachi developed a prototype of the Worksite Augmenting metaverse system that uses the popular Microsoft Teams^*2 software as its base. The development was undertaken in partnership with Microsoft Corporation and Microsoft Japan Co., Ltd. over a three-month period using a scrum management framework. Among the factors behind the adoption of this development approach were the Microsoft’s Industry Solutions Engineering (ISE) team that builds solutions with customers and other partners and the availability of the co-innovation/co-engineering methods for accelerating co-creation. Microsoft Corporation and Hitachi agreed to embark on joint development on the basis of this co-engineering approach and went on to develop a metaverse system that used Microsoft Azure^*2 and Teams.

Figure 3 shows how the resulting prototype is used. A tab displaying 3D data is added to a channel set up on Teams. The browser-based metaverse system and user authentication and permissions are then implemented on Azure and linked to the Teams channel. This prevents access by anyone other than channel participants. Meanwhile, plant data is managed in a database on Azure. This data appears at the bottom right of the screen with the same user interface (UI) as in SharePoint^*2. Furthermore, a function has been added to speed up the dissemination of information by generating links to specific plant data in the 3D space and sharing them via chat in Teams.

By doing it this way, companies that already use Teams can adopt the system without the need for major changes to current practices. This should facilitate early adoption of the technology and its seamless use in daily work.

*2

Teams, Azure, and SharePoint are registered trademarks or trademarks of Microsoft Corporation in the USA and other countries.

Figure 3—Prototype of Extended Workplace Metaverse Using Microsoft TeamsThe metaverse system is linked to a Teams channel where it can be accessed by a single click from users’ daily work routine. To speed the sharing of information via Teams chat, users can click plant data they want to share to generate a link.

5. Conclusions

This article has described a Worksite Augmenting metaverse developed jointly with Microsoft Corporation. In the future, Hitachi intends to draw on the results of on-site PoC projects at industrial plants to put this technology to use in a wide range of tasks at nuclear power plants, including skills transfer and workforce development, as well as for the safe and efficient execution of large projects. In the railway industry, meanwhile, Hitachi intends to draw on the FY2023 prototype and feedback from talking to trade show attendees to assess the potential for using a rolling stock metaverse to help address the challenges facing railway operators. The intention is also to continue testing delivery via Microsoft Teams and Azure to enable deployment across a wide range of applications, including industrial plants and the railway industry.

Acknowledgments

The plant mockup described in this article was developed as part of the Subsidy Program for Decommissioning and Contaminated Water Management started in FY2021 (Development of Technologies for In-depth Investigation of Primary Containment Vessel Inside), a project funded by the Agency for Natural Resources and Energy. Similarly, permission was received from Tobu Railway Co., Ltd. to use its Limited Express Train Spacia X rolling stock in the development of the prototype railway car metaverse. The authors would like to take this opportunity to express their deep gratitude to everyone involved.