Introduction

SoftBank Corp. ("SoftBank") and Keio Research Institute at SFC (“KRIS”) have estabslished the Digital Twin Campus Lab (“DTCL”) at Keio University Shonan Fujisawa Campus (“SFC”), located in Fujisawa City, Kanagawa Prefecture, as a research and development facility for next-generation information infrastructure utilizing advanced technologies such as 5G (5th Generation Mobile Communication System) and 6G (6th Generation Mobile Communication System).

The DTCL is supported by the 5G communication platform and the digital twin platform. This article introduces the roles of each platform and discusses the enhancement of autonomous shuttle bus operations through the utilization of the digital twin platform.

About 5G Communication Platform

To cover the entire campus with a 5G network, four 5G base stations were installed within SFC, establishing a 5G SA (Stand Alone) environment. (Related story: https://www.softbank.jp/en/corp/technology/research/story-event/004/)

MEC (Multi-access Edge Computing), a distinctive feature of 5G SA, is also installed within SFC. By utilizing MEC, it is possible to keep data closed within the campus, ensuring confidentiality. For example, it is also possible to offload AI processing that was previously performed locally to MEC, leveraging its low latency. Additionally, MEC is highly compatible with digital twins that require synchronization with the physical world, as it can only be accessed within the campus.

Furthermore, the 5G core network incorporates Service Based Architecture (SBA). We have developed APIs (enablers) that enable information retrieval and control of the core network from external application functions(AFs). This configuration allows a wide range of students to interact with the 5G core network. For example, by checking the connection status of a particular device or changing the traffic priority, it is possible to prioritize important communication and send it first.

In this way, we are building a customizable 5G communication infrastructure that offers these capabilities.

About Digital Twin Platform

To enable the aggregation of data from various sources, we have built a platform called the Digital Twin Campus Lab platform ("DTCL platform") in a cloud environment.To sense the flow of people and vehicles on the campus, we have installed LiDAR sensors on the rooftops of various buildings within SFC. Six devices cover a wide area to sense buses, vehicles on the roads, movements of students, and people waiting at bus stops.The structural data sensed by these LiDAR sensors is collected on the DTCL platform, and any member who has an access to the DTCL platform can obtain the data.

To obtain traffic signal information on the north side of the campus, we have installed cameras on the rooftops of buildings with a clear line of sight. The captured video data is then converted into traffic signal information on the DTCL platform.

The role of the DTCL platform is to aggregate these sensor data and deliver them in a user-friendly format to authorized users. The video below is an example of mapping the acquired data onto a map, depicting a scene where a bus with students on board arrives.

Efforts to Enhance Autonomous Shuttle Bus Operations

As part of the DTCL activities, KRIS collaborates with Kanagawa Chuo Kotsu Co., Ltd. in research and operation of autonomous shuttle buses. We share sensor information acquired through the digital twin platform with the bus's driving system, aiming to improve the operations of autonomous shuttle buses.

About Autonomous Shuttle Bus

This is a circular route operated within SFC by the School of Nursing and Medical Care at Level 2 autonomous driving (partial automation). The vehicle's self-positioning is estimated using GNSS and LiDAR, and necessary controls such as lane keeping are performed based on high-precision map lane information.

For more details about the operation, please refer to the following URL:
https://www.kri.sfc.keio.ac.jp/ja/wp/wp-content/uploads/2022/05/220509_Kanachu_SFC.pdf

Challenges and Solutions

One of the challenges in autonomous shuttle buses is that the bus's sensors have limited visibility. For example, there are areas outside the range of sensors, such as shadows cast by buildings, and variations in recognition accuracy can occur due to backlighting in camera images. Another challenge is that even under favorable conditions, the detection range of LiDAR and cameras installed on the bus is limited to a few tens of meters.

These limitations in detection range and accuracy pose challenges to the safe operation of autonomous shuttle buses. To achieve more advanced autonomous driving and improve safety, it is necessary to address these challenges.

In this regard, by sharing the sensor data aggregated on the DTCL platform with autonomous buses, it is possible to recognize areas beyond the recognition range of onboard sensors, enabling smooth driving such as gentle acceleration and deceleration in front of traffic signals, as well as safety improvements such as temporary stops based on the prediction of pedestrians’ sudden moves.

Here, we will introduce two use cases related to the enhancement of autonomous shuttle bus operations.

Use case 1: Oncoming Vehicle Detection When Turning Right

During right turns at intersections, we previously had to switch to manual driving as distant oncoming vehicles were not being detected. In this case, by obtaining real-time information about oncoming vehicles from the sensors installed on the rooftops of buildings at the right turn points within SFC, through the DTCL platform, we can now detect oncoming vehicles from distances beyond the range of the autonomous shuttle bus's onboard sensors. As a result, we were able to switch to an operational mode where the bus automatically performs right turns only when there are no oncoming vehicles. In the event that an oncoming vehicle is detected, a warning is displayed on the bus's UI, further enhancing safety.

Displaying Distant Oncoming Vehicle Information When Turning Right

Video of Autonomous Shuttle Bus at Right Turn Point

Use case 2: Comfortable and Safe Bus Operation by Traffic Light Prediction

In an autonomous shuttle bus, information about traffic signals is crucial. For example, knowing when a signal is about to turn red allows for preemptive deceleration, improving passenger comfort and safety. However, there are challenges when detecting signal information using cameras installed on the bus. Issues such as difficulties in detection due to backlighting can arise. Even during manual driving, the transition to a yellow signal near an intersection can impact passenger comfort, requiring deceleration.

In this case, we are also obtaining information about the traffic signal status around SFC from the DTCL platform. Additionally, by continuously collecting signal data, it becomes possible to predict signal cycles based on past information. These cycles can be displayed on the bus's UI, allowing for advance awareness of the signal status and preventing abrupt acceleration or deceleration. This enables improved passenger comfort and safer operation of the bus.

Displaying Traffic Signal Information

Looking Ahead

In the DTCL, we will conduct proofs of concept aimed at enhancing the operation of an autonomous shuttle bus using digital twin information. Going forward, SoftBank and KRIS will continue to promote research and development of next-generation information infrastructure using cutting-edge technologies in the DTCL.

Press release
https://www.softbank.jp/en/corp/news/press/sbkk/2023/20230531_01/