Toward early implementation of autonomous driving services

SoftBank is building "Autonomous Operation Platform" as part of its research and development for implementation of autonomous driving technology. (Reference: Our Initiatives to Achieve an Autonomous Driving Society Through the Utilization of an “Autonomous Driving Operation Platform”)

While there are high expectations for the practical application of autonomous driving in various ways, such as alleviating driver shortages and reducing traffic accidents, the high cost of operation has been cited as an issue. SoftBank aims to solve this issue with its "Autonomous Operation Platform" and to achieve early implementation of highly sustainable autonomous driving services into society.

The importance of remote monitoring operation in autonomous driving society

Remote monitoring of autonomous vehicles accounts for a particularly large portion of autonomous driving operation costs. When providing Level 4 autonomous driving, the placement of "Designated Autonomous Operation Manager" and "Autonomous Operation Operator (temporary) *1" (referred to as "Autonomous Operation Supervisor" hereafter) is required according to Japan's Road Traffic Act, Road Transport Act, and Motor Truck Transportation Business Act.

While the ADS (autonomous driving system) installed in the vehicle is responsible for controlling the vehicle's driving, such as "running," "turning," and "stopping," the remote monitoring work performed by the autonomous operation supervisor must first recognize problems that occur inside and outside the vehicle and then determine the next action to be taken in response.

Therefore, in an autonomous driving society, the key to success in the autonomous driving business will be how many vehicles can be remotely monitored by a single autonomous operation supervisor.

• *1Legal designation and deployment obligations are under consideration by the Ministry of Land, Infrastructure, Transport and Tourism.

SoftBank's vision of remote monitoring five years from now

SoftBank believes that there are three major cases in which problems occurring inside or outside of a vehicle can be remotely identified.

(1) Receiving a notification from the vehicle side when in a minimum risk state*2.
(2) Receiving a notification from occupants, passengers or third parties in the event of any trouble inside the vehicle.
(3) Detecting issues through another layer when the vehicle alone cannot judge the trouble.

Cases (1) and (2) involve notification of problems by the ADS manufacturer and people inside and outside the vehicle, while case (3) involves the use of AI (artificial intelligence), infrastructure coordination, digital twin, and other technologies to detect problems in complex traffic environments that current automated driving systems cannot accurately detect. This refers to a method of detecting problems at a different layer from the vehicle side. By detecting problems at a different layer from the vehicle side, it can contribute to fail-safe mechanisms in an autonomous driving society.

Therefore, SoftBank is developing a method to detect these problems using AI and to notify necessary information to autonomous operation supervisors.

• *2A state in which a vehicle equipped with autonomous driving functions judges traffic conditions appropriately and independently controls the vehicle to stop at the safest possible location to ensure the safety of passengers and other traffic participants as much as possible.

Development of remote monitoring AI

Remote monitoring AI uses AI to analyze telemetry, video data, ADS cognitive data, and other data sent from the vehicle to recognize problems occurring inside and outside the vehicle in real time and generate the information necessary for the autonomous operation supervisor to make decisions.

In recent autonomous driving demonstrations under complex traffic environments in various areas of Japan, autonomous supervisors have recognized problems by looking at various data, such as video images, etc. However, the use of remote monitoring AI will eliminate the need for constant human monitoring of such data, allowing a single autonomous supervisor to monitor more vehicles. A single autonomous operation supervisor can monitor a larger number of vehicles.

Monitoring Vehicles with Remote Monitoring AI

However, as the number of vehicles monitored increases, so does the volume of problems that arise, and if problems that really need to be addressed immediately are buried, this can lead to major service issues.

What is needed here is task management for an autonomous operation supervisor.

Task management by remote monitoring AI

To solve the above issues, SoftBank has developed a system that automates task management for autonomous operation supervisors.
The key to efficient task management is that the system automatically assigns priorities to tasks that should really be handled by humans, and presents them to autonomous operation supervisors. If the autonomous operation supervisor responds to the tasks in order according to the presented priorities, it will be possible to avoid overlooking problems that need to be addressed immediately.

Presentation of task priorities by "Autonomous Operation Platform"

Demonstration experiment using "Autonomous Operation Platform"

In June 2023, a demonstration experiment was conducted in the Takeshiba area (Minato-ku, Tokyo) using "Autonomous Operation Platform" we developed to reduce the manpower required for remote monitoring of autonomous driving.

In the demonstration experiment, data from 10 vehicles was linked to "Autonomous Operation Platform." 2 of the 10 vehicles were actual vehicles, and the remaining 8 vehicles were vehicles running on the ADS simulator in the Takeshiba digital twin environment.

Remote control room in the demonstration experiment

By analyzing various data sent from the vehicles and using the developed remote monitoring AI to focus on issues that really need to be handled by human operators for task management, it was found that a single operation supervisor could perform remote monitoring of a total of 10 vehicles.

Remote monitoring AI screen for a total of 10 vehicles

Evaluation of the effectiveness and scalability of "Autonomous Operation Platform" with Tokio Marine & Nichido Fire Insurance Co., Ltd.

To evaluate the effectiveness and scalability of "Autonomous Operation Platform," we conducted a joint verification with Tokio Marine & Nichido Fire Insurance Co., Ltd. In the verification, actual professional operators who engage in emergency reporting tasks of automobile accidents were asked to use the platform under the assumption of actual operation, and the platform was evaluated for its ability to respond accurately and efficiently to various problems that may occur inside and outside the vehicle from a distance.

The results of the verification confirmed that "Autonomous Operation Platform" can contribute to solving service issues by minimizing the amount of time vehicles are stopped, as well as by enabling two autonomous operation supervisors to respond to issues arising from several hundred autonomous vehicles.



It is expected that the number of autonomous vehicles will further increase in the future, creating further challenges. We will continue to develop the technology for "Autonomous Operation Platform" toward early implementation of highly sustainable autonomous driving services into society.