Toward Early Implementation of Autonomous Driving Services

SoftBank is building "Autonomous Operation Platform" as part of its research and development for implementation of autonomous driving technology into society. (Reference: Our Initiatives to Achieve an Autonomous Driving Society Through the Utilization of an “Autonomous Driving Operation Platform”)

In-Car Duties in an Autonomous Driving Society

In conventional transportation services, drivers are responsible for a variety of in-car duties, including not only "driving" but also "non-driving" duties such as opening and closing doors, checking for safety (e.g., seat belt use), adjusting air conditioning and driving speed, and handling problems such as passengers feeling unwell. When providing autonomous driving services, while various ADS (Autonomous Driving System) manufacturers compete to handle the "driving task," there is not as much progress in adequately considering the "non-driving tasks." In an autonomous driving society, in order to succeed as a service, it is important to be able to provide the same level of attention to detail in "non-driving duties" just like the traditional drivers did.

R&D for the Future of Driverless In-vehicle Space

SoftBank is researching the following two major elements of non-driving operations.
　1. Unmanned service by in-car concierge AI
　2. Automation of passenger abnormality detection (Research aimed at quantitative determination of motion sickness with Tohoku University)

In this article, we will introduce each of these initiatives.

1. Unmanned Service by In-Car Concierge AI using ChatGPT

① What is In-Car Concierge AI?

In-car concierge AI not only makes in-car services unattended, but also aims to provide a more comfortable ride than conventional transportation services by providing voice-activated answers to questions from passengers and voice-activated announcements related to autonomous driving services (e.g., destination weather, notification of approaching destinations, etc.). This not only makes in-car services unmanned, but also aims to provide a more comfortable ride experience than conventional transportation services.

② Development of In-Car Concierge AI

In-car concierge AI must be able to respond flexibly to various questions from passengers. For example, when a passenger wants to change the destination during the ride, the route should be changed by taking into account the vehicle's current location data and traffic conditions, and the route should be reflected in the ADS. Furthermore, it is desirable to be able to respond to a wide variety of inquiries, such as event information and congestion conditions around the destination, which are general inquiries but difficult to answer using rule-based logic. SoftBank has developed logic that can respond to these various types of inquiries.

③ Demonstration Test Drive in Takeshiba Area Using In-Car Concierge AI

In June 2023, a demonstration test drive was conducted in the Takeshiba area (Minato-ku, Tokyo) to verify whether the service can respond flexibly to various types of questions from passengers and whether it can be made unmanned. (Reference: The Future of Remote Monitoring in an Autonomous Vehicles Society)

In the test drive, we confirmed that the system can respond to questions from passengers using a tablet terminal installed in the vehicle, and that it can respond to these questions unattended by linking it to the vehicle and ChatGPT.

Interior of a vehicle during the test drive

Linkage with chatGPT

2. Automation of Passenger Abnormality Detection
（Research aimed at quantitative determination of motion sickness with Tohoku University）

① Purpose of the Research

Without a driver in an autonomous car, it becomes more difficult to understand the situation of passengers. For instance, even when an abnormality occurs in a passenger's body, it may go undetected unless reported by the passenger. In the unfortunate event of a delayed report, there is a risk of the condition worsening. Motion sickness, in particular, is a physical abnormality that is common to people of all ages and genders, so we believe that proactive measures will be necessary as self-driving cars become more widespread. Therefore, SoftBank is collaborating with Tohoku University to research motion sickness detection, which can help anticipate and prevent the risk of exacerbation, as it is believed to contribute to ensuring safety and peace of mind inside the vehicle.

② Details of the Research

Until now, when detecting changes in physical status, sensors have been attached to the human body to measure and identify numerical values, but by using Tohoku University's analysis system, we aim to automatically detect the physical condition based on the passenger's facial expression (feeling sick or vomiting), facial color (pallor), and video pulse wave (autonomic nervous indices, etc.) obtained from the camera images.
SoftBank and Tohoku University are working on a system to identify motion sickness by multiplying other measured values as supplementary information based on values obtained from video images, and are conducting a demonstration experiment.

In addition, in this demonstration experiment, passengers were surveyed before departure and after arrival, and changes in their physical condition were checked and linked to the numerical values, which will be used to develop a system that can more quantitatively identify motion sickness.

③ What We Achieved Through the Research

By referring to the system analysis based on data such as camera images inside the car and the results of a questionnaire on physical changes before and after the ride, it was confirmed that the system can determine whether passengers are experiencing motion sickness symptoms.
In addition, we were able to acquire numerical data ​​for quantifying symptoms from images from cameras and create a mechanism necessary for future automation.

④ Future Prospects

Furthermore, in the future, we will increase the number of verification cycles to obtain more precise quantitative data and establish a mechanism that can be reliably utilized for quantification and automation. Additionally, in our research, we are using various measurement equipment besides cameras. However, we aim to simplify this equipment and create a convenient service that can easily detect motion sickness and other bodily abnormalities using only smartphones in the future.