Blogs

Smart Roads Ahead: How Digital Twins Unite Car and Traffic Data for Safety

#Connected, #Digital Twin

Introduction

In this article, we’ll dive into how data integration from connected vehicles and roadside sensors helps make our roads safer, based on findings from the press release below.

Press Release (in Japanese): Successful accident risk prediction and notification through data integration of vehicles and traffic infrastructure using cellular V2X

1.Navigating Safety Challenges in a Landscape of Connected and Non-Connected Vehicles

SoftBank envisions a future where every vehicle is part of a connected network. But until that day comes, we need to deal with the reality of both connected and non-connected vehicles on the roads. That’s where roadside sensors come into play.

Roadside sensors can detect non-connected vehicles, giving us the ability to monitor all vehicles in an area, whether they’re connected or not. However, to provide real-time feedback to the vehicles, they need to be connected.

2. Real-World Use Cases: A Deep Dive

In this case, we verified the prediction of detailed and diverse vehicle behavior and the real-time notification of information to prevent accidents in a situation where connected and non-connected vehicles coexist, using roadside sensors.

Imagine this: a motorcycle is cruising down the highway, and the driver, distracted, makes a sudden lane change. This scenario poses risks to nearby vehicles. The image below illustrates the experimental setup.

Smart Roads Ahead: How Digital Twins Unite Car and Traffic Data for Safety | SoftBank Research Institute of Advanced Technology | SoftBank

Our experiment included a connected motorcycle, a connected car traveling in the second lane behind it, and a non-connected car traveling in front of the motorcycle. The connected vehicles' information was uploaded to the information collaboration platform via mobile networks. Simultaneously, as long as vehicles were present in the roadside sensors' detection zone, the sensors sent data on each vehicle to the same platform.

SoftBank’s Research Institute of Advanced Technology integrated the self-position estimation results of the connected vehicle with the vehicle detection results from the roadside sensors to create a digital twin(*) that removes duplicates. Using the real-time and predictive capabilities of digital twins, we were able to alert drivers to potential risks at just the right moment.

※Digital Twin: A representation of physical objects or situations in virtual space based on collected data from the real world

     

Japanese version only

3. The Tech Behind It All: Digital Twin for Merging Diverse Data

Next, let's break down the digital twin technology used in our case study. This involves integrating real-time vehicle status into the digital twin using data from connected vehicles and roadside sensors. However, because update cycles, resolutions, and formats vary between data sources, we need to harmonize these differences.

To achieve this, we used multiple object tracking through sensor fusion based on state estimation methods. Here are the advantages of this approach:

1. Can integrate different observed items among sensors
2. Can integrate data even with different observation timings
3. Can estimate states that cannot be directly observed by sensors

Advantages 1 and 2 are why state estimation methods are used for sensor fusion. This approach helps us absorb the differences between data sources, which was a key challenge in our verification.

Smart Roads Ahead: How Digital Twins Unite Car and Traffic Data for Safety | SoftBank Research Institute of Advanced Technology | SoftBank

The following figure illustrates the digital twinization process through multiple object tracking

Smart Roads Ahead: How Digital Twins Unite Car and Traffic Data for Safety | SoftBank Research Institute of Advanced Technology | SoftBank

By cycling through the four processes shown in the figure, we perform state estimation and vehicle tracking, and update the digital twin in real-time. Let’s break down each process:

◾️ Association

Determines the identity between the measured data and the tracked vehicles.


◾️ State update

ses the Kalman filter to update and correct the state of the tracked object with the measured data. The Kalman filter incorporates not only the observed values but also their reliability to adjust the magnitude of the update. In our verification, we set a constant reliability based on the sensor characteristics of each data source.

Additionally, the Kalman filter uses the predicted state of the measured data at its time as the state before the update. Since prediction is possible for any given time, it is possible to update the state even if the update cycles and measurement timings are different between data sources.


◾️ Lifecycle management

Determine whether to start or end tracking of a target. It can start tracking after multiple detections to determine if it is a false alarm, or end tracking after a certain period of time (Time To Live (TTL)) to prevent non-existent objects from being reflected in the digital twin.

In our verification, we set the TTL for tracking termination to 1 second.


◾️ State prediction

Predicts the state of the target object at the time of the measured data or the digital twin's time using the Extended Kalman Filter (EKF). The extended Kalman filter is a technique that applies the Kalman filter, which assumes a linear system, to nonlinear systems by approximating them linearly.


Prediction technology is crucial not only for risk notification based on the digital twin but also for the update process of the digital twin itself. Predictions aren’t 100% accurate, but they’re also not entirely unreliable. The level of uncertainty is quantitatively expressed. By adjusting the state update amount considering the uncertainty, accurate state estimation is possible. Uncertainty varies depending on the prediction period and the state item. For example, the state of a vehicle changes due to factors not modeled, like driver actions, so the uncertainty of the state after 1 second is higher than that after 0.1 seconds. Generally, a vehicle doesn't move sideways significantly in the next moment, so the lateral uncertainty is lower compared to the forward direction. Softbank’s Research Institute of Advanced Technology has developed a sophisticated mathematical model to explain such phenomena (patent application in progress).

We plan to provide more detailed information, including the verification results of this initiative, in a future update. Stay tuned to our website for the latest news.


4. Summary and Future prospects

This effort contributes to the development of digital twin technology and serves as an important step towards a future society where vehicles are highly connected. Digital twins aim not only to link and manage data but also to use it to provide tangible social benefits, such as reducing traffic accident risks.

SoftBank’s Research Institute of Advanced Technology is dedicated to achieving a safer and more secure transportation society and advanced digital twin technology. We are addressing challenges such as high-precision sensor data and high-speed real-time processing. Furthermore, we are exploring the creation of new business models and contributions to the industry by leveraging digital twin technology, aiming to maximize its potential in expanding intelligence and shared perception. We will continue our research and development efforts in these areas.


Writer:Yakumo FujimotoHiroki Goto