Smart Roads Ahead: How Digital Twins Unite Car and Traffic Data for Safety

1．Navigating Safety Challenges in a Landscape of Connected and Non-Connected Vehicles

SoftBank envisions a future where every vehicle is part of a connected network. But until that day comes, we need to deal with the reality of both connected and non-connected vehicles on the roads. That’s where roadside sensors come into play.

Roadside sensors can detect non-connected vehicles, giving us the ability to monitor all vehicles in an area, whether they’re connected or not. However, to provide real-time feedback to the vehicles, they need to be connected.

2. Real-World Use Cases: A Deep Dive

In this case, we verified the prediction of detailed and diverse vehicle behavior and the real-time notification of information to prevent accidents in a situation where connected and non-connected vehicles coexist, using roadside sensors.

Imagine this: a motorcycle is cruising down the highway, and the driver, distracted, makes a sudden lane change. This scenario poses risks to nearby vehicles. The image below illustrates the experimental setup.

Our experiment included a connected motorcycle, a connected car traveling in the second lane behind it, and a non-connected car traveling in front of the motorcycle. The connected vehicles' information was uploaded to the information collaboration platform via mobile networks. Simultaneously, as long as vehicles were present in the roadside sensors' detection zone, the sensors sent data on each vehicle to the same platform.

SoftBank’s Research Institute of Advanced Technology integrated the self-position estimation results of the connected vehicle with the vehicle detection results from the roadside sensors to create a digital twin(*) that removes duplicates. Using the real-time and predictive capabilities of digital twins, we were able to alert drivers to potential risks at just the right moment.

※Digital Twin: A representation of physical objects or situations in virtual space based on collected data from the real world

3. The Tech Behind It All: Digital Twin for Merging Diverse Data

Next, let's break down the digital twin technology used in our case study. This involves integrating real-time vehicle status into the digital twin using data from connected vehicles and roadside sensors. However, because update cycles, resolutions, and formats vary between data sources, we need to harmonize these differences.

In this experiment, when certain information required for risk detection is not directly obtainable from sensors, it must be estimated using other measured values. In this case, the roadside sensors were unable to measure vehicle acceleration directly, necessitating its estimation. While acceleration is the derivative of velocity with respect to time, simply taking numerical differences amplifies noise, resulting in insufficient accuracy.

To achieve this, we used multiple object tracking through sensor fusion based on state estimation methods. Here are the advantages of this approach:

1. Can integrate different observed items among sensors
2. Can integrate data even with different observation timings
3. Can estimate states that cannot be directly observed by sensors

Advantages 1 and 2 are why state estimation methods are used for sensor fusion. This approach helps us absorb the differences between data sources, which was a key challenge in our verification.

The following figure illustrates the digital twinization process through multiple object tracking

By cycling through the four processes shown in the figure, we perform state estimation and vehicle tracking, and update the digital twin in real-time. Let’s break down each process:

◾️ Association

Determines the identity between the measured data and the tracked vehicles.

First, we identify vehicles based on their IDs. For connected vehicles, we refer to the assigned terminal IDs, while for vehicles detected by roadside sensors, we use the vehicle numbers assigned by these sensors. This allows us to determine if the detected vehicle is the same as the one being tracked.

For data where vehicle ID doesn't confirm identity, we search for the nearest pair based on distance. This applies specifically to cases where a connected vehicle being tracked is first detected by a roadside sensor. To avoid including pairs with large distances, we apply a threshold for gating (extraction).

◾️ State update

ses the Kalman filter to update and correct the state of the tracked object with the measured data. The Kalman filter incorporates not only the observed values but also their reliability to adjust the magnitude of the update. In our verification, we set a constant reliability based on the sensor characteristics of each data source.

Additionally, the Kalman filter uses the predicted state of the measured data at its time as the state before the update. Since prediction is possible for any given time, it is possible to update the state even if the update cycles and measurement timings are different between data sources.

◾️ Lifecycle management

Determine whether to start or end tracking of a target. It can start tracking after multiple detections to determine if it is a false alarm, or end tracking after a certain period of time (Time To Live (TTL)) to prevent non-existent objects from being reflected in the digital twin.

In our verification, we set the TTL for tracking termination to 1 second.

◾️ State prediction

Predicts the state of the target object at the time of the measured data or the digital twin's time using the Extended Kalman Filter (EKF). The extended Kalman filter is a technique that applies the Kalman filter, which assumes a linear system, to nonlinear systems by approximating them linearly.

Prediction technology is crucial not only for risk notification based on the digital twin but also for the update process of the digital twin itself. Predictions aren’t 100% accurate, but they’re also not entirely unreliable. The level of uncertainty is quantitatively expressed. By adjusting the state update amount considering the uncertainty, accurate state estimation is possible. Uncertainty varies depending on the prediction period and the state item. For example, the state of a vehicle changes due to factors not modeled, like driver actions, so the uncertainty of the state after 1 second is higher than that after 0.1 seconds. Generally, a vehicle doesn't move sideways significantly in the next moment, so the lateral uncertainty is lower compared to the forward direction. Softbank’s Research Institute of Advanced Technology has developed a sophisticated mathematical model to explain such phenomena (patent application in progress).

4. Experiment Details

The experiment was conducted on a highway under construction from May to July 2024. After confirming on-site conditions and vehicle placement, each system operator verified their readiness before commencing test runs. SoftBank used a platform that links vehicle information with roadside sensor data. Through this platform's logs, we confirmed in real-time that proper connections and data integration were achieved and accurately reproduced in the digital twin. All team members, including those on standby, monitored the experiment in real-time from their respective positions. A successful test run was declared when everyone confirmed no issues. This process was repeated multiple times to gather data on communication and processing delays.

While the experiment faced complex challenges, including real-time troubleshooting and optimizing procedures, our team's cooperation and experience from previous experiments enabled smooth progress.The above describes the details of this experiment. For footage of the test drives, please refer to the video mentioned earlier.

5. Evaluation through Proof-of-Concept Experiment

The results of the experiment are illustrated in the GIF images below, which depict the positions of vehicles reflected in the digital twin in real-time, along with the corresponding measurement data. The timestamps at the top of the images represent the time in the digital twin, while the time difference between the measurements is shown next to each vehicle. As these time differences indicate, the data from connected vehicles and roadside sensors are updated at different intervals. Since roadside sensor data does not include vehicle width, the digital twin information was used for rendering. This confirmed that our approach could uniquely integrate data with different update cycles.

Additionally, the graph below shows the changes in speed and acceleration for each vehicle, as recreated in the digital twin. The plotted data points represent the measurements utilized in the digital twin at each moment, highlighting the different measurement resolutions from each data source. The red line indicates when a risk notification was triggered, showing that the motorcycle decelerated at that precise moment. This demonstrates that acceleration could be accurately estimated from different types of data in online processing.

Through this digital twin technology, we successfully notified connected cars of surrounding risk information.

6. Summary and Future prospects

This effort contributes to the development of digital twin technology and serves as an important step towards a future society where vehicles are highly connected. Digital twins aim not only to link and manage data but also to use it to provide tangible social benefits, such as reducing traffic accident risks.

SoftBank’s Research Institute of Advanced Technology is dedicated to achieving a safer and more secure transportation society and advanced digital twin technology. We are addressing challenges such as high-precision sensor data and high-speed real-time processing. Furthermore, we are exploring the creation of new business models and contributions to the industry by leveraging digital twin technology, aiming to maximize its potential in expanding intelligence and shared perception. We will continue our research and development efforts in these areas.

Writer：Yakumo Fujimoto、Hiroki Goto