SoftBank Corp. Open-sources AITRAS Orchestrator
to Expand AI-RAN Ecosystem

Background

In recent years, under the AI-RAN concept, there has been growing interest—particularly among telecommunications service providers—in going beyond traditional RAN deployments to leverage AI capabilities, such as AI inference, within networks to create new value.

At the same time, AI inference platforms typically consist of heterogeneous resources, including GPUs (graphics processing units), CPUs, and various accelerators. It is also becoming increasingly common for these platforms to span multiple sites and multiple clusters. Operational complexity is increased in such environments, as service providers must determine, based on current conditions, which cluster should host a given workload.

To efficiently manage both AI workloads and RAN workloads on a common platform, an orchestrator that can centrally manage multiple clusters and diverse resource types—and make optimal decisions based on the current situation—is essential.

About the Dynamic Scoring Framework

SoftBank is driving the development of the Dynamic Scoring Framework, a core function of its AI-RAN Orchestrator. The Dynamic Scoring Framework enables the collection and evaluation of resource status in each cluster, across multiple clusters. By using the scoring results, the AI-RAN Orchestrator can conduct resource management with optimization.

For example, scoring GPUs along multiple dimensions—such as power consumption and application performance—and selecting resources or changing configurations in line with specific requirements can be realized.

SoftBank has already implemented concrete optimization examples using these evaluation metrics and is conducting validations under conditions that assume real-world operations.

Overview of the initiative

The Dynamic Scoring Framework is implemented as an add-on for Open Cluster Management (OCM). OCM enables the management of multiple Kubernetes clusters, as well as policy control and enforcement through its Policy Framework.

The Dynamic Scoring Framework is a framework that enables efficient and scalable execution of "scoring" based on user-defined evaluation criteria across multi-cluster environments. Using this framework, users can perform scoring based on the criteria they consider important and leverage the results to achieve optimization of infrastructure resources.

Another noteworthy aspect of this framework is that it is aligned with the architecture of the AI-RAN Platform & Infrastructure Orchestrator published by the AI-RAN Alliance. This architecture represents a common understanding derived from discussions within the Alliance's Working Group 2 (WG2), which focuses on AI-and-RAN use cases.

Furthermore, this initiative is being pursued in collaboration with Red Hat, Inc. ("Red Hat"), with a view to integrate and scale across its entire enterprise open source portfolio, including Red Hat OpenShift and Red Hat Advanced Cluster Management for Kubernetes.

AITRAS Orchestrator covers not only "macro-level" resource optimization across multiple clusters, but also covers "micro-level" resource optimization within a Kubernetes cluster. As an example, SoftBank is working in collaboration with Red Hat to integrate the AITRAS Orchestrator with llm-d^*2. Specifically, based on the characteristics of the models being executed and the resource conditions of each cluster, the AITRAS Orchestrator determines the optimal resource configuration for the llm-d environment. Through this approach, SoftBank verifies that model performance is improved in vLLM^*3 and llm-d environments.

[Notes]

1. *2
   An open source framework designed to efficiently orchestrate vLLM runtime engines
2. *3
   An open source execution platform designed to enable fast and efficient AI inference of large language models (LLMs)