The 400G Network Transformation of Data Centers

by Will Chang

5G is not just about higher bandwidth, but rather bringing new a user experiences with innovative services such as virtual reality gaming, augmented reality training, high quality video streaming, autonomous driving, and more. All this will be enabled with a low latency, high bandwidth and connected infrastructure.

Network transformation takes time, but surprisingly, the pandemic has accelerated the process. With a new normal starting to emerge where people live, work and communicate online. Based on a report from Nokia, US internet traffic volumes have normalized at 25% above pre-pandemic levels. Although the data use growth rate has slowed down, the trend remains unchanged even with things opening back up. It is expected that internet traffic will reach 4.2 zettabytes per year by 2022.

As data demand grows and new services are introduced, the data center is also going through various transformations. The first being the data center moving closer to the end users by upgrading existing central offices with CORD (Central Office Re-architecture as Datacenter), which was a concept first introduced by the Open Network Foundation to offer more efficient data processing for service providers and deliver low latency services to end users. This new architecture enables OTT (over-the-top) service providers to offer low latency and high-quality video content by leveraging distributed CDN (Content Delivery Network) to bring the content closer to their subscribers. The reliance on edge computing enabled faster response times for critical operations and allowed them to begin development on new applications such as autonomous driving and smart manufacturing.

Another major transformation for the data center is the migration to a 400GE infrastructure to fulfill the non-stop data growth. According to Omdia, the push for 400GE data center switches is in part fueled by the server's adoption of 25GE/50GE interfaces in order to meet the growing demand of high speed storage access and huge amount of data processing. From the chart below, although the 400GE port's market share was only 0.3% in 2020, the growth was 3 times that compared to 2019 and it is expected to reach 11% of the market by 2025.
 

Projected port speed shipment trends

Broadcom's Trident series offers balance in feature set, performance and programmability. It is optimized for the versatile requirements of enterprise and telco cloud. This leads us to the Trident 4. It’s the latest member of the Trident silicon product family built with 7nm process technology. It offers a 12.7Tbps switching bandwidth and is the first user programmable silicon from Trident series.

In order to deal with the various services and potential applications in the future, new features could be added into the programmable pipelines to enable new packet processing use cases and optimize the switch for new services, which was not included by Broadcom at the current time that the ASIC was designed.

With the shift to a 400GE architecture and server visualization in the datacenter, east-west traffic occupied most of the traffic pattern. A modern datacenter needs to address the oncoming developments by providing an efficient load balance mechanism. The UfiSpace S9300-32D leverages the DLB (Dynamic Load Balancing) support in the Trident 4 to enhance the load distribution by monitoring the link status and re-assigning traffic flow using the hardware engine.
 

Quality of Service becomes more difficult to maintain as data rates grows and traffic increases. So, service providers will need to have real-time visibility and control over the whole network in order to analyze the network status and troubleshoot.

Traditional telemetry relies on the network management system to poll status from switches every few seconds or even minutes. The UfiSpace S9300-32D supports in-band telemetry, which is a powerful tool for network status monitoring by offering packet level telemetry with microsecond granularity. The footprint of each packet inside the data center is logged including where the packet travelled, how long it stayed in each node and which queue it used. This provides a holistic view of the traffic’s status for network optimization.
 

Packet footprint is logged and forwarded to the NMS

Other features such as large scale forwarding databases with flexible allocations that can be tailored based on application, are all designed for a fast-paced environment and accommodating the rapid evolution of data center applications.
 

UfiSpace S9300-32D, 400G data center switch

The UfiSpace S9300-32D is a 400GE switch designed for the future data center. Equipped with Broadcom’s Trident 4 silicon, it offers 32 x 400GE service capacity with a high performance 8-core Intel CPU. It is ideal as a spine switch in the data center to provide high bandwidth traffic aggregation and enables versatile applications ranging from entertainment such as gaming or video streaming to big data processing for artificial intelligence or the Internet of Things.