IP Infusion OcNOS Data Center for Meeting IXP Network Design Challenges

Aug 7, 2023

IXP Network Design Considerations

An IXP, or Internet Exchange Point, is a data center network that routes traffic between different participants’ networks. The most prevalent IXP architecture is based on Layer 2 Ethernet networks to which the participating networks connect their routers or switches. Each participant maintains a single connection to the network, which simplifies the routing and management of traffic. VLANs can be used on the Layer 2 network to segment and manage traffic.

Designing an IXP involves several important considerations to ensure its efficiency, scalability, and reliability:

  • Managing a large L2 broadcast domain and scaling VLANs
  • Auto-discovering to enable dynamic and automated network provisioning, configuration, and management
  • Having redundant access links and network paths to ensure high availability
  • Allowing for easy expansion as the traffic and participants grow

Some of the existing architectures used by IXPs, such as VPLS, Spanning Tree and MC-LAG, have inherent limits. These architectures either lack redundancy, auto-discovery, or both. Furthermore, all of them rely on data plane flooding of unknown unicasts for MAC learning, which limits the number of MACs and the scale of the network.

Advancements in networking technologies have led to new architectures and solutions for IXPs. In this blog, we will describe IP Infusion’s EVPN VxLAN based solution for IXP network design.

IP Infusion IXP Solution

IP Infusion’s OcNOS Data Center, an open networking OS for the data center, is a great solution for IXP applications. OcNOS Data Center is a proven EVPN VxLAN with MP-BGP based solution, with successful deployments by worldwide IXPs including a Tier-1 provider. It addresses all the design considerations mentioned in the previous section.

EVPN, defined in RFC 7432, is an advanced networking technology that provides both Layer 2 and Layer 3 VPN services over an IP network infrastructure. EVPN is designed to address the evolving connectivity and scalability requirements of modern data centers, service providers, and enterprise networks. It brings together the benefits of traditional Layer 2 VPNs with the flexibility and efficiency of Layer 3 routing.

Control Plane Learning

EVPN uses BGP (Border Gateway Protocol) to distribute MAC addresses and IP prefix reachability information across the network. By centralizing MAC address learning and distribution, control plane learning helps address the limitations of data plane learning for scalability, consistency, and resource consumption. It provides a more efficient and manageable way to handle MAC addresses in larger and more dynamic network environments.

Network Segmentation and Virtualization using VxLAN

VxLAN is designed for creating virtual overlay networks that extend Layer 2 segments across Layer 3 boundaries, providing enhanced scalability, flexibility and multi-tenancy traffic segmentation for modern network architectures. VxLAN is a tunneling protocol where the customer traffic can be tunneled across the underlay IP network without reconfiguration. IP routing is a proven, stable, and scalable underlay network which takes advantage of ECMP to utilize all available network paths. Running on top of a routed IP network, VxLAN increases scalability of Layer 2 domains up to 16 million using 24-bits VxLAN Network Identifier (VNI/VNID). VxLAN overcomes the 4K limitation faced by classic VLANs.

Public and Private Peering

Let’s take a look at how OcNOS Data Center uniquely manages the VxLAN VNI or overlay L2 domain assignment or mapping for both public and private peering services.

  • Public Peering: In this arrangement, multiple networks connect to a shared Layer 2 domain and exchange traffic with each other. Public peering is open to any network and allows networks to exchange traffic with a wide range of other participants, including internet service providers (ISPs), content providers, and mobile operators. To participate in a public peering LAN, the traffic is typically untagged at the IXP network access port.
  • Private Peering: Private peering involves a direct, private connection between two networks within the same IXP’s infrastructure shared with public peering. It offers greater control over the quality of the connection and may provide improved performance compared to public peering. Each peering member is assigned a unique VLAN tag that corresponds to their private peering connection to access the IXP network.

For public peering service, all participant traffic can be identified based on the assigned IXP network access port or redundant ports and being untagged. All participant traffic for public peering will be mapped to a unique VNI assigned for the public peering service.

For private peering service, the traffic of the participants can be identified based on the VLAN tag uniquely assigned to a private peering service. The traffic of the two participants for a private peering will be mapped to a unique VNI assigned for the private peering service. For two reasons, this way of mapping may not be the most efficient and flexible. A participant may want to retain its assigned VLAN tag when changing to a different private peering arrangement with another participant. The other reason is that a typical available VLAN range at an IXP is smaller than 4K, which would limit the scale of the private peering service and other services requiring VLAN use.

IP Infusion OcNOS Data Center provides a unique solution for these two scenarios. OcNOS Data Center can identify participant traffic through a port+VLAN combination, as opposed to just VLAN alone, when mapping to a service VNI. For example, participant A and participant B are in a private peering service with VNI 10110 ID. They also have access ports 3 and 4, respectively on the same access switch A1, both using 110 VLAN tag for the traffic being exchanged through the private peering service. Participant A now wants to change private peering to be with participant C under a new service with VNI 10120 ID. OCNOS Data Center can now make the following new mapping arrangement:

In the above example, Participant A can retain the existing VLAN tag and does not need to make any change when making a new private peering arrangement. For the IXP provider, only a new switch/port/VLAN to VNI mapping needs to be provisioned and no new switch needs to be used to support overlapping VLAN IDs across different private peering services. By supporting overlapping VLAN IDs on the same switch based on unique port+VLAN to VNI mapping, OcNOS Data Center allows the IXP to scale the service while minimizing the number of switches required for a much-improved TCO.

Multihoming for Redundancy

OcNOS Data Center supports all-active EVPN multihoming to allow devices to be connected to multiple Provider Edge (PE) routers, or IXP network access switches, for redundancy and load balancing. It ensures seamless failover and mobility while maintaining consistent MAC and IP address distribution. In the event of a PE router or access link failure, traffic is redirected without disruption. When a device moves from one PE router to another (e.g. during service upgrade to a higher speed access port), EVPN ensures that the device’s connectivity is maintained and seamlessly transferred to the new PE router. The failover and migration process is dynamic and managed by the EVPN control plane. All-active EVPN multihoming also allows for load balancing of traffic between multiple PE routers to achieve better performance and improved utilization of bandwidth.

Auto-discovery

OcNOS Data Center EVPN VxLAN supports auto-discovery of VXLAN tunnels and MAC address reachability information among the participating network devices. This enables devices to establish VxLAN tunnels and exchange traffic without requiring manual configuration of each individual connection, primarily through BGP signaling.

OcNOS Data Center EVPN VxLAN extends auto-discovery to handle scenarios where devices are multi-homed across different access switches. When a participant’s router or switch is connected to multiple IXP network access switches for redundancy, EVPN VxLAN auto-discovery ensures that the MAC address information is dynamically updated and correctly distributed to all relevant access switches.

OcNOS Data Center EVPN VXLAN auto-discovery can be integrated into network automation and orchestration tools, providing a unified view of the network for real-time monitoring and streamlined management.

Easy Network Expansion

Many modern data centers adopt a leaf-spine network architecture for scalability and simplicity. The IXP participants connect their equipment to the IXP network’s leaf switches, while spine switches provide high-speed connectivity between leaf switches. This design minimizes latency and enables easy expansion.

The distributed nature of the spine-leaf architecture complements the scalability and multi-homing features of EVPN. EVPN can be used to provide efficient and seamless Layer 2 and Layer 3 connectivity across the spine-leaf fabric, facilitating mobility, redundancy, and efficient use of network resources. The combination of EVPN and spine-leaf architecture helps data centers achieve the performance required to support modern applications, cloud services, and multi-tenant environments. The OcNOS Data Center powered IXP EVPN network not only provides a well-connected and resilient environment for IXP participants, but can offer additional services such as colocation and private VLAN.

OcNOS Data Center is available as a “whitebox” leaf and spine switching platform. OcNOS Data Center HW and SW products enable IXPs to build and expand the network using switches with the latest generation speeds and support access link speeds up to 400Gbps.

Traffic and Flooding Management

IXPs requires traffic engineering to optimize traffic flows in the network to minimize latency and congestion. OcNOS Data Center supports fine grained QoS policies using ingress/egress traffic policing to manage member traffic.

When it comes to mitigating the vulnerability to traffic injection from peers and non-peers as a security threat, OcNOS Data Center supports storm control with rate limiting.

OcNOS Data Center also provides a set of features to minimize ARP and ND flooding. By supporting EVPN learning from ARP requests, it can prevent flooding ARP requests and replies that can cause significant broadcast traffic and negatively impact network performance. OcNOS Data Center uses EVPN MP-BGP to distribute ARP information to only the relevant network segments or devices to reduce unnecessary traffic, improving network efficiency and stability.

A public peering LAN at a large IXP would inevitably become very large with hundreds or even over a thousand participants. Since all participants of a public peering LAN are in a single VNI or L2 domain, when an access link goes down unexpectedly, the traffic destined to that access link could start flooding to all other ports as part of BUM treatment. To avoid such flooding, OcNOS Data Center supports MAC hold timer to hold the MAC in hardware for some time until the control plane communicates to all devices to remove the unicast entry.

To Learn More about IP Infusion OcNOS Data Center

Additional information about OcNOS Data Center can be found in the following documents available on the IP Infusion website.

For more information, Contact us today to learn more about OcNOS Data Center.

Alan Huang is the Senior Product Manager, Data Center for IP Infusion.