Networks are critical infrastructure in nearly every computing system, but they are particularly important in High Performance Computing (HPC) systems, where users tackle problems at massive scale. The purpose of this document is to provide a starting point for further research by compiling and briefly synthesizing information across vendors and network layers. Due to the breadth of topics covered, it is most useful to new students studying HPC networks, but it remains useful to professionals as a collection of useful references. While there are common elements between HPC clusters and HPC on the cloud, this document is focused solely on cluster implementations.

Discussion of these components is organized according to Figure 1 in a structure similar to the Hybrid Model elaborated on in Appendix A.1, alongside all other major network architecture and performance models. After covering the layers and the components that reside within them, we will illustrate their connections with an example architecture and discuss performance metrics.

2 Libraries & APIs

HPC development relies on a wide range of communication and networking libraries designed to help move data efficiently while abstracting some amount of underlying complexity. Some libraries operate at a low-level to provide experts precise control, while others provide higher-level abstractions to make parallel programming more accessible.

2.1 High-level Communication Libraries

Most HPC programmers do not directly interface with the networking details. Instead, they use high-level libraries that run on all systems. To be successful, these libraries must abstract underlying networking details (topology, functionality, etc.), while achieving high performance using one or more networking technologies discussed later in this paper. This enables users, including domain scientists, to write performant code regardless of where they are running it.

MPI

The Message Passing Interface (MPI) [38] is a standardized API for distributed-memory parallel programming, providing point‑to‑point and collective communication across nodes. The primary model used in MPI is the two-sided model which consists of two main types of communication: point-to-point and collective. Point-to-point communication functions allow data to be exchanged directly between specified pairs of processes, while collective functions (Broadcast, Allreduce, etc) communicate between larger groups of processes and are often composed of more complicated algorithms built from many point-to-point or other networking functions. Modern implementations of MPI offer functionality designed to enable users to easily combine off- and on-node parallelism (see Section 2.2).

Collective Communication Libraries

These libraries are subsets of MPI focused on frequently used point-to-point and collective operations. Without the burden of implementing the entire MPI standard, they can focus developer effort on optimizing collectives used by target applications such as deep learning and simulations. The most well-known are vendor-provided APIs optimized for specific hardware. Examples include NCCL [42] for NVIDIA GPUs, RCCL [4] for AMD GPUs, and oneCCL [32] for Intel processors. Alternatively, HiCCL [24] is a research implementation that seeks to enable hardware-agnostic performant communication.

SHMEM

SHMEM [50] is a Partitioned Global Address Space (PGAS) model offering one‑sided put/get operations and low‑latency remote memory. This model is distinct because its semantics mimic shared memory despite it being a message passing model underneath.

OpenSHMEM [10] provides a standard reference implementation to unify pre-existing libraries and guide future development. NVSHMEM [43] is NVIDIA’s implementation of OpenSHMEM tailored to CUDA-enabled devices. It operates with a symmetric heap; each process allocates memory of identical size at the same time. To avoid a CPU bottleneck (a performance limitation due to the CPU needing to orchestrate communication), users may optionally use the InfiniBand GPUDirect Async transport to directly interface with the NIC. However, this approach does not guarantee memory consistency, which is still reliant on the CPU in NVSHMEM. AMD’s OpenSHMEM implementation rocSHMEM [55] is comparable to NVSHMEM, but adds memory consistency enforcement from within the kernel via the GPU-IB backend, removing the CPU bottleneck [65].

2.2 On-node Shared Memory APIs

These APIs provide abstractions to streamline on-node communication tuning while reducing developer effort.

OpenMP

OpenMP [16] is an established API for shared memory parallel programming, namely offering parallel loops and compiler directives. OpenMP traditionally targets shared-memory multi-processors that are connected at a hardware level and use the same memory, which are increasingly being used in combination with off-node parallelism. Additionally, some implementation are designed to be used across multi-processors over a network as in [29].

Kokkos

Kokkos [64] focuses on data management and parallel execution via the View datatype, which manages complex multi-dimensional arrays with various data types and layouts using shared-pointers and C++ templates alongside parallel computation dispatches that run on top of parallel back-ends (ex. OpenMP, CUDA, etc.). These features are designed to provide performance portability by abstracting features of several on-node performance models. For example, an application using Kokkos can be migrated from an NVIDIA platform to an AMD platform simply by modifying a couple of build flags. Other libraries such as RAJA [8] and Alpaka [71] achieve similar portability. Portability efforts have also expanded into Python with the addition of PyKokkos [2].

2.3 Portable Networking APIs

Libraries at this layer exist to bridge the gap between the single-provider libraries below and the high-level communication abstractions above by providing guarantees of portability. While high level libraries can still run on individual network libraries, these portable network libraries remove code duplication and reduce development time by abstracting similar vendor-specific objects and functionality into higher-level generic abstractions, as seen in Tables 1 and 2.

LibFabric

LibFabric [22] is a high‑performance abstraction layer that provides a unified API for exposing the capabilities of diverse network resources. However, unlike fully standardized communication libraries (e.g., MPI, OpenSHMEM), LibFabric does not require network providers to implement the full extent of the LibFabric API. Instead, vendors are encouraged to implement essential capabilities such as remote memory access (RMA) and message passing so that their networks are compatible with LibFabric, if only in a minimally viable capacity. These essential capabilities include scalable messaging, RMA, and atomic operations for HPC fabrics to meet the requirements of MPI and SHMEM programming models, among others.

UCX

Unified Communication X (UCX) [60] is a portable network framework that sits on top of other high-level transport APIs such as LibFabric rather than vendor-specific implementations. UCX is comprised of three components:

•

UC-S provides basic portable services, such as memory management, data structures, and platform functionality
•

UC-T implements transport functionality for a variety of different hardware and interfaces with things like InfiniBand (see Section 2.4) and GPUs
•

UC-P sits on top of the other two to implement higher-level programming abstractions for high-level models like MPI

Portals

Portals [6] is a network programming interface that facilitates performant, scalable communication between nodes. Portals is designed for the implementation of high-performance programs and libraries, and supports MPI and OpenSHMEM, as well as other parallel models. Portals is based on the use of put or write and get or read operations along with various atomic memory operations, known as a one-sided model. This one-sided model involves an initiator process that specifies the action and a target process that the action is happening to. To properly support two-sided models, such as MPI, Portals has versions of put and get that uses additional matching criteria for the memory address. The Portals API is intended to be portable, however it is reliant on reference implementations to provide support for different systems. At the time of writing, the official reference implementation only supports InfiniBand, UDP, and shared memory. Portals also hides the majority of the internal objects that appear in LibFabric and others, which is why it does not have an entry in Table 1.

2.4 Network Provider APIs

These lowest-level APIs provide a means for directly controlling network hardware of specific vendors, allowing users to take advantage of vendor-specific features like hardware offload, as in InfiniBand Host Channel Adapters. The network provider APIs in this section represent the network hardware present in nearly all HPC clusters in use today. The remaining networks are either custom-made or use high-bandwidth Ethernet.

OPX

OPX is the current iteration of Cornelis Networks’ OmniPath network API. OPX is implemented as a LibFabric network provider, meaning there is no vendor-specific API to learn. Because of this, OPX’s fundamental capabilities are those of LibFabric - scalable message passing, RMA, and atomics - however, OPX does provide a variety of runtime parameters that can be used to more finely tune performance that do not exist in other providers [17].

Slingshot / Cassini eXtended Interconnect (CXI)

Slingshot is an interconnection network designed for exa-scale HPC systems and data centers from HPE/Cray. Like OmniPath, HPE exposes Slingshot’s capabilities through LibFabric rather than developing its own API [36].

InfiniBand and RoCE

Unlike the previous two HPC network providers, InfiniBand has its own specialized APIs in addition to a LibFabric provider. The main API, VPI Verbs is a single API that supports both InfiniBand and RDMA over Converged Ethernet (RoCE) [49, 54].

Table 1: A Translation Table for InfiniBand, LibFabric, and UCX Objects*

Object Description	LibFabric Equivalent	UCX Equivalent	InfiniBand Equivalent
Top-level group object	fi_fabric	–	ibv_device/ ibv_context
Mid-level group of related objects	fi_domain	ucp_context	ibv_pd
Connection request listener	fi_pep	ucp_listener_conn_x	ibv_qp*
Events queue	fi_eq	–	rdma_event_channel
Communication hub	fi_ep	ucp_worker	ibv_qp*
Completion queue	fi_cq	–	ibv_cq
Address list	fi_av	–	ibv_ah
Memory registered for RMA operations	fi_mr	ucp_mem_h	ibv_mr

Table 2: A Translation Table for InfiniBand, LibFabric, UCX, and Portals Functions*

Function Purpose	LibFabric Equivalent	UCX Equivalent	InfiniBand Equivalent	Portals Equivalent
Listen for connection requests	fi_listen	ucp_listener_create	rdma_listen	–
Initiate connection request	fi_connect	ucp_ep_create	rdma_connect	–
Accept connection request	fi_accept	–	rdma_accept	–
Reject connection request	fi_reject	ucp_listener_reject	rdma_reject	–
Retrieve connection request	fi_eq_read	–	rdma_get_request	–
Add address to list	fi_av_insert	–	–	PtlSetMap
Look up an address in table	fi_av_lookup	–	–	PtlGetMap
Send a message	fi_send	ucp_tag_send_nb	ibv_post_send	–
Receive a message	fi_recv	ucp_tag_recv_nb	ibv_post_recv	–
Poll completion queue	fi_cq_read	ucp_worker_progress	ibv_poll_cq	PtlEQGet
Register memory region for RMA	fi_mr_reg	ucp_mem_map	ibv_reg_mr	PtlMDBind
Access the memory region key	fi_mr_key	ucp_rkey_unpack	–	–
Write data via RMA	fi_write	ucp_put	ibv_post_send	PtlPut
Read data via RMA	fi_read	ucp_get	ibv_post_send	PtlGet

* OPX and Slingshot are not present in these tables because they are only implemented as LibFabric providers.

3 Transport Protocols

The purpose of transport layer protocols is to define the semantics of connection management as well as communication initiation and finalization. While different network vendors have their own transport protocols, many of the capabilities, connection modes and communication semantics are based largely on the original transport protocols, TCP and UDP.

3.1 TCP & TCP/IP

TCP (Transmission Control Program) was the core protocol of the peer-to-peer ARPAnet system in the early 1970s. Later, TCP was supplemented with IP (Internetwork Protocol) to expand and handle the complexity of the fledgling internet. TCP (Transmission Control Protocol) then became the host (peer)-to-host protocol for local handling of messages and the IP layer was used for routing message packets to their intended destination.

The Secure Shell (SSH) tool uses encryption to secure the by-default insecure network connections of TCP/IP. SSH adds encryption via key-pair authentication; the public key is shared with systems you want to connect with, and the private key is used to authenticate the public key. SSH also allows tools like Secure Copy Protocol (SCP), which uses SSH to copy files, and can even be used to forward general internet traffic securely [7].

3.2 UDP

The other main transport protocol is UDP – User Datagram Protocol. UDP is a connectionless, best-effort transport protocol emphasizing minimal overhead at the cost of no delivery guarantees. A UDP session may start with a TCP connection for a user to join a UDP stream, but no further guarantees of reliability are provided past that point.

4 Network Control Plane

The control plane merges routing, topology, and connection management to define policies for how traffic should flow.

4.1 SDN Controllers

Software-defined Networking (SDN) controllers manage data flow by adapting to topology of each network and implementing data movement algorithms required to satisfy application requirements, essentially functioning as a network operating system. This ensures more flexibility and the ability to monitor and change data flow in real-time [72]. Mellanox NEO is one example of an SDN controller designed for HPC [45].

4.2 Fabric Managers

A fabric manager discovers the fabric of a system, assigns message routes, enforces policies, and handles fault detection and recovery. They maintain global state to keep the fabric stable and optimized. Some examples of fabric managers include Subnet Managers and UFM for InfiniBand [44] and the open fabric manager OpenSM [46].

4.3 Routing Dæmons

Refer to caption — Figure 2: Functional overlap in the Network Control Plane

Routing dæmons implement routing protocols that compute forwarding tables (also known as a forwarding information bases) based on network topology and policy that enable adaptive routing (see Section 5.5).

Quagga [51] is a foundational suite of IP routing software that was later forked and continued as FRRouting (FRR) [20]. FRR exchanges routing information, makes policy decisions, and informs the OS kernel of these decisions. It operates as a suite of dæmons, most of which are protocols, with one dæmon (zebra) responsible for coordinating decisions.

Another example is The BIRD Internet Routing Dæmon, a Linux routing dæmon. Like other routing dæmons, BIRD handles routing and topology, but it offloads data transfer operations onto the Linux kernel[69].

4.4 VLAN/PFC Configuration

Virtual Local Area Network (VLAN) configuration partitions physical networks to improve efficiency and reduce network congestion. Priority flow control (PFC) configuration similarly manages traffic in high-performance networks. It enables prioritization of certain types of traffic to facilitate critical communication even on congested networks.

4.5 Storage Area Networks

Storage Area Networks (SANs) are independent networks dedicated to storage within a data center or supercomputer that allow data to be stored and retrieved via network communication with a high degree of concurrent access and data redundancy. The primary technologies used to create a SAN are SCSI, Fibre Channel, and NVMe, however NVMe’s storage network capabilities are built into its regular capabilities, thus it is not discussed in this section.

Small Computer Systems Interface (SCSI) and Internet SCSI

The Small Computer Systems Interface (SCSI, pronounced “scuzzy”) was created in 1986 to provide a standardized parallel electronic bus for servers to interface with hardware peripherals (i.e., storage devices) faster and with more options than previous interfaces [58]. The Internet Small Computer Systems Interface (iSCSI) is a TCP/IP interface that makes remote SCSI requests, enabling data to be requested and retrieved from storage devices remotely, allowing for simple creation of storage area networks on existing SCSI nodes [63, 70].

Fibre Channel (FC)

Fibre Channel is a type of Storage Area Network consisting of an FC Fabric and its constituent Switches. The FC Fabric is a network of Host Bus Adapters (the FC equivalent of NICs) that represents the interior of the SAN network where the only network activity is from endpoints inside the SAN. FC Switches represent the edges of the SAN and are capable of using multiple protocols, such as iSCSI and Ethernet. Regarding Ethernet, FC has a specific protocol for remote requests like iSCSI that uses Ethernet rather than TCP/IP [18].

5 Hardware & Link-Layer Protocols

As essential as the various software layers are in routing data to the correct destination, none of them would mean anything without the hardware to physically move data from device to device. There are three main components required to successfully transfer data between devices, which all differ based on the network they are part of. These components are the connection medium, which carries analog signals over either a cable or a wireless signal, the link-layer device, which translates between the analog signals of the physical connection and the digital signals of the device, and the link-layer protocol, which defines how bits must be formatted in order to be interpreted correctly. Some high-performance link-layer devices may also enable additional functionality, such as RDMA. This section will cover all of the connection mediums, link-layer devices, and protocols used in high-performance networks.

5.1 Connection Mediums

While wireless connection mediums provide convenience, they are not able to match the bandwidth and latency of wired connections, and for this reason, wireless mediums are rarely used in HPC networks. Instead, supercomputers will generally use a combination of twisted-pair and fiber optic cables to maximize the benefits of each while minimizing trade-offs.

Twisted-pair

Twisted-pair cables consist of two intertwined copper wires that each carry opposite signals, one positive voltage and one negative voltage. The link-layer device can then “read” the value carried by the twisted-pair cable by taking the difference in voltage of the two wires. If the voltage is positive, the transferred bit is read as a 1, or 0 if the voltage is negative. The twisted-pair method is more reliable than a single wire, but doubles the power required and lowers the overall bandwidth due to the higher risk of electrical interference.

Fiber Optic cables

Table 3: Summary of Wired Medium Attributes

Attribute	Twisted-Pair	Fiber Optic
Max Bandwidth	10 \unit\giga\per	100+ \unit\giga\per
Avg. Propagation Speed	540 \unit\nano\per	3.34 \unit\nano\per
Cost	Moderate	High
Interference Risk	Moderate	Low
Degradation Risk	High	Low

Fiber optic cables are made of either a clear plastic or silica glass cable of propagating visible light. This construction allows them to propagate data much faster and farther than electrical signals are able to. Fiber optic cables are also able to achieve much higher bandwidths due to the lack of electrical interference and frequency division multiplexing, sending data from different packets of different wavelengths of light. While many networks (i.e., Ethernet) still use twisted-pair, fiber is increasingly common [61].

Forms of Fiber Optic and Twisted-Pair Cable in High-Performance Networks

The four major types of networks present in supercomputers all use a combination of 3 types of cable. Direct Attach Copper (DAC) is shielded twisted-pair cable with a connector attached. DAC is not powered and is not signal boosted, meaning the cable can only be as long as electrical signals last before they degrade, and should only be used over short distances. Active Copper Cable (ACC) is similar to DAC but adds integrated signal boosters to allow signals to travel further. Active Optical Cable (AOC) are essentially high quality fiber optic cables with the optical transceivers (required to convert the light signals into electricity) already attached to the ends [27].

5.2 Fabrics

Fabrics provide a network-level interconnect with routing, flow control, and mechanisms for fault handling.

Ethernet Protocol and Devices

Ethernet is a link-layer protocol for creating local-area-networks (LANs) that has been around in various forms for over 50 years, evolving from a three megabit coaxial connection to modern versions that can transmit upwards of 100 gigabits with fiber optic connections and can be used on the majority of network devices. Like all networks, Ethernet has its own variation on the traditional mediums. In this case, Ethernet uses the RJ45 connector to interface between link-layer devices and an 8-pair variant of twisted-pair cable. Ethernet is compatible with fiber optic cables, but they must go through a separate physical layer device that converts the optic signals to electric signals, such as a Fiber Media Converter [61].

Omnipath Protocols and Devices

Cornelis OmniPath uses a proprietary link-layer protocol that provides reliable data transfer and packet integrity guarantees. The primary link-layer device for OmniPath is the Cornelis SuperNIC which can accept all three cable formats [12].

InfiniBand and RoCE Protocols and Devices

InfiniBand is a complete network specification containing proprietary protocols at every layer that determine InfiniBand’s control features and packet structure [30]. InfiniBand Host Channel Adapters and higher-level hardware (e.g., ConnectX network adapters) are the only devices that can use these policies. Active Optical Cables are the primary cable used with Infiniband hardware [23].

Slingshot Protocols and Devices

Slingshot uses a proprietary, high-performance form of Ethernet as its link-layer protocol, which is implemented in all HPE NICs, which enables interoperability with other Ethernet devices. HPE NICs accept AOC and DAC cables as their primary connections [28] and provide congestion control and adaptive routing (see Section 5.5) with low latency and packet overhead [59].

5.3 Interconnects

Interconnects are high-bandwidth links between components in a single system.

PCIe

PCI Express (PCIe) is a standard used to connect hardware components that was designed to replace older expansion bus standards to better serve today’s computing devices. PCIe is implemented as a simple fabric with a tree-like topology in which each leaf node represents a device or other endpoint. PCIe has recently been used as the core infrastructure for other interconnects including Compute Express Link (CXL) , which moves data at a rate of over 128 billion transfers per second while maintaining backward compatibility the PCIe standard [13].

NVLink

NVLink is NVIDIA’s GPU interconnect that allows GPUs (and sometimes CPUs) to share memory and exchange data faster than PCIe [41, 14]. NVLink 5.0, part of NVIDIA’s Blackwell architecture, is designed for modern GPU workloads (including AI workloads).

xGMI

xGMI is AMD’s GPU interconnect protocol implemented through Infinity Fabric. [57]. It is comparable to NVLink in design goals and application targeting. Distinctly, it only interconnects up to 8 GPUs per switch with a theoretical max of 64 GB/s between two GPUs, while realized performance tends to be around 45-48 GB/s [35].

5.4 Topologies

Topologies are physical and logical patterns used to interconnect computing components. In HPC, efficient topology designs tend to focus on minimal diameters between components (e.g., dragonfly) or high path diversity (e.g. fat tree) [39].

Torus: This topology is composed of an $n$ -dimensional grid of compute nodes in which all edges are connected to the nearest neighbor on a given axis [34]. This reduces diameter compared to a pure mesh because of its ring-like structure. A notable modern example is the Tofu interconnect [1].

Hypercube: Hypercube topologies limit the number of transfers needed to reach a given node by maintaining a high degree of connectivity. Each node-level device acts as a binary tree of next hops for each message, limiting the number of hops to $O(log(N))$ where $N$ is the number of nodes in the network [48].

Dragonfly: This topology connects a group of routers using a subnetwork to form a virtual high-radix router with the intention of minimizing global diameter [34]. This is the topology utilized in our representative example in Section 6.

Slimfly: This topology leverages degree-diameter graphs to achieve a diameter of 2. However, high path diversity is not a design consideration [39].

Fat Tree: This topology is an extension of the tree topology, where the root node is at the center and child nodes split farther from there. Loads on each link increase towards the root. In the fat tree topology, switches near the root are replaced with many smaller switches operating as one. This has been frequently used in conjunction with InfiniBand technology [34].

Jellyfish / Random Regular Graph (RRG): This topology connects switches with a random graph. This offers high-connectivity and low-diameter, but routing must be managed thoughtfully because shortest path approaches are not effective [68, 39].

5.5 Adaptive Routing

Adaptive routing allows switches to dynamically select from multiple viable ports based on real-time congestion conditions to avoid deadlock. Address issues are resolved using path selection (i.e., which paths may be used) and load balancing (i.e., which of the paths identified should be chosen) [9].

6 Putting It All Together

To illustrate how all of the technologies covered in previous sections are implemented, this section focuses on a holistic analysis of one of the most notable HPC architectures as of the writing of this guide: El Capitan.

El Capitan is an exascale supercomputer with over 11,000 nodes in 90 compute cabinets, necessitating significant networking infrastructure [66]. Networks connect components in this system on a variety of scales: within the APU, within the node, from node to node within the system, and externally to users and the internet. This example emphasizes the need for co-design between networking and computation: the tightly-coupled nature of the APU informed decisions that impacted how the network would operate at the node-level and beyond.

Figure 3: Depiction of El Capitan’s node-level architecture.

An example node on this system is represented in Figure 3. At the node level, xGMI3 (Section 5.3) Infinity Fabric links are used for communication between APUs. PCIe (Section 5.3) connects to near-node local storage and individual NICs within the Cassini ASIC (application-specific integrated circuit) [56]. In the diagram, the distinction between these connections is support for extended speed mode (ESM). Slingshot (Section 5.2) is used for node interconnection, with four interfaces per node and a dragonfly topology.

At the processor level, the six accelerator complex dies (XCDs) (operating as GPUs) connect with three CPU complex dies (CCDs) and eight high-bandwidth memory stacks (HBM) via Infinity Fabric [66]. Each of these components has its own cache: 4MB of L2 cache on the XCDs, 32MB of L3 cache on the CCDs, and an Infinity Cache for the HBM. Infinity Fabric is also the means of interfacing externally with other APUs in the node.

To optimize application performance, users may utilize MPI (Section 2.1) using the Cray MPICH implementation available on El Capitan. It is GPU-aware and works with the ROCm compiler (Section 2.1). All of these come together, from the physical layer to the control layer to the application layer, to form a complete system for users. Many users might not use anything more complex than MPI, but for HPC users, performance is indispensable.

Appendix A Network Classifications and Performance Modeling

Understanding and optimizing new technology is of utmost importance in HPC and computer science, thus it is only natural that models are developed to serve these purposes. The models covered in this section are some of the most widely adopted and useful for classifying network technologies and understanding performance.

A.1 Network Architecture Models

Classification and organization of network technologies is a difficult task, and one that only becomes more daunting as time goes on and networks become more complex. The following three models are relevant not only to traditional networks, but also modern HPC networks.

The OSI Model

Table 4: OSI Model layer descriptions.

OSI Layer	Description
Application	Source and destination of all messages
Presentation	Compresses and encrypts data
Session	Manages communication sessions
Transport	Manages connections with other endpoints
Network	Routes messages to the correct endpoints
Data-Link	Formats data & moves it to/from link
Physical	Carries signal over long distances

The Open Systems Interconnection (OSI) Model was created by the International Organization for Standardization [33] to serve as a common model for network standards development in 1984, then updated in 1994. The model’s design was based on theories of networking rather than observation, because the ISO sought to create a new standard that would allow future computers to remain ”open” to one another instead of creating a model to classify existing network technologies. While it is more detailed than other models, it is less useful for describing real network technologies than other models in both everyday computing and HPC [47].

The TCP/IP Model

Table 5: Comparison between OSI model and the TCP/IP model layers.

TCP/IP Layer	OSI Layer(s)
	Application
	Presentation
Application	Session
Transport	Transport
Network	Network
	Data-Link
Network Link	Physical

In contrast to the OSI model, the TCP/IP model arose directly out of a need to organize the standard protocols needed to use the early internet. The Internet Engineering Task Force (IETF) was organized in 1986 by the US Defense Advanced Research Projects Agency (DARPA) to manage and operate the early internet, and to set new standards that dictate how networks operate. Two such standards, RFC 1122 and 1123, established the required software and protocols for networked hosts, which were organized by layer [52, 53]. These layers eventually morphed into the TCP/IP model shown in Table 5. Given its roots in the actual technologies the internet runs on, the TCP/IP model is often a more applicable framework than the OSI model, though it is less detailed.

The Hybrid Model

Table 6: Comparison between OSI Model and Hybrid Model layers.

Hybrid Model	Relation to OSI Model
	Application Layer &
Application Layer	Presentation Layer
	Session Layer &
Transport Layer	Transport Layer
Network Layer	Network Layer
Link-Layer	Data-Link Layer
Physical Layer	Physical Layer

A Hybrid model shown in Table 6, popularized by the first edition of Andrew S. Tanenbaum’s Computer Networks [62], synthesizes the organizational structure of the OSI Model with TCP/IP’s focus on concrete protocols (or, in our interpretation, on actual network technology rather than theory). The result of this combination is essentially the OSI model with the presentation and session layers removed because they are functionally empty in implementations. Their responsibilities were already handled by applications or lower layers.

A.2 Understanding Network Performance

Each of the following subsections will present new hardware properties and network metrics that can be used in the models presented at the end of this section. To keep things clear, a variable (and its appropriate metric units) is assigned to each value to represent it in formulas throughout this section.

Physical Properties of Network Hardware

Metrics of this category are known before the network is built and often define the limits of performance, not realistic expectations.

Propagation speed $s_{\mathrm{prop}}(\unit{\per})$ : the maximum speed at which a single bit travels across a link, determined by the material and structure of the link. The propagation delay $t_{\mathrm{prop}}$ (\unit) of a link is the minimum time required for a single bit to traverse the length of the link. This is sometimes confused with latency, which will be covered in the next section.

link length = $d_{\mathrm{link}}(\unit{})$

$t_{\mathrm{prop}}(\unit{})=d_{\mathrm{link}}(\unit{})/s_{\mathrm{prop}}(\unit{\per})$

Transfer rate $s_{\mathrm{transfer}}(\unit{\per})$ : the maximum speed at which a single bit can be injected into or removed from a link. The transfer rate is determined by data-link layer hardware (e.g., NICs, routers, etc.), not the link itself. The transmission delay $t_{\mathrm{transfer}}(s)$ of a packet is the minimum time required to inject the entirety of the packet onto the physical link.

amount of bytes per packet $=a_{\mathrm{packet}}(\unit{})$

$t_{\mathrm{transfer}}(\unit{})=a_{\mathrm{packet}}(\unit{})/s_{\mathrm{transfer}}(\unit{\per})$

Bandwidth $b_{\mathrm{max}}(\unit{\per})$ : the maximum amount of data that can travel over a link during a given period of time. This is fundamentally different from storage capacity, which is the amount of data that a device can hold ( $a_{\mathrm{device}}(\unit{})$ ). To state the difference more clearly, storage capacity describes the amount of data that can be stored on a hard drive, while bandwidth describes the hard drive while it is riding at 60 miles per hour down the highway.

Measurements of Network Performance

These properties define what the network between two endpoints is actually capable of, and can be used for predictive models. These values are usually determined experimentally using benchmarks and profiling tools.

Processing Delay $t_{\mathrm{process}}(\unit{})$ : the total compute time required before a network device can start putting data onto a link. This can include formatting incoming bits into packets, validating checksums, or calculating the next device to route a message to. Processing delays are consistent for similar communication patterns, so they translate well to predictive models.

Queuing Delay $t_{\mathrm{queue}}(\unit{})$ : the total time spent waiting to be processed. Like processing delay, queuing delay is consistent in similar situations. Queuing delay is difficult to measure directly, so a more reliable method is to determine how long all of the other delays along the network path take, and then subtracting those values from the end-to-end delay, also known as latency.

Latency $t_{\mathrm{total}}(\unit{})$ : the total time spent from the moment a message enters the source networking device to the moment it leaves the destination networking device. It can be predicted if all of the component times for each link are known by summing all of the delays for each link.

$n=$ # of endpoints

$t_{\mathrm{total}}(\unit{})=\sum_{j=1}^{n-1}(t_{\mathrm{prop}}^{j}(\unit{})+t_{\mathrm{transfer}}^{j}(\unit{})+t_{\mathrm{process}}^{j}(s)+t_{\mathrm{queue}}^{j}(\unit{}))+\sum_{k=0,n}(t_{\mathrm{transfer}}^{k}(\unit{})+t_{\mathrm{process}}^{k}(\unit{}))$

The first summation represents all of the time spent traveling between hosts, and the second summation represents all of the time spent on the hosts required to move the message.

Throughput $b_{\mathrm{actual}}$ (b/s): the measure of data flow during a benchmark or application. As its assigned variable name suggests, throughput can also be defined as the actual bandwidth the network can achieve under a real workload, whereas the word bandwidth itself usually refers to the maximum data flow of a connection. The average throughput of a link can be obtained by filling a network connection with more data than it can handle at one time and monitoring how much of the input data is received at the destination endpoint over a set time period. Dividing the amount of data received by the length of time allotted will give the average throughput.

Network Performance Models

All of the models in this section are used for predicting or understanding communication performance using either real values, including those discussed earlier in this chapter, or symbolic values.

Traffic Intensity $i_{\mathrm{traffic}}$ : an indicator of how well matched the throughput of an application is to the transfer rate of the device injecting data onto the physical link. A value near one indicates both are evenly matched, while a value above one indicates that the throughput of the link has much higher capacity than the NIC (or other device) can make use of. Values less than one indicate the link’s throughput is not sufficient and that messages in the attached data-link layer device will have longer queuing delays. At traffic intensities less than one, queuing delays approach the same value as transmission delays for each endpoint, and can be used as such in models.

$i_{\mathrm{traffic}}=b_{\mathrm{actual}}(\unit{\per})/s_{\mathrm{transfer}}(\unit{\per})$

Little’s Law ( $L=\lambda W$ ): a definition of the relationship between the amount of data entering a network ( $L$ ), the rate at which packets are transferred onto the network ( $\lambda$ ), and the total amount of time a packet spends on the network ( $W$ ). Like LogP, Little’s Law establishes a relationship between fundamental values of network performance rather than a predictive model [67, 37].

PRAM (Parallel Random Access Machine): an abstraction based on the sequential Random Access Machine used for reasoning about the time and memory complexity of parallel programs, and the relationship between sequential and parallel implementations of the same task. As with RAM, PRAM neglects some of the details of parallelism, such as communication and synchronization [19].

The Hockney Model ( $T=s+rn$ ): a fundamental latency prediction model created by Roger W. Hockney in 1982. The three variables of the model (startup time $s$ , bandwidth $r$ - originally called transfer rate -, and amount of data $n$ ) allow for quick but useful estimates of end-to-end latency. A large part of the model’s simplicity comes from the aggregation of multiple factors into the single ”startup time” variable, which includes physical limitations like propagation and transmission delay as well as software overhead such as the TCP connection process (see section 3.1 for more details) [25].

LogP ( $L+o+g+P$ ): a model for understanding communication costs and how they affect the performance of multi-process applications. Each of the letters of LogP represents a different area of performance: $L$ represents the upper bound of hardware latency, $o$ represents software overhead, $g$ represents the gap (transmission delay for small messages), and $P$ represents the number of communicating processes. Unlike many other models, LogP is used for understanding communication rather than predicting performance, as shown in Figure 4. The LogP model is only meant to be used with very small messages [15].

LogGP ( $L+o+g+G+P$ ): an improved version of LogP that also accounts for long messages. LogGP adds the capital G, which represents the ”Gap per byte” of long messages, and is equal to the inverse of bandwidth. Because G is a per-byte value, the LogGP model can account for messages of every size by simply changing the coefficient of G to the number of bytes per message [3].

LogGPS: the motivation for the LogGPS model was to capture the costs of varying modes of inter-process data transfer. The two primary transfer protocols used by high-level communication libraries like MPI are eager and rendezvous. Eager protocol is used for small messages and transfers data directly from the sending process to a temporary buffer on the receiving process, which is later copied to the buffer specified by the user. Rendezvous is used for messages that are too large to work with eager and other more optimal protocols, and requires full synchronization between processes before the message is transferred directly into the intended buffer [31].

Other LogP variants: there are a number of other variants on LogP that each aim to include one or more elements of communication not addressed by other models. Explaining each one in depth is out of scope for this document, but a brief explanation and appropriate resources are included for completeness.

•

LoGPC - Modifies the LogGP model to include the effects of network contention on communication [40]
•

LogGOPS - Extends LogGPS by adding $O$ , per-byte overhead. [26]
•

LogGOPSC - Extends LogGOPS to account for network contention [5]
•

LogGPO - Factors in the impact of communication-computation overlap on the LogGP model [11]

LogGOPSim: a simulator capable of predicting parallel application performance. Application communication patterns are specified in terms of the LogGOPS model (introduced in the same paper) before they are run through the simulator, which can accurately predict and visualize the time requirements for various algorithms, and includes constraints not explicitly accounted for in LogP-based models, such as MPI matching. Importantly, this model assumes that there is no network contention (i.e. that there is infinite capacity in the link, but a finite flow rate) [26].

References

[1] Y. Ajima, T. Shimizu, and S. Sumimoto (2009-11) Tofu: A 6D Mesh/Torus Interconnect for Exascale Computers . Computer 42 (11), pp. 36–40. External Links: ISSN 1558-0814, Document, Link Cited by: §5.4.
[2] N. Al Awar, S. Zhu, G. Biros, and M. Gligoric (2021-06) A performance portability framework for Python. In Proceedings of the ACM International Conference on Supercomputing, pp. 467–478 (en). External Links: ISBN 978-1-4503-8335-6, Link, Document Cited by: §2.2.
[3] A. Alexandrov, M. F. Ionescu, K. E. Schauser, and C. Scheiman (1995) LogGP: incorporating long messages into the LogP model—one step closer towards a realistic model for parallel computation. In Proceedings of the Seventh Annual ACM Symposium on Parallel Algorithms and Architectures, SPAA ’95, New York, NY, USA, pp. 95–105. External Links: ISBN 0897917170, Link, Document Cited by: Appendix A.
[4] AMD Corporation ROCm communication collectives library (rccl). External Links: Link Cited by: §2.1.
[5] Y. Baicheng, Z. Yi, X. Limin, H. Jiantong, and W. Zhaokai (2019) LogGOPSC: A Parallel Computation Model Extending Network Contention into LogGOPS. In 2019 IEEE International Conference on Cluster Computing (CLUSTER), Vol. , pp. 1–2. External Links: Document Cited by: 3rd item.
[6] B. W. Barrett, R. Brightwell, R. E. Grant, W. Schonbein, S. Hemmert, K. Pedretti, K. Underwood, R. Riesen, T. Hoefler, M. Barbe, L. H. S. Filho, A. Ratchov, and A. B. Maccabe (2022) The Portals 4.3 Network Programming Interface. Technical report Sandia National Laboratories. Cited by: §2.3.
[7] D.J. Barrett, R.E. Silverman, and R.G. Byrnes (2005) SSH, the secure shell: the definitive guide. Definitive Guide Series, O’Reilly Media. External Links: ISBN 9780596008956, LCCN 2005283598, Link Cited by: §3.1.
[8] D. A. Beckingsale, T. R. Scogland, J. Burmark, R. Hornung, H. Jones, W. Killian, A. J. Kunen, O. Pearce, P. Robinson, and B. S. Ryujin (2019-11) RAJA: Portable Performance for Large-Scale Scientific Applications. In 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC), Denver, CO, USA, pp. 71–81 (en). External Links: ISBN 978-1-7281-6003-0, Link, Document Cited by: §2.2.
[9] M. Besta, J. Domke, M. Schneider, M. Konieczny, S. D. Girolamo, T. Schneider, A. Singla, and T. Hoefler (2021) High-performance routing with multipathing and path diversity in ethernet and hpc networks. IEEE Transactions on Parallel and Distributed Systems 32 (4), pp. 943–959. External Links: Document Cited by: §5.5.
[10] B. Chapman, T. Curtis, S. Pophale, S. Poole, J. Kuehn, C. Koelbel, and L. Smith (2010-10) Introducing OpenSHMEM: SHMEM for the PGAS community. In Proceedings of the Fourth Conference on Partitioned Global Address Space Programming Model, New York New York USA (en). External Links: ISBN 978-1-4503-0461-0, Document Cited by: §2.1.
[11] W. Chen, J. Zhai, J. Zhang, and W. Zheng (2009-10) LogGPO: An accurate communication model for performance prediction of MPI programs. Science in China Series F: Information Sciences 52, pp. 1785–1791. External Links: Document Cited by: 4th item.
[12] CN5000 OmniPath. External Links: Link Cited by: §5.2.
[13] COMPUTE EXPRESS LINK CONSORTIUM, INC (2025) Compute Express Link® (CXL®) Specification. External Links: Link Cited by: §5.3.
[14] N. Corporation The NVIDIA Blackwell Architecture. External Links: Link Cited by: §5.3.
[15] D. Culler, R. Karp, D. Patterson, A. Sahay, K. E. Schauser, E. Santos, R. Subramonian, and T. von Eicken (1993-07) LogP: towards a realistic model of parallel computation. In Proceedings of the Fourth ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, Vol. 28, New York, NY, USA, pp. 1–12. External Links: ISBN 0897915895, Link, Document, ISSN 0362-1340 Cited by: Appendix A.
[16] L. Dagum and R. Menon (1998) OpenMP: an industry standard API for shared-memory programming. IEEE Computational Science and Engineering 5 (1), pp. 46–55. External Links: Document Cited by: §2.2.
[17] Libfabric Programmer’s Manual: fi_opx. External Links: Link Cited by: §2.4.
[18] Understanding fibre channel. External Links: Link Cited by: §4.5.
[19] S. Fortune and J. Wyllie (1978) Parallelism in random access machines. In Proceedings of the Tenth Annual ACM Symposium on Theory of Computing, STOC ’78, New York, NY, USA, pp. 114–118. External Links: ISBN 9781450374378, Link, Document Cited by: Appendix A.
[20] FRRouting. External Links: Link Cited by: §4.3.
[21] W. Gropp Lecture 26: performance models for distributed memory parallel computing. External Links: Link Cited by: Figure 4.
[22] P. Grun, S. Hefty, S. Sur, D. Goodell, R. D. Russell, H. Pritchard, and J. M. Squyres (2015) A Brief Introduction to the OpenFabrics Interfaces - A New Network API for Maximizing High Performance Application Efficiency. In 2015 IEEE 23rd Annual Symposium on High-Performance Interconnects, Vol. , pp. 34–39. External Links: Document Cited by: §2.3.
[23] Host Channel Adapters (HCAs). External Links: Link Cited by: §5.2.
[24] M. Hidayetoglu, S. G. de Gonzalo, E. Slaughter, P. Surana, W. Hwu, W. Gropp, and A. Aiken (2024) HiCCL: A Hierarchical Collective Communication Library. External Links: 2408.05962, Link Cited by: §2.1.
[25] R. W. Hockney (1982) The science of computer-aided design. Academic Press, London, UK. External Links: ISBN 978-0123510501 Cited by: Appendix A.
[26] T. Hoefler, T. Schneider, and A. Lumsdaine (2010) LogGOPSim: simulating large-scale applications in the LogGOPS model. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, HPDC ’10, New York, NY, USA, pp. 597–604. External Links: ISBN 9781605589428, Link, Document Cited by: 2nd item, Appendix A.
[27] DACs, ACCs, AOCs, and Transceiver Interconnects. External Links: Link Cited by: §5.1.
[28] HPE Slingshot QuickSpecs. External Links: Link Cited by: §5.2.
[29] Y.C. Hu, H. Lu, A.L. Cox, and W. Zwaenepoel (1999) OpenMP for networks of SMPs. In Proceedings 13th International Parallel Processing Symposium and 10th Symposium on Parallel and Distributed Processing. IPPS/SPDP 1999, Vol. , pp. 302–310. External Links: Document Cited by: §2.2.
[30] InfiniBand trade association. External Links: Link Cited by: §5.2.
[31] F. Ino, N. Fujimoto, and K. Hagihara (2001-06) LogGPS: a parallel computational model for synchronization analysis. SIGPLAN Not. 36 (7), pp. 133–142. External Links: ISSN 0362-1340, Link, Document Cited by: Appendix A.
[32] Intel Corporation OneCCL specification. UXL Foundation. External Links: Link Cited by: §2.1.
[33] ISO - International Organization for Standardization. External Links: Link Cited by: §A.1.
[34] N. Jain, A. Bhatele, S. White, T. Gamblin, and L. V. Kale (2016) Evaluating HPC Networks via Simulation of Parallel Workloads. In SC ’16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Vol. , pp. 154–165. External Links: Document Cited by: §5.4, §5.4, §5.4.
[35] J. Kolla, P. Alizadeh, and G. Lee Understanding RCCL Bandwidth and xGMI Performance on AMD Instinct™ MI300X. External Links: Link Cited by: §5.3.
[36] Libfabric and the HPE Slingshot NIC offloads. External Links: Link Cited by: §2.4.
[37] J. C. Little A Proof for the Queuing Formula: $L=\lambda W$ . External Links: Link Cited by: Appendix A.
[38] Message Passing Interface Forum (2025) MPI: a message-passing interface standard version 5.0. External Links: Link Cited by: §2.1.
[39] M. A. Mollah, P. Faizian, M. S. Rahman, X. Yuan, S. Pakin, and M. Lang (2018) A Comparative Study of Topology Design Approaches for HPC Interconnects. In 2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), Vol. , pp. 392–401. External Links: Document Cited by: §5.4, §5.4, §5.4.
[40] C. A. Moritz and M. I. Frank (1998) LoGPC: modeling network contention in message-passing programs. In Proceedings of the 1998 ACM SIGMETRICS Joint International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS ’98/PERFORMANCE ’98, New York, NY, USA, pp. 254–263. External Links: ISBN 0897919823, Link, Document Cited by: 1st item.
[41] Nvidia Corp NVIDIA NVLink SGXLS10 Switch Systems User Manual. External Links: Link Cited by: §5.3.
[42] NVIDIA Corporation NVIDIA collective communications library (nccl). External Links: Link Cited by: §2.1.
[43] NVIDIA openshmem library (nvshmem) documentation. NVIDIA. Note: Available: https://docs.nvidia.com/nvshmem/api/ Cited by: §2.1.
[44] NVIDIA Unified Fabric Manager (UFM). Nvidia Corporatation. External Links: Link Cited by: §4.2.
[45] NVIDIA NVIDIA NEO v2.7. External Links: Link Cited by: §4.1.
[46] OpenSM user’s manual. Mellanox Technologies. External Links: Link Cited by: §4.2.
[47] (1994) ISO/IEC 7498-1:1994 Information Technology — Open Systems Interconnection — Basic Reference Model: The Basic Model. Standard, International Organization for Standardization. External Links: Link Cited by: §A.1.
[48] G. Ostrouchov (1987-03) Parallel computing on a hypercube: an overview of the architecture and some applications. Cited by: §5.4.
[49] D. K. Panda and S. Sur (2011) InfiniBand. In Encyclopedia of Parallel Computing, D. Padua (Ed.), pp. 927–935. External Links: ISBN 978-0-387-09766-4, Link, Document Cited by: §2.4.
[50] PGAS Forum PGAS Forum. Note: http://www.pgas.org/ Cited by: §2.1.
[51] Quagga Software Routing Suite. External Links: Link Cited by: §4.3.
[52] R. Braden (Ed.) Requirements for Internet Hosts - Communication Layers. External Links: Link Cited by: §A.1.
[53] R. Braden (Ed.) Requirements for Internet Hosts - Application and Support. External Links: Link Cited by: §A.1.
[54] RDMA over Converged Ethernet. External Links: Link Cited by: §2.4.
[55] RocSHMEM documentation. AMD. Note: Available: https://rocm.docs.amd.com/projects/rocSHMEM/ Cited by: §2.1.
[56] D. Roweth, G. Faanes, J. Treger, and M. Terpstra (2022) HPE Slingshot Launched into Network Space. In CUG (Cray User Group) Proceedings, Cited by: §6.
[57] G. Schieffer, R. Shi, S. Markidis, A. Herten, J. Faj, and I. Peng (2024) Understanding Data Movement in AMD Multi-GPU Systems with Infinity Fabric. External Links: 2410.00801, Link Cited by: §5.3.
[58] INCITS 131-1994[s2013]: small computer systems interface. American National Standards Institute. External Links: Link Cited by: §4.5.
[59] D. D. Sensi, S. D. Girolamo, K. H. McMahon, D. Roweth, and T. Hoefler (2020) An In-Depth Analysis of the Slingshot Interconnect. CoRR abs/2008.08886. External Links: Link, 2008.08886 Cited by: §5.2.
[60] P. Shamis, M. G. Venkata, M. G. Lopez, M. B. Baker, O. Hernandez, Y. Itigin, M. Dubman, G. Shainer, R. L. Graham, L. Liss, et al. (2015) UCX: an open source framework for HPC network APIs and beyond. In 2015 IEEE 23rd Annual Symposium on High-Performance Interconnects, pp. 40–43. Cited by: §2.3.
[61] A.S. Tanenbaum, S. Tanenbaum, N. Feamster, and D.J. Wetherall (2022) Computer networks. Pearson. External Links: ISBN 9789356063600, Link Cited by: §5.1, §5.2.
[62] A. S. Tanenbaum (1981) Computer networks. Prentice-Hall, Englewood Cliffs, N.J.. External Links: ISBN 0131651838 Cited by: §A.1.
[63] J. Tate, P. Beck, H. H. Ibarra, S. Kumaravel, L. Miklas, and I. B. M. Corporation. I. T. S. Organization (2017) Introduction to storage area networks: ”sg24-5470-08”. IBM Corporation. External Links: Link Cited by: §4.5.
[64] C. R. Trott, D. Lebrun-Grandié, D. Arndt, J. Ciesko, V. Dang, N. Ellingwood, R. Gayatri, E. Harvey, D. S. Hollman, D. Ibanez, N. Liber, J. Madsen, J. Miles, D. Poliakoff, A. Powell, S. Rajamanickam, M. Simberg, D. Sunderland, B. Turcksin, and J. Wilke (2022) Kokkos 3: programming model extensions for the exascale era. IEEE Transactions on Parallel and Distributed Systems 33 (4), pp. 805–817. External Links: Document Cited by: §2.2.
[65] D. Unat, I. Turimbetov, M. K. T. Issa, D. Sağbili, F. Vella, D. D. Sensi, and I. Ismayilov (2026) The Landscape of GPU-Centric Communication. External Links: 2409.09874, Link Cited by: §2.1.
[66] Using El Capitan Systems: Hardware Overview | HPC @ LLNL. Lawrence Livermore National Laboratories. External Links: Link Cited by: §6, §6.
[67] S. P. van der Zee and H. Theil (1961-12) Priority Assignment in Waiting-Line Problems Under Conditions of Misclassification. Oper. Res. 9 (6), pp. 875–885. External Links: ISSN 0030-364X, Link, Document Cited by: Appendix A.
[68] X. Yuan, S. Mahapatra, W. Nienaber, S. Pakin, and M. Lang (2013) A new routing scheme for Jellyfish and its performance with HPC workloads. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC ’13, New York, NY, USA. External Links: ISBN 9781450323789, Link, Document Cited by: §5.4.
[69] O. Zajicek (2015) BIRD Internet Routing Daemon. External Links: Link Cited by: §4.3.
[70] E. Zeidner, C. Sapuntzakis, M. Chadalapaka, J. Satran, and K. Meth (2004-04) Internet Small Computer Systems Interface (iSCSI). Request for Comments, RFC Editor. Note: RFC 3720 External Links: Document, Link Cited by: §4.5.
[71] E. Zenker, B. Worpitz, R. Widera, A. Huebl, G. Juckeland, A. Knüpfer, W. E. Nagel, and M. Bussmann (2016-05-23) Alpaka - an abstraction library for parallel kernel acceleration. IEEE Computer Society. External Links: 1602.08477, Link Cited by: §2.2.
[72] L. Zhu, Md. M. Karim, K. Sharif, F. Li, X. Du, and M. Guizani (2019) SDN controllers: benchmarking & performance evaluation. CoRR abs/1902.04491. External Links: Link, 1902.04491 Cited by: §4.1.

1 Introduction