• Modeling and Evaluation of Chip-to-Chip Scale Silicon Photonic Networks
  Abstract
  Silicon photonic interconnects have been proposed as a solution to address chip I/O communication bottlenecks in multi-core architectures. In this paper, we perform comprehensive design exploration of inter-chip photonic links and networking architectures. Because the energy efficiencies of such architectures have been shown to be highly sensitive to link utilization, our design exploration covers designs where sharing occurs. By means of shared buses and silicon photonic switches, link utilizations can be improved. To conduct this exploration, we introduce a modeling methodology that captures not only the physical layer characteristics in terms of link capacity and energy efficiency but also the network utilization of silicon photonic chip-to-chip designs. Our models show that silicon photonic interconnects can sustain very high loads (over 100 Tb/s) with low energy costs (< 1 pJ/bit). On the other hand, resource-sharing architectures typically used to cope with low and sporadic loads come at a relatively high energy cost.

  Authors affliliation: Columbia University, USA

  R. Hendry, S. Rumley, D. Nikolova and K. Bergman
• Accelerating Spark with RDMA for Big Data Processing: Early Experiences
  Abstract
  Apache Hadoop MapReduce has been highly successful in processing large-scale data-intensive batch applications on commodity clusters. However, for low-latency interactive applications and iterative computations, Apache Spark, an emerging in-memory processing framework, has been stealing the limelight. Recent studies have shown that current generation Big Data frameworks (like Hadoop) cannot efficiently leverage advanced features (e.g. RDMA) on modern clusters with high-performance networks. One of the major bottlenecks is that these middleware are traditionally written with sockets and do not deliver best performance on modern HPC systems with RDMA-enabled high performance interconnects. In this paper, we first assess the opportunities of bringing the benefits of RDMA into the Spark framework. We further propose a high performance RDMA- based design for accelerating data shuffle in the latest Spark package on high-performance networks. Performance evaluation shows that our proposed design can achieve 79-83% performance improvement for GroupBy, compared with the default Spark running with IP over InfiniBand (IPoIB FDR) on a 128-256 core cluster. We adopt a plug-in based approach that can make our design to be easily integrated with newer Spark releases. To the best our knowledge, this is the first design for accelerating Spark with RDMA for Big Data processing.

  Authors affliliation: The Ohio State University, USA

  X. Lu, M. Rahman, N. Islam, D. Shankar and D.K. Panda
• End-to-End Adaptive Packet Aggregation for High-Throughput I/O Bus Network Using Ethernet
  Abstract
  Input/output (I/O) bus networks using Ethernets transfer a PCI Express (PCIe) I/O packet between a host and an I/O device by encapsulating it into an Ethernet frame. Because the size of PCIe packets is generally small in such systems, the overhead to individually encapsulate them into Ethernet frames lowers the PCIe bandwidth provided by Ethernet connections. This decrease in bandwidth directly degrades the performance of I/O devices, which transmit high-throughput PCIe traffic. Examples of such devices include PCIe solid-state drives and graphics processing units. We propose a method of aggregating multiple PCIe packets into a single Ethernet frame in an end-to- end manner to provide high-throughput PCIe bandwidth. The aggregation is performed at the bottleneck-link rate of the Ethernet path through which those packets are transferred. The number of aggregated PCIe packets is adaptively determined by aggregating ones that reside in a transmission queue when it is scheduled to transmit packets. This enables low-latency bandwidth enhancement because the proposed method does not increase the transmission latency of PCIe packets by waiting for more packets to aggregate. In addition, it does not require individual configuration of operation parameters, such as aggregation threshold and time out value, depending on the rate of the PCIe traffic of I/O devices. We implemented our method in a PCIe-to-Ethernet bridge prototype using a field-programmable gate array. As a result, I/O performance improved up to 41%.

  Authors affliliation: NEC, Japan

  J. Suzuki, Y. Hayashi, M. Kan, S. Miyakawa and T. Yoshikawa

• Strategies for Mitigating TCAM Space Bottlenecks
  Abstract
  Transport networks satisfy requests to forward data in a given topology. At the level of a network element, forwarding decisions are defined by flows. To implement desired data properties during forwarding, a network operator imposes economic models by applying policies to flows. In real applications, the number of different policies is much smaller than the number of flows. In this work, we draw from our experience in classifier design for commercial systems and demonstrate how to share classifiers that represent policies between flows while still implementing them per flow per policy state. The resulting space saving is several orders of magnitude higher than any state-of-the art methods which reduce space of classifiers representation.

  Authors affliliation: Purdue University*, USA
  Steklov Math. Institute†, Russia
  Cisco Systems, USA

  K. Kogan*, S. Nikolenko†, P. Eugster* and E. Ruan
• The Buffer Size vs. Link Bandwidth Tradeoff in Lossless Networks
  Abstract
  

  Data center networks demand high bandwidth switches. These networks also sustain common incast scenarios, which require large switch buffers. Therefore, network and switch designers encounter a buffer-bandwidth tradeoff as follows. Large switch buffers allow absorbing larger incast workload. However, higher switch bandwidth allows both faster buffer draining and more link pausing, which reduces buffering demand for incast. As the two features compete for silicon resources and device power budget, modeling their relative impact on the network is critical.

  In this work our aim is to evaluate this buffer-bandwidth tradeoff. We analyze the worst case incast scenario in the lossless network and find by how much the buffer size can be reduced, while the link bandwidth is increased to stand in the same network performance. In addition, we analyze the multi-level incast cascade and support our findings by simulations. Our analysis shows that increasing bandwidth allows reducing the buffering demand by at least the same ratio. In particular, we show that the switch buffers can be omitted if the links bandwidth is doubled.

  Authors affliliation: Mellanox Technologies, Israel

  A. Shpiner, E. Zahavi and O. Rottenstreich

• Quasi Fat Trees for HPC Cloud and their Fault-Resilient Closed-Form Routing
  Abstract
  Fat trees are a common topology used by High Performance Clusters and Data Center Networks. We present Quasi Fat Tree (QFT), a new flavor of fat trees, that emerged in recent years as a dominant network topology. This topology is appealing to cluster designers because it offers better performance for many concurrent small jobs which may fit in its extended 3-hop host groups. In this paper, we formulate the graph structure of this new topology, and derive a closed-form and fault resilient contention-free routing algorithm for all global shift permutations. This routing is important for optimizing the run-time of large computing jobs which utilize MPI collectives. The algorithm is verified by running its implementation as an OpenSM routing engine on various sizes of QFT topologies.

  Authors affliliation: Technion, Israel

  E. Zahavi, I. Keslassy and A. Kolodny
• Reuse Distance Based Circuit Replacement in Silicon Photonic Interconnection Networks for HPC
  Abstract Optical interconnects, which support the transport of large bandwidths over warehouse-scale distance, can help to further scale data-movement capabilities in high performance computing (HPC) platforms. However, due to the circuit switching nature of optical systems and additional peculiarities, such as sensitivity to temperature and the need for wavelength channel locking, optical links generally show longer link initialization delays. These delays are a major obstacle in exploiting the high bandwidth of optics for application speedups, especially when low-latency remote direct memory access (RDMA) is required or small messages are used.
  These limitations can be overcome by maintaining a set of frequently used optical circuits based on the temporal locality of the application and by maximizing the number of reuses to amortize initialization overheads. However, since circuits cannot be simultaneously maintained between all source-destination pairs, the set of selected circuits must be carefully managed. This paper applies techniques inspired by cache optimizations to intelligently manage circuit resources with the goal of maximizing the circuit successful 'hit' rate. We propose the concept of "circuit reuse distance" and design circuit replacement policies based on this metric. We profile the reuse distance based on a group of representative HPC applications with different communications patterns and show the potential to amortize circuit setup delay over multiple circuit requests. We then develop a Markov transition matrix based reuse distance predictor and two circuit replacement policies. The proposed predictor provides significantly higher accuracy than traditional maximum likelihood prediction and the two replacement policies are shown to effectively increase the hit rate compared to the Least Recently Used policy. We further investigate the tradeoffs between the realized hit rate and energy consumption. Finally, the feasibility of the proposed concept is experimentally demonstrated using silicon photonic devices in an FPGA-controlled network testbed.

  Authors affliliation: Columbia University*, USA
  Cornell University†,USA
  University of Delaware, USA

  K. Wen*, D. Calhoun*, S. Rumley*, L. Zhu*, Y. Liu†, L.-W. Luo, R. Ding†, T. Baehr-Jones, M. Hochberg, M. Lipson† and K. Bergman*
• The Tofu Interconnect 2
  Abstract
  The Tofu Interconnect 2 (Tofu2) is a system interconnect designed for the successor model of the FUJITSU Supercomputer PRIMEHPC FX10. Tofu2 uses a 25 Gbps transmission technology that is about two times faster than those of existing HPC interconnects, and uses optical transceivers in an extremely high ratio. Despite the major change of physical transport medium, Tofu2 inherits and enhances the features of Tofu1. This paper describes the specifications including frame format, implementation, and preliminary evaluation results of Tofu2. The effective throughput of Put transfer was evaluated to be 11.46 GB/s that was about 92% link efficiency. The total throughput of simultaneous 4-way transfer was evaluated to be 45.82 GB/s. Tofu2 reduced one-way communication latency by about 0.2 usec. The elimination of the host bus contributed about 60 nsec of reduction, and the rest of the reduction had been derived from the cache injection technique.

  Authors affliliation: Fujitsu Limited, Japan

  Y. Ajima, T. Inoue, S. Hiramoto, S. Ando, M. Maeda, T. Yoshikawa, K. Hosoe and T. Shimizu

• Traffic Optimization in Multi-Layered WANs using SDN
  Abstract
  Wide area networks (WAN) forward traffic through a mix of packet and optical data planes, composed by a variety of devices from different vendors. Multiple forwarding technologies and encapsulation methods are used for each data plane (e.g. IP, MPLS, ATM, SONET, Wavelength Switching). Despite standards defined, the control planes of these devices are usually not interoperable, and different technologies are used to manage each forwarding segment independently (e.g. OpenFlow, TL-1, GMPLS). The result is lack of coordination between layers and inefficient resource usage. In this paper we discuss the design and implementation of a system that uses unmodified OpenFlow to optimize network utilization across layers, enabling practical bandwidth virtualization. We discuss strategies for scalable traffic monitoring and to minimize losses on route updates across layers. We explore two use cases that benefit from multi-layer bandwidth on demand provisioning. A prototype of the system was built open using a traditional circuit reservation application and an unmodified SDN controller, and its evaluation was per-formed on a multi-vendor testbed.

  Authors affliliation: UC San Diego*, USA
  Snet/Lawrence Berkeley Lab†, USA
  Infinera Corporation, USA

  H. Rodrigues*, I. Monga†, A. Sadasivarao, S. Syed, C. Guok†, E. Pouyoul†, C. Liou and T. Rosing*
• IBRMP: a Reliable Multicast Protocol for InfiniBand
  Abstract
  Modern distributed applications in high-performance computing (HPC) fields often need to disseminate data efficiently from one cluster to an arbitrary number of others by using multicast techniques. InfiniBand, with its high throughput, low latency and low overhead communications, has been increasingly adopted as an HPC cluster interconnection. Although InfiniBand hardware multicast is efficient and scaleable, it is based on Unreliable Datagrams (UD) which cannot guarantee reliable data distribution. This makes InfiniBand multicast not the best fit for modern distributed applications. This paper presents the design and implementation of a reliable multicast protocol for InfiniBand (IBRMP). IBRMP is based on InfiniBand unreliable hardware multicast, and utilizes InfiniBand Reliable Connection (RC) to guarantee data delivery. According to our experiments, IBRMP takes full advantage of InfiniBand multicast which reduces communication traffic significantly. In our testing environment, using IBRMP is up to five times faster than using only RC to disseminate data among a group of receivers.

  Authors affliliation: University of New Hampshire, USA

  Q. Liu and R. Russell

Srini Seetharaman & Anirudh Ramachandran, Deutsche Telekom

John Lockwood & Imran Khan, Algo-Logic

D. K. Panda & Xiaoyi Lu, Ohio State University

Luigi Rizzo, Universita` di Pisa