|Representing Chelsio’s second‐generation RDMA (Remote Direct Memory Access) protocol support (Internet Wide Area RDMA Protocol – iWARP ), Chelsio’s Terminator series (T4 and T5) adapters have been field proven in numerous large, 100+ node clusters, including a 1300‐node cluster at Purdue University. For Linux, Chelsio supports Message Passing Interface (MPI) through integration with the OpenFabrics Enterprise Distribution (OFED), which has included Terminator 3 drivers since release 1.2. The T4 design reduces RDMA latency from T3’s already low 6 µs to about 2 µs. Furthermore the T5 design reduces the latency even more to around 1.6 µs. Various benchmark reports have been published to substantiate T4′s lead. This superior scaling with node count suggests that T4 offers latencies comparable to InfiniBand QDR in real‐world applications.
RoCE FAQ: The pitfalls in RoCE answered with respect to iWARP.
RoCE, or InfiniBand over Ethernet, raises many questions when practical deployment issues and limitations are encountered, and the answers are cause of concern to potential users, most of the time.
RoCE: The Fine Print
The promise of RoCE is to bring RDMA’s benefits to Ethernet. In the interest of truth in advertising, here is the missing fine print.
RoCE: The Grand Experiment
Upon closer examination, the CE component of the name is revealed to be a misnomer at best, since in a dedicated fabric, the CE suite of protocols (also called DCB) effectively boils down to Ethernet’s PAUSE.
Lowest UDP, TCP, and RDMA Over Ethernet Latency
In the HPC Linux for Wall Street conference,NYC today, Chelsio demonstrated a user-mode UDP and TCP latency of 1.6 µs and 2.0 µs respectively. Using its WireDirect software suite, both numbers represented industry record performance. The software provides direct network access to user space and is binary compatible with existing TCP and UDP sockets applications. User-mode UDP showed 3 million messages/second with excellent jitter profile and no dropped packets. Similarly, user-mode TCP demonstrated 2.3 million messages/second with nearly zero packet jitter, thanks to the use of T5’s offload engine. These preliminary results are expected to improve before general availability.
40G SMB Direct RDMA Over Ethernet For Windows Server 2012
Chelsio Communications, a leading provider of High Speed Ethernet Unified Wire adapters and ASICs, today announced that it will demonstrate 40Gb SMB performance on its new T5 ASIC this week at SNW Spring conference in Orlando, FL. The demonstration will show Microsoft’s SMB Direct running at line-rate 40Gb using iWARP. This will be the first demonstration of Chelsio’s T5 40G storage technology – a converged interconnect solution that simultaneously supports all of the networking, cluster and storage protocols. Chelsio offers a complete suite of drivers for Windows Server 2012, including NDIS, Network Direct for HPC applications, Network Direct Kernel for system services, iSCSI, FCoE and Hyper-V support for virtualized environments. This constitutes as one of the most comprehensive server adapter solutions that can unleash the full value of a Windows Server installation.
Gnodal And Chelsio Complete 10 Gigabit Ethernet Switch Server Adapter Testing
Today at SC12, Gnodal Limited and Chelsio announced industry leading benchmarks of MPI performance using the OSU MPI stack in a test with the Gnodal GS7200 10/40GbE switch and Chelsio T420-LL-CR adapters. Gnodal Limited is a networking company delivering industry leading performance, port density and low power to the high-performance data center with GS-Series switches.
LAMMPS, LS-DYNA, HPL, and WRF on iWARP vs. InfiniBand FDR
The use of InfiniBand as interconnect technology for HPC applications has been increasing over the past few years, replacing the aging Gigabit Ethernet as the most commonly used fabric. The main reason for preferring IB over 10Gbps Ethernet is it’s native support for RDMA, a technology that forms the basis for high performance MPI implementations. Today, a mature competitive RDMA solution over Ethernet – the iWARP protocol – is available and enables MPI applications to run unmodified over the familiar and preferred Ethernet technology. Offering the same API to applications and inboxed within the same middleware distributions, the technology can be dropped in seamlessly in place of the esoteric fabric. While current solutions are 10Gbps Ethernet-based, higher speed 40Gbps and 100Gbps implementations are slated for imminent availability. Nevertheless, as this paper shows with real application benchmarks, iWARP today offers competitive application level performance at 10Gbps against the latest FDR IB speeds.
InfiniBand Migration to iWARP
With the advent of 40GbE, and the arrival of 100GbE, Ethernet today can match or exceed InfiniBand in raw speed. Coupled with mature iWARP implementations, this sets the stage for migrating compute clusters from legacy IB networks to Ethernet, without any performance penalty, while realizing all the economies of scale that an all‐Ethernet environment allows.
InfiniBand’s Fifteen Minutes
Finally, a key differentiator for IB and the main reason for its recent resurgence is the RDMA communication interface it provides. It allows very efficient communication, where most of the data transfer is handled silently by the adapter, without the involvement of the main CPU. Thus, it frees up the cycles for the host system to process useful application workloads. In the datacenter age, at a time where system efficiency and power savings are critical metrics, increased efficiency is directly translated into dollars – both in terms of CAPEX and OPEX. Although making use of RDMA requires rewriting of applications, the gained efficiencies offer sufficient return on investment in areas such as HPC, storage system back-end and some datacenter and cloud applications.
iWARP SMB Direct Technology Brief
Using Chelsio’s T4 Unified Wire Network Adapters with industry leading iWARP, Chelsio enables Microsoft Windows Server 2012 to deliver superior performance in the areas of high bandwidth and low CPU utilization rivaling that of competing technologies.
iWARP Benchmarks with Arista Switch
Arastra’s switch along with Chelsio’s adapter provides a high throughput, low latency 10 Gigabit Ethernet based solution. The switch delivers an outstanding balance of performance and value with key data center class features. The latency and throughput performance shown in this report demonstrate that 10 Gigabit Ethernet is well suited for operation in demanding clustering applications.
Low Latency for High Frequency Trading
Chelsio’s T420-LL-CR adapter achieves all the requirements to make it ideal for low latency High Frequency Trading (HFT) operations. At the same time, when combined with iWARP, enabling NFSRDMA, LustreRDMA and similar protocols, the adapter makes for an ideal Unified Target adapter, simultaneously processing iSCSI, FCoE, TOE, NFSRDMA, LustreRDMA, CIFS and NFS traffic.
A Rocky Road for RoCE
By throwing overboard critical pieces of the IB and TCP stacks which provide stability and scalability, RoCE shines at simple micro-benchmarks in back-to-back or similarly limited deployment scenarios. However, it stands to fail in large clustered application performance, where all its limitations would be exposed. Effectively, the protocol represents an attempt by InfiniBand vendors at enticing the customer with a good Ethernet clustering benchmark story, but switching to selling InfiniBand gear in the end.
Ultra Low Latency Data Center Switches with iWARP NICs
For years, InfiniBand was the dominant interconnect technology for HPC applications, but it has now been eclipsed by Ethernet as the preferred networking protocol where scalability and ultralow latency are required. Juniper Networks’ QFX3500 Switch is a high-performance, ultralow latency, 10GbE switch specifically designed to address a wide range of demanding deployment scenarios such as traditional data centers, virtualized data centers, high-performance computing, network attached storage, converged server I/O and cloud computing.
HPC Converging on Low Latency iWARP
HPC cluster architectures are moving away from proprietary and expensive networking technologies towards Ethernet as the performance/latency of TCP/IP continues to lead the way. InfiniBand, the once-dominant interconnect technology for HPC applications leveraging MPI and RDMA, has now been supplanted as the preferred networking protocol in these environments.
High Frequency Trading
HFT has transformed the investment landscape, now accounting for more than two thirds of all current trading volume. As the traffic volumes and complexity has grown, so has the consequences of inefficiencies in the network architecture.