RoCE (RDMA over Converged Ethernet) raises many questions when practical deployment issues and limitations are encountered. The answers to the questions arising on RoCE are almost always cause of concern to potential users.
The truth about the capabilities and limitations of RoCE hasn’t exactly been forthcoming, leaving customers interested in RDMA with many unanswered questions. A list of answers to those frequently asked questions and comparison of RoCE Vs. iWARP are compiled and explained here:
Is RoCE the standard RDMA over Ethernet protocol?
Does RoCE lower CPU utilization?
Does RoCE reduce memory copies?
Does RoCE allow user-space I/O?
Is RoCE an alternative to InfiniBand?
Is RoCE more efficient than iWARP?
Is RoCE easy to deploy and use?
Does RoCE take advantage of Ethernet economies of scale?
Does RoCE inter-operate with switches from different vendors?
Can RoCE share a channel with other traffic?
Does making QoS configuration changes to a switch affect RoCE operation?
Does RoCE restrict QoS traffic marking configuration?
Does RoCE scale?
Does RoCE real application performance match micro-benchmarks?
Does RoCE operate over long distance links?
Does RoCE operate over WAN links or cross subnet boundaries?
Is RoCE routable?
Can I use standard traffic management and monitoring tools for RoCE ?
Can I configure RoCE congestion management to suit my environment?
The industry standard for RDMA over Ethernet is iWARP, which uses the familiar TCP/IP stack as foundation. High performance iWARP implementations are available and compete directly with InfiniBand in real application benchmarks. iWARP:
- Resilient RoCE: Misconceptions vs. Reality
- Adventures In RDMA
- Resilient RoCEv4: The Experiment Continues
- High Performance S2D with Chelsio 40GbE
- RoCE Exposed
- RoCE Fails to Scale
- RoCE – Plug and Debug
- The Case Against iWARP
- RoCE is Dead, Long Live RoIP?
- RoCE at a Crossroads
- RoCE: The Missing Fine Print
- RoCE: Autopsy of an Experiment
- A Rocky Road for RoCE
- RFC 5040 – A Remote Direct Memory Access Protocol Specification
- RFC5041 – Direct Data Placement over Reliable Transports
- OpenFabrics Enterprise Alliance
- Priority Flow Control – Building a Reliable Solution, Cisco Systems
- LAMMPS, LS-DYNA and LINPACK on RoCE vs. iWARP
- RDMA over Converged Ethernet, A Personal Obsession
- GPUDirect over 40GbE iWARP RDMA vs RoCE