OpenFabric Workshop, Sonoma, CA
May 1, 2007
Helen Chen, Noah Fischer, Matt Leininger, and Mitch Williams
Sandia National Laboratories
SAND 2007 –2137C
Outline
- Motivation and Background Information
- Previous Study –NFS over RDMA (SDR IB)
- This Study –extends the previous study to include DDR IB and 10 GbE iWARP
- The Testbed
- The Benchmark
- Results and Analysis
- Summary and Future Plans
- The parallel NFS research collaboration with Open Grid Computing
Motivation
The NFS RDMA Architecture
- NFS is a family of protocol layered over RPC
- XDR encodes RPC requests and results onto RPC transports
- NFS RDMA is implemented as a new RPC transport mechanism
- Selection of transport is an NFS mount option
|  |
Brent Callaghan, Theresa Lingutla-Raj, Alex Chiu, Peter Staubach, Omer Asad, “NFS over RDMA”, ACM SIGCOMM 2003Workshops, August 25-27 2003
The NFS Protocol Stack
Relevant OFA Stack
iWARP -RDMA protocol for TCP/IP
- iWARP is the suite of RDMA protocols for TCP/IP
- RNIC is a RDMA capable NIC with offloaded iWARP as well as TCP/IP (TOE)
- RNIC typically exposes NIC, TOE and iWARP interfaces to upper layer applications
|  |
Previous Study –NFS over IB
RDMA vs. TCP (IPoIB)
- NFS over RDMA can easily fill the 10 Gigabit (1GB) pipe
- Iozone -i 0 –i 1 –r 64k –s 2g
- Client Cache < 2g < Server CacheNFS
http://www.openfabrics.org/archives/sep2006devcon.htm
Write Performance Issues
- Linux’s NFS client implementation lack concurrency
- Pdflush activated when dirty page cache reached 34%
- Application I/O’s blocked while cached data being flushed
- Most visible with RDMA due to huge bandwidth capacity and CPU efficiency
- Being addressed by the Linux kernel community (Talpey, Tucker, et. al.)
- Support Multi-threading to take advantage of multi core hardware
- Tune Linux VM to flush data as often as network bandwidth allowsin the background
This study evaluates NFS RDMA transport vs. TCP, using Iozone reading from server cache
This Study –DDR IB and iWARP RPC Transport with SRP IB Storage
Key Testbed Hardware
- Mainboard: iWILL DK8ES
- Dual Core Dual Socket 2.4 Ghz AMD Opteron
- Dual Channel 400 Registered memory
- 4 GB on server
- 2 GB on client
- DDR IB Switch:Mellanox InfiniScale III 24-port switch
- DDR IB HCA: PCI-E Mellanox MT25204 InfiniHost III Lx
- 10 GbE Switch: Fujitsu XG700 CX4
- 10 GbE RNIC: PCI-E Chelsio Terminator 3
- SDP IB SRP Storage: DDN S2A 9550
Key Testbed Software
- RedHat Enterprise AS Release 4 Update 4
- Kernel: Linux 2.6.18.8
- NFS/RDMA update 7
- oneSIS used to boot all the nodes
- OpenFabric Enterprise Distribution 1.2 Beta
NFS Test Configuration
- One NFS server and one to four clients
- Ext2 filesystem built on IB SRP Storage at SDR
- TCP/IPoIB-UD (MTU 2048), TCP/IPoIB-CM (MTU 65520), and IB RDMA transport at DDR
- Host TCP/IP, TOE, and RNIC (iWARP) transport at 10GbE rate (MTU 9000)
- Clients ran IOZONE reading 128KB records
- Read 2GB file on all clients to avoid client-side cache effect and server-side disk I/O
To allow the evaluation of the NFS RDMA transport
- System resources monitored using “vmstat”at one second intervals
- All tests repeated 10 times
NFS Throughput
NFS Throughput Conclusion
- NFS over RDMA
- Can take advantage of the IB DDR pipe (theoretical maximum bandwidth 2.0GB)
- Throughput is limited by the 10GbE rate (theoretical maximum bandwidth 1.25GB)
- Both RDMA transport out performed their TCP counter part, most noticeable in IB
- NFS over TCP
- IPoIB-CM significantly better than IPoIB-UD
- Both the RNIC TOE and NIC performed surprisingly well
- Can also easily filled the 10GbE pipe
- A great all-in-one adapter
NFS CPU Efficiency
NFS CPU Efficiency Conclusion
- Host Efficiency is based on CPU per MB transferred
- IB RDMA and 10GbE iWARP delivered comparable CPU efficiency
- RDMA demonstrated better CPU performance than TCP
- Most significantly in IB
- Both TOE and host TCP performed extremely well, with TOE better than host stack
SRP Target and Initiator Configuration
- Target –DDN S2A9550
- 1 Controller, 1 SDR IB link
- 4 Power LUN’s each stripped across 4 Tiers (8 plus 1 Parity of 250GB SATAII disks)
- Block size = 4096
- Initiator –OFED 1.2 beta
- Increased maximum number of gather/scatter entries per I/O
- modprobe ib_srp srp_sg_tablesize (scatter and gather) =64
- Increased Filesystem read-ahead sector count to 1024
SRP Performance
- 1 to 4 concurrent sessions from 1 to 4 Initiators
“Iozone –i 0 –r 128k –s 8g –f /mnt/srp1/test”
8g > Initiator memory; measurement reflects SRP performance
SRP Performance Conclusion
- Good but still room for tuning
- Adjust maximum outstanding SCSI requests per LUN
- Increase maximum SCSI command payload size
- Evaluate Linux I/O Scheduling Algorithms
- QoS?
- etc…
Future Plans: The Need for pNFS
- Large number of concurrent requests from parallel applications
- Require parallelism in addition to RDMA
The pNFS Architecture
- pNFS extends NFSv4
- To allow out-of-band I/O
- A Standards-based scalable I/O solution
- Asymmetric, Out-of-band solutions offer scalability
- Control path (open/close) different from Data Path (read/write)
http://www3.ietf.org/proceedings/04nov/slides/nfsv4-8/pnfs-reqs-ietf61.ppt
The Sandia -Open Grid Computing Research Collaboration
 | - Based on the CITI implementation at UMICH
- Modified to stripe pNFS file data across RDMA enabled Linux NFSv3 Filers
- Open Source Linux environment
|
Future SRP Study
- Each Storage Server has to handle multiple independent large sequential writes and/or reads
- Concurrent sequential I/O requests from compute nodes turned into random large accesses on target
- A challenge for Parallel Filesystem and Storage vendors
Acknowledgment
- Tom Tucker from Open Grid Computing and Tom Talpey from Network Appliance for their in depth technical support for NFS/RDMA
- Chas Williams from NRL, Randy Kreiser from DDN, and Dror Goldenberg from Mellanox for their assistance in IB SRP
- Felix Marti from Chelsio and Steve Wise from Open Grid Computing for their expertise in iWARP
- Jim Brandt from Sandia for his technical input and review
Click here to download a pdf version of this page
