On 11/28/2011 03:39 PM, Mark Wagner wrote:
Hi Yaniv
Sorry for the delay in responding, I wanted to run this by our team to
make sure that you could benefit from their experiences as well. I have
included some responses from Ben inline (b>>>>).
-mark
Thanks, we've done (and continue to do) some of our own tests. No
concrete conclusions yet. See additional notes below.
On 11/20/2011 08:21 AM, Yaniv Kaul wrote:
> As we are getting some newer gear, I was wondering what is a better
> setup for network-based storage in our lab (all our hosts will have
> >1 interface, and our storages - either 4x1Gb or some with 10Gb).
>
> NFS:
> Am I correct to assume that NFS, from a single host, will not benefit
> much from bonding, when connected to a single storage domain (= a
> single mount). Isn't it using a single TCP connection? (which btw
> implies we might get better perf. with multiple mounts?).
>
b>>>> My own experience is that NFS like any IP app can benefit
greatly from bonding. The advantage of bonding is that you don't have
to manually balance application network traffic across NICs, and it
provides high-availability in event of port or cable failure. I've
seen up to 300 MB/s with a 4-way 1-Gbps bond with mode balance-alb (6)
or trunking (4) or balance-xor (2). Each mode has its limitations. I
have no experiencing bonding with 10-GbE but I see no reason why
10-GbE wouldn't work in principle. Modes 2 (balance by TCP port + IP
addr ) or 6 work best when a node is communicating with multiple other
nodes in parallel. Mode 4 requires that the network switch be
configured for trunking, but is the least restrictive. Mode 6 uses
ARP to load balance peer MAC addresses across NICs, but usually works
only within a single LAN.
- I haven't tested balance_tlb/alb, but I don't see how it'll help with
a single connection. I think the only viable solution is going 10Gb in
such case(s).
- Mode 4 with xmit_hash_policy set to layer3+4 is nice, but 'is not
fully 802.3ad compliant', so I'll try it out for fun, but am not happy
with it.
- Mode 2 with layer3+4 xmit_hash_policy - sounds interesting for iSCSI -
if I manage to get multiple connections (see below)
- (IIRC, mode 6 is not supported over bridges or something)
b>>>> Yes NFS is using a single connection from KVM host to NFS
server for a given block device, but in my experience with NetApp at
least, performance was pretty good even so, and it is possible to tune
TCP for higher-network-latency environments.
b>>> on the other hand, if you want to reserve network bandwidth for
particular block devices, not using bonding, or using 2 pairs of
bonded NICs could help.
> iSCSI:
> Am I better off configuring multipathing rather than bonding for
> iSCSI, for the same reason? Configure two IPs on two interfaces on
> the host, same on the storage? (which btw hints at why iSCSI from the
> QEMU level is an interesting feature).
> Should I 'cheat' and configure multiple IPs on the bonded interfaces
> on the storage side?
>
>
> For both I'm sure jumbo frames would be nice, and we will be using it
> partially in our labs.
>
b>>>> For ISCSI/TCP or NFS/TCP Jumbo frames are essential for bulk
transfer workloads Just be careful when using non-TCP protocols (e.g.
NFS/UDP) with jumbo frames.
Jumbo frames are not a good (enough) solution for two reasons:
1. They require the whole switch to be configured with support for it
(at least our Cisco switches), which is a bit limiting. We have done it
on a dedicated switch.
2. They only give ... +5% or so improvement? That's perhaps all I'll be
able to do for NFS, but I'm hoping to do more for iSCSI:
Our current idea is to create the storage domain's VG out of two LUNs -
on two different targets. We have to have two connections - one to each
target, and have those connections run in parallel (and NOT share a
single physical link). We have partially succeeded to do so, but are
still working on it.
BTW, our biggest performance improvement so far came from changing the
block size of the backing storage to 32K (instead of the default 8K)...
Y.
> TIA,
> Y.