Performance Difference Between Linux NFS Server and Netapp

[Hugh Caley]

I've been doing a little limited benchmarking of NFS performance, and
doing a comparison between Linux NFS servers and a Netapp. All tests
done over gigabit, clients are Linux (i386 and Opteron), OS is Fedora
Core 2,
kernel is 2.6.10-1.771_FC2smp, filesystem is Reiser. I basically copy a
205
megabyte file to the server three times and average the results. Sure,
limited,
but it seems to have real world relevance here.

The difference in performance is kind of startling. I had originally
put the Linux-based servers (with a Nexsan Atabeast backend) in as a
replacement for an older EMC Celerra and a few older Suns. On both the
Suns and Celerra I would see single-client performance topping out at
between 120 and 150 megabits per second.

The Linux setup (Supermicro Dual 2.6 Ghz Xeons with 4 gigabytes of RAM,
recently boosted to 8 gigabytes of RAM) regularly returns about 300
megabits per second. Given the difference in price between the Linux
boxes and the Celerra, everyone's pretty happy ;)

However, recently I've been trying the same testing between Linux
clients and a Netapp, and can achieve between 500 and 600 megabits per
second! Darned impressive.

I realize that the disk subsystem of the Netapp is going to stomp on my
Atabeast, but neither 300 nor 600 megabits is anywhere near the speed I
can get if I run the same test locally on the Linux server boxes, which
can regularly top 2000 megabits per second, so I really don't think that
the differences in disk architecture should make this much difference.

I guess my question is, are my Linux NFS servers seriously
underperforming? Could I expect to achieve better? I realize that my
"benchmark" is pretty limited, but the difference is pretty big. Since
the same client
on all the Linux boxen gets these results, I'm guessing that the client end
has little to do with it.

How does Netapp do it? Don't suppose they'd tell me ...

Hugh

--
Hugh Caley | Unix Systems Administrator | CIS
AFFYMETRIX INC. | 6550 Vallejo St. Ste 100 | Emeryville, CA 94608
Tel: [phone] | ***@affymetrix.com
--
"Brain-eating mutants are bad for business" - Battle Angel
Batmensch <***@plasmabat.com>



-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
core and dual graphics technology at this free one hour event hosted by HP,
AMD, and NVIDIA. To register visit http://www.hp.com/go/dualwebinar
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Roger Heflin]

Locally disk cache will cause the results to be junk,
locally the cache will be used, with NFS the cache
will not be used for writes.

With Suse 9 Enterprise and a disk subsystem that will
sustain rates of 125MiB/second I can get write rates of
95MiB/second and read rates of 115MiB/second, this is
with sustained over 12 minutes of writes.

2000mbit/second is a rate that no ide disk subsystem
will obtain, so double check your local performance,
if it is fast single disk it will get 50mb/second, if
you are using a built-in ata controller, with all channels
having fast disks you won't be able exceed 90mb/second
no matter what disk you use or how many or how you stripe
them, as the build-in mb controllers are not the best,
and this is on the more expensive dual-cpu mbs, it may
be worse on the single cpu machines.

You will need to use a test larger than the ram size of
the machine to get a realistic rate, at least 2x, and
probably 4x, and maybe both just to see how things fall
over.

Roger

-----Original Message-----
Sent: Monday, July 11, 2005 5:56 PM
Subject: [NFS] Performance Difference Between Linux NFS
Server and Netapp
I've been doing a little limited benchmarking of NFS
performance, and doing a comparison between Linux NFS servers
and a Netapp. All tests done over gigabit, clients are Linux
(i386 and Opteron), OS is Fedora Core 2, kernel is
2.6.10-1.771_FC2smp, filesystem is Reiser. I basically copy a
205
megabyte file to the server three times and average the
results. Sure, limited, but it seems to have real world
relevance here.
The difference in performance is kind of startling. I had
originally put the Linux-based servers (with a Nexsan
Atabeast backend) in as a replacement for an older EMC
Celerra and a few older Suns. On both the Suns and Celerra I
would see single-client performance topping out at between
120 and 150 megabits per second.
The Linux setup (Supermicro Dual 2.6 Ghz Xeons with 4
gigabytes of RAM, recently boosted to 8 gigabytes of RAM)
regularly returns about 300 megabits per second. Given the
difference in price between the Linux boxes and the Celerra,
everyone's pretty happy ;)
However, recently I've been trying the same testing between
Linux clients and a Netapp, and can achieve between 500 and
600 megabits per second! Darned impressive.
I realize that the disk subsystem of the Netapp is going to
stomp on my Atabeast, but neither 300 nor 600 megabits is
anywhere near the speed I can get if I run the same test
locally on the Linux server boxes, which can regularly top
2000 megabits per second, so I really don't think that the
differences in disk architecture should make this much difference.
I guess my question is, are my Linux NFS servers seriously
underperforming? Could I expect to achieve better? I
realize that my "benchmark" is pretty limited, but the
difference is pretty big. Since the same client on all the
Linux boxen gets these results, I'm guessing that the client
end has little to do with it.
How does Netapp do it? Don't suppose they'd tell me ...
Hugh
--
Hugh Caley | Unix Systems Administrator | CIS AFFYMETRIX INC.
| 6550 Vallejo St. Ste 100 | Emeryville, CA 94608
--
"Brain-eating mutants are bad for business" - Battle Angel
-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!'
webinar happening
July 14 at 8am PDT/11am EDT. We invite you to explore the
latest in dual
core and dual graphics technology at this free one hour event
hosted by HP,
AMD, and NVIDIA. To register visit http://www.hp.com/go/dualwebinar
_______________________________________________
https://lists.sourceforge.net/lists/listinfo/nfs

-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
core and dual graphics technology at this free one hour event hosted by HP,
AMD, and NVIDIA. To register visit http://www.hp.com/go/dualwebinar
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Joshua Baker-LePain]

Post by Roger Heflin
2000mbit/second is a rate that no ide disk subsystem
will obtain, so double check your local performance,

I have a 3ware based server that begs to differ with you -- over 150MB/s
writing, over 300MB/s reading. That's on a software RAID0 stripe across 2
12 port controllers, with each controller doing hardware RAID5.

Post by Roger Heflin
You will need to use a test larger than the ram size of
the machine to get a realistic rate, at least 2x, and
probably 4x, and maybe both just to see how things fall
over.

Absolutely.
--
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University


-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
core and dual graphics technology at this free one hour event hosted by HP,
AMD, and NVIDIA. To register visit http://www.hp.com/go/dualwebinar
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Hugh Caley]

The Atabeast is an ATA-raid array, but as far as the server knows it is
a 2 Gb fibre-attached SCSI disk. I get 800-900 Mb/sec sequential writes
using ext3; with Reiser or JFS it is much faster.

Regardless, I don't think this is very relevant. My only point was
that the disk subsystem is capable of much better performance than I'm
currently seeing over NFS. 300 Mb/sec seems slow to me when I have a
Dual-2.6 Ghz Linux box running kernel 2.6.10 (Fedora core 2) attached to
a 10 disk RAID5, and the box is doing nothing else but serving this
single NFS server.

Hugh

Post by Joshua Baker-LePain

Post by Roger Heflin
2000mbit/second is a rate that no ide disk subsystem
will obtain, so double check your local performance,

I have a 3ware based server that begs to differ with you -- over 150MB/s
writing, over 300MB/s reading. That's on a software RAID0 stripe across 2
12 port controllers, with each controller doing hardware RAID5.

Post by Roger Heflin
You will need to use a test larger than the ram size of
the machine to get a realistic rate, at least 2x, and
probably 4x, and maybe both just to see how things fall
over.

Absolutely.

--
"Take off your pants and get in the reptile tank." - Joe Rogan, Fear Factor
Hugh Caley ***@plasmabat.com



-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Hugh Caley]

Post by Hugh Caley
How does Netapp do it? Don't suppose they'd tell me ...

Quite simple really, you pay them loads of money, they
give part of it to engineers doing work on nfs tuning.
- battery backed cache with full os support,
...
- special filesystem which allows writing data on any
...

I actually don't think filesystem type is relevant to my question. I
tested this out pretty thoroughly with all the journaling filesystems
available from the Fedora Core 2 kernels (Ext3, Reiser, XFS, JFS), and
although Ext3 had by far the worst local performance, it's performance
over NFS with my simple sequential write test was exactly the same as
the others. I mean, I'm talking about a single sequential write to a
10-disk RAID5 attached with 2 Gb fibre channel; whichever file system I
use on the RAID should be a whole lot faster than over NFS, and indeed
they all are, Ext3 about 3 times, the others much more.

So, unless there are hooks between the filesystem and the NFSd that are
not obvious to a user (and the fact that changing filesystems doesn't
seem to make much difference seems to rule that out) and the fact that
the NFS client seems capable of much better performance when used
against a Netapp, it makes me wonder what's going on with the server
portion.

- Highly tuned nfs inplementation with dedicated operating system.
The bruteforce linux approach is quite good for the money spent,
but don't expect the features/performance of a way more expensive
enterprise solution.

A valid point, of course, but I don't think I'm actually expecting a
single NFSd to act like an expensive Netapp. I do think that wondering
why the Netapp is twice as fast for a sequential write is a valid
question, even if the OS and NFS server subsystem are free. I was kind
of hoping someone would just say "you're getting what you should expect
to get" or "wow, that's slow, try this and this and this".

And I did mention that moving to the Linux/NFS/Atabeast solution was
quite a bit better than what we had before ;) I'm not really
complaining, just wondering here.

Hugh
--
Whatsoever hath no fins nor scales in the waters, that shall be an abomination unto you. - Leviticus 11:9-12
http://godhatesshrimp.com
Hugh Caley ***@plasmabat.com



-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Chris Penney]

Post by Hugh Caley
A valid point, of course, but I don't think I'm actually expecting a
single NFSd to act like an expensive Netapp. I do think that wondering
why the Netapp is twice as fast for a sequential write is a valid
question, even if the OS and NFS server subsystem are free. I was kind
of hoping someone would just say "you're getting what you should expect
to get" or "wow, that's slow, try this and this and this".

You referenced that you were getting 300 megabits (or 37MB/s). I have
several SLES 9 nfs servers (using self compiled 2.6.11.5
<http://2.6.11.5>kernel) running on IBM x345 hardware (dual cpu pentum
4, 2gb ram, dual
qlogic hbas) connected to a single LSI storage array that presents four luns
(two from controller A and two from B). Each lun is 1TB and made from
hardware raid 8+1. Luns are merged together using device mapper.
It's not uncommon with my setup to get a sustained write speed of 75MB/s on
one of our SLES 9 compute systems (AMD Opterons) when doing a sequential
write of an 8GB file. With two systems writing at the same time I get
aggregate bandwidth better than 75MB/s (can't recall what it is).
I use tcp/nfs3 and for write testing I use 'iozone -c -e -s 8192m -i 0'. I
use 128 nfsds, export with 'rw,sync,no_subtree_check,no_root_squash' and add
the following to sysctl.conf:
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 65536 8388608
net.ipv4.tcp_mem = 8388608 8388608 8388608
On nfs clients (Sun, Linux, IRIX) I use the mount options:
nosuid,rw,bg,hard,intr,vers=3,proto=tcp,rsize=32768,wsize=32768. On AIX I
use the same options, but also add the critical 'combehind' (without it
writes of large files [ie. close to the size of physical mem] is just
horrid).
Chris

[Hugh Caley]

Thank you! Just the sort of thing I'm looking for ...

Hugh

Subject: Re: [NFS] Performance Difference Between Linux NFS Server and Netapp

Post by Hugh Caley
=20
A valid point, of course, but I don't think I'm actually expecting a
single NFSd to act like an expensive Netapp. I do think that wondering
why the Netapp is twice as fast for a sequential write is a valid
question, even if the OS and NFS server subsystem are free. I was kind
of hoping someone would just say "you're getting what you should expect
to get" or "wow, that's slow, try this and this and this".

You referenced that you were getting 300 megabits (or 37MB/s). I have=20
several SLES 9 nfs servers (using self compiled 2.6.11.5
<http://2.6.11.5>kernel) running on IBM x345 hardware (dual cpu pentum
4, 2gb ram, dual
qlogic hbas) connected to a single LSI storage array that presents four lun=
s=20 ...

--
Whatsoever hath no fins nor scales in the waters, that shall be an abomination unto you. - Leviticus 11:9-12
http://godhatesshrimp.com
Hugh Caley ***@plasmabat.com



-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

[Sten Spans]

Post by Hugh Caley
How does Netapp do it? Don't suppose they'd tell me ...

Quite simple really, you pay them loads of money, they
give part of it to engineers doing work on nfs tuning.

They generally have 3 performance advantages:

- battery backed cache with full os support,
which should make it perform better than battery backed
cache on normal raid cards. They use this cache for
journaling. I suppose one could try something similar
with pci-ram cards with battery and store the journal
on these.

- special filesystem which allows writing data on any
disk, this allows distributing the workload across the
available disks, more disks more performance
( if the fibrechannel can handle it ).

- Highly tuned nfs inplementation with dedicated operating system.

The bruteforce linux approach is quite good for the money spent,
but don't expect the features/performance of a way more expensive
enterprise solution.
--
Sten Spans

"There is a crack in everything, that's how the light gets in."
Leonard Cohen - Anthem


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - ***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs