On 05/02/18 14:52, J. Bruce Fields wrote:
> Yet another poor NFSv3 performance issue. If I do a "ls
-lR" of a certain
> NFS mounted directory over a slow link (NFS over Openvpn over FTTP
> 80/20Mbps), just after mounting the file system (default NFSv4 mount with
> async), it takes about 9 seconds. If I run the same "ls -lR" again, just
> after, it takes about 60 seconds.
A wireshark trace might help.
Also, is it possible some process is writing while this is happening?
--b.
Ok, I have made some wireshark traces and put these at:
https://www.beam.ltd.uk/files/files//nfs/
There are other processing running obviously, but nothing that should be
doing anything that should really affect this.
As a naive input, it looks like the client is using a cache but checking
the update times of each file individually using GETATTR. As it is using
a simple GETATTR per file in each directory the latency of these RPC
calls is mounting up. I guess it would be possible to check the cache
status of all files in a dir at once with one call that would allow this
to be faster when a full readdir is in progress, like a "GETATTR_DIR
<dir>" RPC call. The overhead of the extra data would probably not
affect a single file check cache time as latency rather than amount of
data is the killer.
> So much for caching ! I have noticed
> Makefile based builds (over Ethernet 1Gbps) taking a long time with a second
> or so between each directory, I think this maybe why.
>
> Listing the directory using a NFSv3 mount takes 67 seconds on the first
> mount and about the same on subsequent ones. No noticeable caching (default
> mount options with async), At least NFSv4 is fast the first time !
>
> NFSv4 directory reads after mount:
>
> No. Time Source Destination Protocol Length Info
> 667 4.560833210 192.168.202.2 192.168.201.1 NFS 304
> V4 Call (Reply In 672) READDIR FH: 0xde55a546
> 668 4.582809439 192.168.201.1 192.168.202.2 TCP 1405
> 2049 → 679 [ACK] Seq=304477 Ack=45901 Win=1452 Len=1337 TSval=2646321616
> TSecr=913651354 [TCP segment of a reassembled PDU]
> 669 4.582986377 192.168.201.1 192.168.202.2 TCP 1405
> 2049 → 679 [ACK] Seq=305814 Ack=45901 Win=1452 Len=1337 TSval=2646321616
> TSecr=913651354 [TCP segment of a reassembled PDU]
> 670 4.583003805 192.168.202.2 192.168.201.1 TCP 68
> 679 → 2049 [ACK] Seq=45901 Ack=307151 Win=1444 Len=0 TSval=913651376
> TSecr=2646321616
> 671 4.583265423 192.168.201.1 192.168.202.2 TCP 1405
> 2049 → 679 [ACK] Seq=307151 Ack=45901 Win=1452 Len=1337 TSval=2646321616
> TSecr=913651354 [TCP segment of a reassembled PDU]
> 672 4.583280603 192.168.201.1 192.168.202.2 NFS 289
> V4 Reply (Call In 667) READDIR
> 673 4.583291818 192.168.202.2 192.168.201.1 TCP 68
> 679 → 2049 [ACK] Seq=45901 Ack=308709 Win=1444 Len=0 TSval=913651377
> TSecr=2646321616
> 674 4.583819172 192.168.202.2 192.168.201.1 NFS 280
> V4 Call (Reply In 675) GETATTR FH: 0xb91bfde7
> 675 4.605389953 192.168.201.1 192.168.202.2 NFS 312
> V4 Reply (Call In 674) GETATTR
> 676 4.605491075 192.168.202.2 192.168.201.1 NFS 288
> V4 Call (Reply In 677) ACCESS FH: 0xb91bfde7, [Check: RD LU MD XT DL]
> 677 4.626848306 192.168.201.1 192.168.202.2 NFS 240
> V4 Reply (Call In 676) ACCESS, [Allowed: RD LU MD XT DL]
> 678 4.626993773 192.168.202.2 192.168.201.1 NFS 304
> V4 Call (Reply In 679) READDIR FH: 0xb91bfde7
> 679 4.649330354 192.168.201.1 192.168.202.2 NFS 2408
> V4 Reply (Call In 678) READDIR
> 680 4.649380840 192.168.202.2 192.168.201.1 TCP 68
> 679 → 2049 [ACK] Seq=46569 Ack=311465 Win=1444 Len=0 TSval=913651443
> TSecr=2646321683
> 681 4.649716746 192.168.202.2 192.168.201.1 NFS 280
> V4 Call (Reply In 682) GETATTR FH: 0xb6d01f2a
> 682 4.671167708 192.168.201.1 192.168.202.2 NFS 312
> V4 Reply (Call In 681) GETATTR
> 683 4.671281003 192.168.202.2 192.168.201.1 NFS 288
> V4 Call (Reply In 684) ACCESS FH: 0xb6d01f2a, [Check: RD LU MD XT DL]
> 684 4.692647455 192.168.201.1 192.168.202.2 NFS 240
> V4 Reply (Call In 683) ACCESS, [Allowed: RD LU MD XT DL]
> 685 4.692825251 192.168.202.2 192.168.201.1 NFS 304
> V4 Call (Reply In 690) READDIR FH: 0xb6d01f2a
> 686 4.715060586 192.168.201.1 192.168.202.2 TCP 1405
> 2049 → 679 [ACK] Seq=311881 Ack=47237 Win=1452 Len=1337 TSval=2646321748
> TSecr=913651486 [TCP segment of a reassembled PDU]
> 687 4.715199557 192.168.201.1 192.168.202.2 TCP 1405
> 2049 → 679 [ACK] Seq=313218 Ack=47237 Win=1452 Len=1337 TSval=2646321748
> TSecr=913651486 [TCP segment of a reassembled PDU]
> 688 4.715215055 192.168.202.2 192.168.201.1 TCP 68
> 679 → 2049 [ACK] Seq=47237 Ack=314555 Win=1444 Len=0 TSval=913651509
> TSecr=2646321748
> 689 4.715524465 192.168.201.1 192.168.202.2 TCP 1405
> 2049 → 679 [ACK] Seq=314555 Ack=47237 Win=1452 Len=1337 TSval=2646321749
> TSecr=913651486 [TCP segment of a reassembled PDU]
> 690 4.715911571 192.168.201.1 192.168.202.2 NFS 1449
> V4 Reply (Call In 685) READDIR
>
> NFS directory reads later:
>
> No. Time Source Destination Protocol Length Info
> 664 9.485593049 192.168.202.2 192.168.201.1 NFS 304
> V4 Call (Reply In 669) READDIR FH: 0x1933e99e
> 665 9.507596250 192.168.201.1 192.168.202.2 TCP 1405
> 2049 → 788 [ACK] Seq=127921 Ack=65730 Win=3076 Len=1337 TSval=2645776572
> TSecr=913106316 [TCP segment of a reassembled PDU]
> 666 9.507717425 192.168.201.1 192.168.202.2 TCP 1405
> 2049 → 788 [ACK] Seq=129258 Ack=65730 Win=3076 Len=1337 TSval=2645776572
> TSecr=913106316 [TCP segment of a reassembled PDU]
> 667 9.507733352 192.168.202.2 192.168.201.1 TCP 68
> 788 → 2049 [ACK] Seq=65730 Ack=130595 Win=1444 Len=0 TSval=913106338
> TSecr=2645776572
> 668 9.507987020 192.168.201.1 192.168.202.2 TCP 1405
> 2049 → 788 [ACK] Seq=130595 Ack=65730 Win=3076 Len=1337 TSval=2645776572
> TSecr=913106316 [TCP segment of a reassembled PDU]
> 669 9.508456847 192.168.201.1 192.168.202.2 NFS 989
> V4 Reply (Call In 664) READDIR
> 670 9.508472149 192.168.202.2 192.168.201.1 TCP 68
> 788 → 2049 [ACK] Seq=65730 Ack=132853 Win=1444 Len=0 TSval=913106338
> TSecr=2645776572
> 671 9.508880627 192.168.202.2 192.168.201.1 NFS 280
> V4 Call (Reply In 672) GETATTR FH: 0x7e9e8300
> 672 9.530375865 192.168.201.1 192.168.202.2 NFS 312
> V4 Reply (Call In 671) GETATTR
> 673 9.530564317 192.168.202.2 192.168.201.1 NFS 280
> V4 Call (Reply In 674) GETATTR FH: 0xcb837ac9
> 674 9.551906321 192.168.201.1 192.168.202.2 NFS 312
> V4 Reply (Call In 673) GETATTR
> 675 9.552064038 192.168.202.2 192.168.201.1 NFS 280
> V4 Call (Reply In 676) GETATTR FH: 0xbf951d32
> 676 9.574210528 192.168.201.1 192.168.202.2 NFS 312
> V4 Reply (Call In 675) GETATTR
> 677 9.574334117 192.168.202.2 192.168.201.1 NFS 280
> V4 Call (Reply In 678) GETATTR FH: 0xd3f3dc3e
> 678 9.595902902 192.168.201.1 192.168.202.2 NFS 312
> V4 Reply (Call In 677) GETATTR
> 679 9.596025484 192.168.202.2 192.168.201.1 NFS 280
> V4 Call (Reply In 680) GETATTR FH: 0xf534332a
> 680 9.617497794 192.168.201.1 192.168.202.2 NFS 312
> V4 Reply (Call In 679) GETATTR
> 681 9.617621218 192.168.202.2 192.168.201.1 NFS 280
> V4 Call (Reply In 682) GETATTR FH: 0xa7e5bbc5
> 682 9.639157371 192.168.201.1 192.168.202.2 NFS 312
> V4 Reply (Call In 681) GETATTR
> 683 9.639279098 192.168.202.2 192.168.201.1 NFS 280
> V4 Call (Reply In 684) GETATTR FH: 0xa8050515
> 684 9.660669335 192.168.201.1 192.168.202.2 NFS 312
> V4 Reply (Call In 683) GETATTR
> 685 9.660787725 192.168.202.2 192.168.201.1 NFS 304
> V4 Call (Reply In 686) READDIR FH: 0x7e9e8300
> 686 9.682612756 192.168.201.1 192.168.202.2 NFS 1472
> V4 Reply (Call In 685) READDIR
> 687 9.682646761 192.168.202.2 192.168.201.1 TCP 68
> 788 → 2049 [ACK] Seq=67450 Ack=135965 Win=1444 Len=0 TSval=913106513
> TSecr=2645776747
> 688 9.682906293 192.168.202.2 192.168.201.1 NFS 280
> V4 Call (Reply In 689) GETATTR FH: 0xa8050515
>
> Lots of GETATTR calls the second time around (each file ?).
>
> Really NFS is really broken performance wise these days and it "appears"
> that significant/huge improvements are possible.
>
> Anyone know what group/who is responsible for NFS protocol these days ?
>
> Also what group/who is responsible for the Linux kernel's implementation of
> it ?
>
--
Dr Terry Barnaby BEAM Ltd
Phone: +44 1454 324512 Northavon Business Center,
Email: terry(a)beam.ltd.uk Dean Rd, Yate
Web:
www.beam.ltd.uk Bristol, BS37 5NH, UK
BEAM Engineering: Instrumentation, Electronics/Software/Systems