NFSv4 O_DIRECT with 3.16.2 kernel issue (qemu kvm cache=none / RHEL5 guest)

Discussion:

Zdenek Kaspar

2014-09-12 16:23:26 UTC

Hello everyone, got this specific problem with RHEL5 guest under KVM
hypervisor.

client/server kernel version: 3.16.2

/etc/exports: rw,async,no_subtree_check,no_root_squash
/proc/mounts: nfs4 rw,relatime,vers=4.0,rsize=524288,wsize=524288,
namlen=255,hard,proto=tcp,port=0,
timeo=600,retrans=2,sec=sys,clientaddr=xxx.xxx.xxx.xxx,
local_lock=none,addr=xxx.xxx.xxx.yyy 0 0

CONFIG_NFS_FS=m
# CONFIG_NFS_V2 is not set
CONFIG_NFS_V3=m
# CONFIG_NFS_V3_ACL is not set
CONFIG_NFS_V4=m
# CONFIG_NFS_SWAP is not set
# CONFIG_NFS_V4_1 is not set
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
CONFIG_NFSD_V4=y
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_NFS_COMMON=y

CentOS-5.10-i386-bin-1of8.iso -- did not test everything on centos,
behaved same as RHEL5. So let's use it as VM reproducer:

dd if=/dev/zero of=kvm-test1.img bs=1M count=2048

start VM using:
file=kvm-test1.img,cache=writeback,aio=native,if=virtio,boot=on

create custom layout: /dev/vda1 / ext3 max size primary
unselect Desktop-Gnome, select Customize now
use only: Base System/Base and install it..

[***@test1 ~]# find / >/dev/null
everything works as expected with writeback

shutdown, backup kvm-test1.img (fs will explode)
start VM using:
file=kvm-test1.img,cache=none,aio=native,if=virtio,boot=on

[***@test1 ~]# find / >/dev/null
and check dmesg, reboot etc, ie:
EXT3-fs error (device vda1): htree_dirblock_to_tree: bad entry in
directory #460317: rec_len is smaller than minimal - offset=0, inode=0,
rec_len=0, name_len=0
EXT3-fs error (device vda1): htree_dirblock_to_tree: bad entry in
directory #98503: rec_len is smaller than minimal - offset=0, inode=0,
rec_len=0, name_len=0

Tested locally on ext4, xfs - WORK
Tested on NFSv4 share with ext4, xfs - FAIL
Tested NFSv4 share sync, async - FAIL
Tested on NFSv3 share with xfs - WORK
KVM with qemu rhel-6.5, 2.1.1 - SAME RESULTS

NOTE: it seems just the rhel5 guest triggers it easily..

HTH, Z.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-***@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Trond Myklebust

2014-09-12 16:31:29 UTC

Permalink

Post by Zdenek Kaspar
Hello everyone, got this specific problem with RHEL5 guest under KVM
hypervisor.
client/server kernel version: 3.16.2
/etc/exports: rw,async,no_subtree_check,no_root_squash
/proc/mounts: nfs4 rw,relatime,vers=4.0,rsize=524288,wsize=524288,
namlen=255,hard,proto=tcp,port=0,
timeo=600,retrans=2,sec=sys,clientaddr=xxx.xxx.xxx.xxx,
local_lock=none,addr=xxx.xxx.xxx.yyy 0 0
CONFIG_NFS_FS=m
# CONFIG_NFS_V2 is not set
CONFIG_NFS_V3=m
# CONFIG_NFS_V3_ACL is not set
CONFIG_NFS_V4=m
# CONFIG_NFS_SWAP is not set
# CONFIG_NFS_V4_1 is not set
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
CONFIG_NFSD_V4=y
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_NFS_COMMON=y
CentOS-5.10-i386-bin-1of8.iso -- did not test everything on centos,
dd if=/dev/zero of=kvm-test1.img bs=1M count=2048
file=kvm-test1.img,cache=writeback,aio=native,if=virtio,boot=on
create custom layout: /dev/vda1 / ext3 max size primary
unselect Desktop-Gnome, select Customize now
use only: Base System/Base and install it..
everything works as expected with writeback
shutdown, backup kvm-test1.img (fs will explode)
file=kvm-test1.img,cache=none,aio=native,if=virtio,boot=on
EXT3-fs error (device vda1): htree_dirblock_to_tree: bad entry in
directory #460317: rec_len is smaller than minimal - offset=0, inode=0,
rec_len=0, name_len=0
EXT3-fs error (device vda1): htree_dirblock_to_tree: bad entry in
directory #98503: rec_len is smaller than minimal - offset=0, inode=0,
rec_len=0, name_len=0
Tested locally on ext4, xfs - WORK
Tested on NFSv4 share with ext4, xfs - FAIL
Tested NFSv4 share sync, async - FAIL
Tested on NFSv3 share with xfs - WORK
KVM with qemu rhel-6.5, 2.1.1 - SAME RESULTS
NOTE: it seems just the rhel5 guest triggers it easily..
HTH, Z.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

Don't use 3.16.x kernels with NFS yet. There are known bugs in the
read/write code that need a stable backport of a set of fixes that are
in 3.17-rc.
--
Trond Myklebust

Linux NFS client maintainer, PrimaryData

trond.myklebust-7I+n7zu2hftEKMMhf/***@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-***@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Zdenek Kaspar

2014-10-06 15:18:25 UTC

Permalink

Post by Trond Myklebust

Don't use 3.16.x kernels with NFS yet. There are known bugs in the
read/write code that need a stable backport of a set of fixes that are
in 3.17-rc.

Hi Trond, noticed big nfs update that got into -stable, running 3.16.4
on both sides and problem is fixed, thx!

Z.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-***@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Trond Myklebust

2014-10-06 15:19:48 UTC

Permalink

Post by Zdenek Kaspar

Post by Trond Myklebust

Don't use 3.16.x kernels with NFS yet. There are known bugs in the
read/write code that need a stable backport of a set of fixes that are
in 3.17-rc.

Hi Trond, noticed big nfs update that got into -stable, running 3.16.4
on both sides and problem is fixed, thx!
Z.

Cool! Thanks for having tested and for confirming.

Cheers
Trond
--
Trond Myklebust

Linux NFS client maintainer, PrimaryData

trond.myklebust-7I+n7zu2hftEKMMhf/***@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-***@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html