'du' on XFS crashes my kernel
Hi all,
Recently my server has been experiencing crashes, and the system logs indicate that it was caused by XFS when running du command. The log content is as follows. [341534.127141] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [341534.127191] IPv6: ADDRCONF(NETDEV_CHANGE): vethdd5557d6: link becomes ready [341534.127251] cni0: port 4(vethdd5557d6) entered blocking state [341534.127252] cni0: port 4(vethdd5557d6) entered forwarding state [341546.280806] cni0: port 4(vethdd5557d6) entered disabled state [341546.338512] device vethdd5557d6 left promiscuous mode [341546.338521] cni0: port 4(vethdd5557d6) entered disabled state [341558.138857] usercopy: Kernel memory exposure attempt detected from SLUB object 'kmalloc-16' (offset 13, size 35)! [341558.149235] ------------[ cut here ]------------ [341558.149237] kernel BUG at mm/usercopy.c:102! [341558.153617] invalid opcode: 0000 [#1] SMP NOPTI [341558.158246] CPU: 53 PID: 50423 Comm: du Kdump: loaded Tainted: G W OE --------- -t - 4.18.0 #1 [341558.169650] Hardware name: ZTE R5300 G4X/R53G4X, BIOS 03.06.0400 09/01/2022 [341558.176711] RIP: 0010:usercopy_abort+0x74/0x76 [341558.181253] Code: 0f 45 c6 51 48 89 f9 48 c7 c2 3d 9c ea 85 41 52 48 c7 c6 c5 7b e9 85 48 c7 c7 08 9d ea 85 48 0f 45 f2 48 89 c2 e8 b9 33 e6 ff <0f> 0b 49 89 e8 31 c9 44 89 e2 31 f6 48 c7 c7 71 9c ea 85 e8 74 ff [341558.200144] RSP: 0018:ff48e500225a3cf8 EFLAGS: 00010246 [341558.205468] RAX: 0000000000000065 RBX: ff3bb03416f1c4fd RCX: 0000000000000000 [341558.212703] RDX: 0000000000000000 RSI: ff3bb07f7f156a08 RDI: ff3bb07f7f156a08 [341558.219937] RBP: 0000000000000023 R08: 0000000000037b2b R09: 0000000000aaaaaa [341558.227171] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000001 [341558.234398] R13: ff3bb03416f1c520 R14: 0000000000000023 R15: ff3bb03416f1c4fd [341558.241627] FS: 00007f1045872b88(0000) GS:ff3bb07f7f140000(0000) knlGS:0000000000000000 [341558.249820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [341558.255664] CR2: 0000558aae40aff0 CR3: 0000007ec20de005 CR4: 0000000000761ee0 [341558.262896] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [341558.270132] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [341558.277367] PKRU: 55555554 [341558.280165] Call Trace: [341558.282708] __check_heap_object+0xda/0x110 [341558.286991] __check_object_size+0xfa/0x181 [341558.291272] filldir64+0xbe/0x130 [341558.294727] xfs_dir2_sf_getdents.isra.8+0x130/0x230 [xfs] [341558.300331] xfs_readdir+0x15b/0x190 [xfs] [341558.304522] iterate_dir+0x13c/0x190 [341558.308194] ksys_getdents64+0x9c/0x130 [341558.312131] ? iterate_dir+0x190/0x190 [341558.315977] __x64_sys_getdents64+0x16/0x20 [341558.320260] do_syscall_64+0x5b/0x1b0 [341558.324020] entry_SYSCALL_64_after_hwframe+0x65/0xca [341558.329170] RIP: 0033:0x7f1045604401 [341558.332840] Code: 0f 05 eb 02 89 18 48 89 d0 5b c3 53 8b 47 14 49 89 f8 39 47 10 48 8d 77 20 7c 37 48 63 3f ba 00 08 00 00 b8 d9 00 00 00 0f 05 <85> c0 48 89 c3 7f 15 c1 e8 1f 74 3a 83 fb fe 74 35 f7 db e8 89 09 [341558.351717] RSP: 002b:00007fffa21af410 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9 [341558.359388] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1045604401 [341558.366622] RDX: 0000000000000800 RSI: 0000558aae40a0c0 RDI: 0000000000000005 [341558.373858] RBP: 0000558aad5bfc40 R08: 0000558aae40a0a0 R09: 0000000000000000 [341558.381095] R10: 0000000000000000 R11: 0000000000000246 R12: 0000558aae40a0a0 [341558.388334] R13: 0000558aad5bfc40 R14: 0000000000000000 R15: 0000000000000000 [341558.395568] Modules linked in: ceph(OE) udp_diag tcp_diag inet_diag xt_multiport veth vxlan ip6_udp_tunnel udp_tunnel nbd(OE) rbd(OE) libceph(OE) dns_resolver xt_statistic xt_nat nf_tables ip_set ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_mangle ip6t_MASQUERADE ip6table_filter ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables iptable_mangle xt_comment xt_mark xt_conntrack ipt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack binfmt_misc bonding esp6_offload esp6 esp4_offload esp4 mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel xfs kvm nvme irqbypass crct10dif_pclmul crc32_pclmul libcrc32c [341558.470406] ghash_clmulni_intel pcspkr nvme_core mei_me i2c_i801 joydev mei sg wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad br_netfilter bridge stp llc overlay ip_tables ext4 mbcache jbd2 sd_mod ast mlx5_core(OE) i2c_algo_bit mlxfw(OE) ttm tls(t) vfio_mdev(OE) drm_kms_helper vfio_iommu_type1 syscopyarea vfio sysfillrect ahci sysimgblt fb_sys_fops mdev(OE) libahci drm crc32c_intel libata mlx_compat(OE) I tried to use crash tool for analysis, and here are some information I have analyzed. It looks like XFS encountered a UAF problem causing a system bug. crash-8.0.2> bt -FFsx PID: 50423 TASK: ff3bb06b17418000 CPU: 53 COMMAND: "du" #0 [ff48e500225a3a80] machine_kexec+0x1be at ffffffff84e57f3e ff48e500225a3a88: 00002adfb820a900 ff3bb00000000000 ff48e500225a3a98: 0000000015001000 ff3bb00015001000 ff48e500225a3aa8: 0000000015000000 7ffefbffaa800800 ff48e500225a3ab8: dddc2adfb820a900 ff48e500225a3c48 ff48e500225a3ac8: ff48e500225a3ae0 ff48e500225a3c48 ff48e500225a3ad8: __crash_kexec+109 #1 [ff48e500225a3ad8] __crash_kexec+0x6d at ffffffff84f56bed ff48e500225a3ae0: [ff3bb03416f1c4fd:kmalloc-16] 0000000000000023 ff48e500225a3af0: [ff3bb03416f1c520:kmalloc-16] 0000000000000001 ff48e500225a3b00: 0000000000000023 [ff3bb03416f1c4fd:kmalloc-16] ff48e500225a3b10: 0000000000000001 0000000000000000 ff48e500225a3b20: 0000000000aaaaaa 0000000000037b2b ff48e500225a3b30: 0000000000000065 0000000000000000 ff48e500225a3b40: 0000000000000000 ff3bb07f7f156a08 ff48e500225a3b50: ff3bb07f7f156a08 ffffffffffffffff ff48e500225a3b60: usercopy_abort+116 0000000000000010 ff48e500225a3b70: 0000000000010246 ff48e500225a3cf8 ff48e500225a3b80: 0000000000000018 dddc2adfb820a900 ff48e500225a3b90: ff48e500225a3c48 000000000000000b ff48e500225a3ba0: crash_kexec+61 #2 [ff48e500225a3ba0] crash_kexec+0x3d at ffffffff84f57acd ff48e500225a3ba8: .LC0+142 0000000000000246 ff48e500225a3bb8: oops_end+189 #3 [ff48e500225a3bb8] oops_end+0xbd at ffffffff84e20e9d ff48e500225a3bc0: 0000000000000006 [ff3bb06b17418000:task_struct(3831:cri-containerd-1415d8f4b03b128efdfb1d94a9b92eab05336df4419a5b5e060bb67cb3cbcc66.scope)] ff48e500225a3bd0: 0000000000000000 do_trap+124 #4 [ff48e500225a3bd8] do_trap+0x7c at ffffffff84e1d6fc ff48e500225a3be0: .LC0+142 usercopy_abort+116 ff48e500225a3bf0: ff48e500225a3c48 0000000000000000 ff48e500225a3c00: 0000000000000000 0000000000000000 ff48e500225a3c10: 0000000000000000 0000000000000000 ff48e500225a3c20: do_invalid_op+54 #5 [ff48e500225a3c20] do_invalid_op+0x36 at ffffffff84e1dfc6 ff48e500225a3c28: usercopy_abort+116 0000000000000000 ff48e500225a3c38: 0000000000000000 invalid_op+20 #6 [ff48e500225a3c40] invalid_op+0x14 at ffffffff85800cc4 [exception RIP: usercopy_abort+116] RIP: ffffffff850b49b5 RSP: ff48e500225a3cf8 RFLAGS: 00010246 RAX: 0000000000000065 RBX: ff3bb03416f1c4fd RCX: 0000000000000000 RDX: 0000000000000000 RSI: ff3bb07f7f156a08 RDI: ff3bb07f7f156a08 RBP: 0000000000000023 R8: 0000000000037b2b R9: 0000000000aaaaaa R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000001 R13: ff3bb03416f1c520 R14: 0000000000000023 R15: ff3bb03416f1c4fd ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 ff48e500225a3c48: [ff3bb03416f1c4fd:kmalloc-16] 0000000000000023 ff48e500225a3c58: [ff3bb03416f1c520:kmalloc-16] 0000000000000001 ff48e500225a3c68: 0000000000000023 [ff3bb03416f1c4fd:kmalloc-16] ff48e500225a3c78: 0000000000000001 0000000000000000 ff48e500225a3c88: 0000000000aaaaaa 0000000000037b2b ff48e500225a3c98: 0000000000000065 0000000000000000 ff48e500225a3ca8: 0000000000000000 ff3bb07f7f156a08 ff48e500225a3cb8: ff3bb07f7f156a08 ffffffffffffffff ff48e500225a3cc8: usercopy_abort+116 0000000000000010 ff48e500225a3cd8: 0000000000010246 ff48e500225a3cf8 ff48e500225a3ce8: 0000000000000018 usercopy_abort+116 ff48e500225a3cf8: .LC0+19 000000000000000d ff48e500225a3d08: 0000000000000023 __check_heap_object+218 #7 [ff48e500225a3d10] __check_heap_object+0xda at ffffffff8508d0ea ff48e500225a3d18: __check_object_size+250 #8 [ff48e500225a3d18] __check_object_size+0xfa at ffffffff850b48ba ff48e500225a3d20: 0000000000000000 ff48e500225a3ed0 ff48e500225a3d30: 0000558aae40a0f0 0000000000000038 ff48e500225a3d40: filldir64+190 #9 [ff48e500225a3d40] filldir64+0xbe at ffffffff850d149e ff48e500225a3d48: 0000000000000daf ff48e500225a3ed0 ff48e500225a3d58: [ff3bb04952f3b800:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)] [ff3bb03416f1c4f0:kmalloc-16] ff48e500225a3d68: [ff3bb03416f1c4fa:kmalloc-16] 0000000000000000 ff48e500225a3d78: xfs_dir2_sf_getdents+304 #10 [ff48e500225a3d78] xfs_dir2_sf_getdents+0x130 at ffffffffc6e056e0 [xfs] ff48e500225a3d80: 00f11634b03bff10 [ff3bb040814dc1a0:kmalloc-32] ff48e500225a3d90: ff48e500225a3dd0 0000000000000000 ff48e500225a3da0: ff48e500225a3ed0 000000000000000a ff48e500225a3db0: [ff3bb04952f3b800:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)] 0000000000000001 ff48e500225a3dc0: xfs_readdir+347 #11 [ff48e500225a3dc0] xfs_readdir+0x15b at ffffffffc6e05e7b [xfs] ff48e500225a3dc8: 0000000000000000 [ff3bb040814dc1a0:kmalloc-32] ff48e500225a3dd8: 0000000000000000 0000000000000000 ff48e500225a3de8: 0000000000000000 0000000000000000 ff48e500225a3df8: 0000000000000000 0000000000000000 ff48e500225a3e08: [ff3bb04952f3b800:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)] 0000000000000000 ff48e500225a3e18: 0000000000000000 0000000000000000 ff48e500225a3e28: 0000000000000000 0000000000000000 ff48e500225a3e38: 0000000000000000 0000000000000000 ff48e500225a3e48: 0000000000000000 dddc2adfb820a900 ff48e500225a3e58: [ff3bb05261a06e00:filp(3831:cri-containerd-1415d8f4b03b128efdfb1d94a9b92eab05336df4419a5b5e060bb67cb3cbcc66.scope)] 00000000fffffffe ff48e500225a3e68: [ff3bb04952f3b938:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)] [ff3bb04952f3b9e0:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)] ff48e500225a3e78: ff48e500225a3ed0 0000000000000001 ff48e500225a3e88: iterate_dir+316 #12 [ff48e500225a3e88] iterate_dir+0x13c at ffffffff850d138c ff48e500225a3e90: 00000000653b4e62 00000000000000d9 ff48e500225a3ea0: ff48e500225a3f28 [ff3bb05261a06e00:filp(3831:cri-containerd-1415d8f4b03b128efdfb1d94a9b92eab05336df4419a5b5e060bb67cb3cbcc66.scope)] ff48e500225a3eb0: 0000000000000800 [ff3bb05261a06e00:filp(3831:cri-containerd-1415d8f4b03b128efdfb1d94a9b92eab05336df4419a5b5e060bb67cb3cbcc66.scope)] ff48e500225a3ec0: 0000000000000000 ksys_getdents64+156 #13 [ff48e500225a3ec8] ksys_getdents64+0x9c at ffffffff850d200c ff48e500225a3ed0: filldir64 0000000000000daf ff48e500225a3ee0: 0000558aae40a0f0 0000558aae40a0d8 ff48e500225a3ef0: ffffffea000007d0 dddc2adfb820a900 ff48e500225a3f00: 0000000000000000 00000000000000d9 ff48e500225a3f10: 0000000000000000 0000000000000000 ff48e500225a3f20: 0000000000000000 ff48e500225a3f58 ff48e500225a3f30: __x64_sys_getdents64+22 #14 [ff48e500225a3f30] __x64_sys_getdents64+0x16 at ffffffff850d20b6 ff48e500225a3f38: do_syscall_64+91 #15 [ff48e500225a3f38] do_syscall_64+0x5b at ffffffff84e041db ff48e500225a3f40: 0000000000000000 0000000000000000 ff48e500225a3f50: entry_SYSCALL_64_after_hwframe+101 #16 [ff48e500225a3f50] entry_SYSCALL_64_after_hwframe+0x65 at ffffffff858000ad RIP: 00007f1045604401 RSP: 00007fffa21af410 RFLAGS: 00000246 RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1045604401 RDX: 0000000000000800 RSI: 0000558aae40a0c0 RDI: 0000000000000005 RBP: 0000558aad5bfc40 R8: 0000558aae40a0a0 R9: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000558aae40a0a0 R13: 0000558aad5bfc40 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: 00000000000000d9 CS: 0033 SS: 002b crash-8.0.2> dis -rx xfs_dir2_sf_getdents+304 0xffffffffc6e055b0 <xfs_dir2_sf_getdents>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] 0xffffffffc6e055b5 <xfs_dir2_sf_getdents+0x5>: mov 0x8(%rdx),%r9 0xffffffffc6e055b9 <xfs_dir2_sf_getdents+0x9>: movzbl 0x9(%rdi),%ecx 0xffffffffc6e055bd <xfs_dir2_sf_getdents+0xd>: movabs $0x7fffffff8,%r10 0xffffffffc6e055c7 <xfs_dir2_sf_getdents+0x17>: mov 0x14(%rdi),%r8d 0xffffffffc6e055cb <xfs_dir2_sf_getdents+0x1b>: lea 0x0(,%r9,8),%rax 0xffffffffc6e055d3 <xfs_dir2_sf_getdents+0x23>: and %r10,%rax 0xffffffffc6e055d6 <xfs_dir2_sf_getdents+0x26>: sar %cl,%rax 0xffffffffc6e055d9 <xfs_dir2_sf_getdents+0x29>: cmp %eax,%r8d 0xffffffffc6e055dc <xfs_dir2_sf_getdents+0x2c>: jb 0xffffffffc6e057d9 <xfs_dir2_sf_getdents+0x229> 0xffffffffc6e055e2 <xfs_dir2_sf_getdents+0x32>: push %r15 0xffffffffc6e055e4 <xfs_dir2_sf_getdents+0x34>: shl %cl,%r8 0xffffffffc6e055e7 <xfs_dir2_sf_getdents+0x37>: push %r14 0xffffffffc6e055e9 <xfs_dir2_sf_getdents+0x39>: push %r13 0xffffffffc6e055eb <xfs_dir2_sf_getdents+0x3b>: push %r12 0xffffffffc6e055ed <xfs_dir2_sf_getdents+0x3d>: mov %rsi,%r12 0xffffffffc6e055f0 <xfs_dir2_sf_getdents+0x40>: push %rbp 0xffffffffc6e055f1 <xfs_dir2_sf_getdents+0x41>: mov %rdx,%rbp 0xffffffffc6e055f4 <xfs_dir2_sf_getdents+0x44>: push %rbx 0xffffffffc6e055f5 <xfs_dir2_sf_getdents+0x45>: sub $0x10,%rsp 0xffffffffc6e055f9 <xfs_dir2_sf_getdents+0x49>: mov 0x68(%rsi),%rax 0xffffffffc6e055fd <xfs_dir2_sf_getdents+0x4d>: mov 0x60(%rsi),%r13 0xffffffffc6e05601 <xfs_dir2_sf_getdents+0x51>: mov %rdi,0x8(%rsp) 0xffffffffc6e05606 <xfs_dir2_sf_getdents+0x56>: mov 0x68(%rax),%ecx 0xffffffffc6e05609 <xfs_dir2_sf_getdents+0x59>: mov 0x6c(%rax),%ebx 0xffffffffc6e0560c <xfs_dir2_sf_getdents+0x5c>: add %r8,%rcx 0xffffffffc6e0560f <xfs_dir2_sf_getdents+0x5f>: add %r8,%rbx 0xffffffffc6e05612 <xfs_dir2_sf_getdents+0x62>: sar $0x3,%rcx 0xffffffffc6e05616 <xfs_dir2_sf_getdents+0x66>: sar $0x3,%rbx 0xffffffffc6e0561a <xfs_dir2_sf_getdents+0x6a>: mov %ecx,%eax 0xffffffffc6e0561c <xfs_dir2_sf_getdents+0x6c>: cmp %rax,%r9 0xffffffffc6e0561f <xfs_dir2_sf_getdents+0x6f>: jg 0xffffffffc6e05658 <xfs_dir2_sf_getdents+0xa8> 0xffffffffc6e05621 <xfs_dir2_sf_getdents+0x71>: and $0x7fffffff,%ecx 0xffffffffc6e05627 <xfs_dir2_sf_getdents+0x77>: mov (%rdx),%rax 0xffffffffc6e0562a <xfs_dir2_sf_getdents+0x7a>: mov $0x4,%r9d 0xffffffffc6e05630 <xfs_dir2_sf_getdents+0x80>: mov %rbp,%rdi 0xffffffffc6e05633 <xfs_dir2_sf_getdents+0x83>: mov %rcx,0x8(%rdx) 0xffffffffc6e05637 <xfs_dir2_sf_getdents+0x87>: mov $0x1,%edx 0xffffffffc6e0563c <xfs_dir2_sf_getdents+0x8c>: mov 0x20(%rsi),%r8 0xffffffffc6e05640 <xfs_dir2_sf_getdents+0x90>: mov $0xffffffffc6e49a70,%rsi 0xffffffffc6e05647 <xfs_dir2_sf_getdents+0x97>: call 0xffffffff85a011f0 <__x86_indirect_thunk_rax> 0xffffffffc6e0564c <xfs_dir2_sf_getdents+0x9c>: test %eax,%eax 0xffffffffc6e0564e <xfs_dir2_sf_getdents+0x9e>: jne 0xffffffffc6e0577f <xfs_dir2_sf_getdents+0x1cf> 0xffffffffc6e05654 <xfs_dir2_sf_getdents+0xa4>: mov 0x8(%rbp),%r9 0xffffffffc6e05658 <xfs_dir2_sf_getdents+0xa8>: mov %ebx,%eax 0xffffffffc6e0565a <xfs_dir2_sf_getdents+0xaa>: cmp %r9,%rax 0xffffffffc6e0565d <xfs_dir2_sf_getdents+0xad>: jge 0xffffffffc6e05790 <xfs_dir2_sf_getdents+0x1e0> 0xffffffffc6e05663 <xfs_dir2_sf_getdents+0xb3>: cmpb $0x1,0x1(%r13) 0xffffffffc6e05668 <xfs_dir2_sf_getdents+0xb8>: sbb %rax,%rax 0xffffffffc6e0566b <xfs_dir2_sf_getdents+0xbb>: xor %r15d,%r15d 0xffffffffc6e0566e <xfs_dir2_sf_getdents+0xbe>: and $0xfffffffffffffffc,%rax 0xffffffffc6e05672 <xfs_dir2_sf_getdents+0xc2>: cmpb $0x0,0x0(%r13) 0xffffffffc6e05677 <xfs_dir2_sf_getdents+0xc7>: lea 0xa(%r13,%rax,1),%r14 0xffffffffc6e0567c <xfs_dir2_sf_getdents+0xcc>: jne 0xffffffffc6e0570d <xfs_dir2_sf_getdents+0x15d> 0xffffffffc6e05682 <xfs_dir2_sf_getdents+0xd2>: jmp 0xffffffffc6e05760 <xfs_dir2_sf_getdents+0x1b0> 0xffffffffc6e05687 <xfs_dir2_sf_getdents+0xd7>: mov 0x20(%rdx),%rax 0xffffffffc6e0568b <xfs_dir2_sf_getdents+0xdb>: mov %r14,%rsi 0xffffffffc6e0568e <xfs_dir2_sf_getdents+0xde>: mov %r13,%rdi 0xffffffffc6e05691 <xfs_dir2_sf_getdents+0xe1>: and $0x7fffffff,%ebx 0xffffffffc6e05697 <xfs_dir2_sf_getdents+0xe7>: call 0xffffffff85a011f0 <__x86_indirect_thunk_rax> 0xffffffffc6e0569c <xfs_dir2_sf_getdents+0xec>: mov %r14,%rdi 0xffffffffc6e0569f <xfs_dir2_sf_getdents+0xef>: mov %rax,(%rsp) 0xffffffffc6e056a3 <xfs_dir2_sf_getdents+0xf3>: mov 0x68(%r12),%rax 0xffffffffc6e056a8 <xfs_dir2_sf_getdents+0xf8>: mov 0x10(%rax),%rax 0xffffffffc6e056ac <xfs_dir2_sf_getdents+0xfc>: call 0xffffffff85a011f0 <__x86_indirect_thunk_rax> 0xffffffffc6e056b1 <xfs_dir2_sf_getdents+0x101>: mov %rbx,0x8(%rbp) 0xffffffffc6e056b5 <xfs_dir2_sf_getdents+0x105>: mov (%r12),%rdi 0xffffffffc6e056b9 <xfs_dir2_sf_getdents+0x109>: movzbl %al,%esi 0xffffffffc6e056bc <xfs_dir2_sf_getdents+0x10c>: call 0xffffffffc6e05560 <xfs_dir3_get_dtype> 0xffffffffc6e056c1 <xfs_dir2_sf_getdents+0x111>: movzbl (%r14),%edx 0xffffffffc6e056c5 <xfs_dir2_sf_getdents+0x115>: lea 0x3(%r14),%rsi ——> 0x3( r14) is the second parameter, so r14 stores the “sfep” 0xffffffffc6e056c9 <xfs_dir2_sf_getdents+0x119>: mov %rbx,%rcx 0xffffffffc6e056cc <xfs_dir2_sf_getdents+0x11c>: mov 0x0(%rbp),%r11 0xffffffffc6e056d0 <xfs_dir2_sf_getdents+0x120>: movzbl %al,%r9d 0xffffffffc6e056d4 <xfs_dir2_sf_getdents+0x124>: mov (%rsp),%r8 0xffffffffc6e056d8 <xfs_dir2_sf_getdents+0x128>: mov %rbp,%rdi 0xffffffffc6e056db <xfs_dir2_sf_getdents+0x12b>: call 0xffffffff85a01330 <__x86_indirect_thunk_r11> 0xffffffffc6e056e0 <xfs_dir2_sf_getdents+0x130>: test %eax,%eax crash-8.0.2> dis -rx filldir64+190 0xffffffff850d13e0 <filldir64>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] 0xffffffff850d13e5 <filldir64+0x5>: push %r15 0xffffffff850d13e7 <filldir64+0x7>: push %r14. ———> the second pushes up from the stack frame base is the “sfep” Up to this point, I think the second element in the stack of filldir64 is sfep. 0xffffffff850d13e9 <filldir64+0x9>: push %r13 0xffffffff850d13eb <filldir64+0xb>: lea 0x1b(%rdx),%r13d 0xffffffff850d13ef <filldir64+0xf>: push %r12 0xffffffff850d13f1 <filldir64+0x11>: and $0xfffffff8,%r13d 0xffffffff850d13f5 <filldir64+0x15>: push %rbp 0xffffffff850d13f6 <filldir64+0x16>: push %rbx 0xffffffff850d13f7 <filldir64+0x17>: movl $0xffffffea,0x24(%rdi) 0xffffffff850d13fe <filldir64+0x1e>: cmp %r13d,0x20(%rdi) 0xffffffff850d1402 <filldir64+0x22>: jl 0xffffffff850d14f9 <filldir64+0x119> 0xffffffff850d1408 <filldir64+0x28>: mov %rsi,%r15 0xffffffff850d140b <filldir64+0x2b>: mov 0x18(%rdi),%rsi 0xffffffff850d140f <filldir64+0x2f>: mov %rdi,%rbp 0xffffffff850d1412 <filldir64+0x32>: test %rsi,%rsi 0xffffffff850d1415 <filldir64+0x35>: je 0xffffffff850d143f <filldir64+0x5f> 0xffffffff850d1417 <filldir64+0x37>: mov %gs:0x15c80,%rax 0xffffffff850d1420 <filldir64+0x40>: mov (%rax),%rax 0xffffffff850d1423 <filldir64+0x43>: test $0x4,%al 0xffffffff850d1425 <filldir64+0x45>: jne 0xffffffff850d1500 <filldir64+0x120> 0xffffffff850d142b <filldir64+0x4b>: stac 0xffffffff850d142e <filldir64+0x4e>: xor %eax,%eax 0xffffffff850d1430 <filldir64+0x50>: mov %rcx,0x8(%rsi) 0xffffffff850d1434 <filldir64+0x54>: clac 0xffffffff850d1437 <filldir64+0x57>: test %eax,%eax 0xffffffff850d1439 <filldir64+0x59>: jne 0xffffffff850d14e2 <filldir64+0x102> 0xffffffff850d143f <filldir64+0x5f>: mov 0x10(%rbp),%r12 0xffffffff850d1443 <filldir64+0x63>: stac 0xffffffff850d1446 <filldir64+0x66>: xor %eax,%eax 0xffffffff850d1448 <filldir64+0x68>: mov %r8,(%r12) 0xffffffff850d144c <filldir64+0x6c>: clac 0xffffffff850d144f <filldir64+0x6f>: test %eax,%eax 0xffffffff850d1451 <filldir64+0x71>: jne 0xffffffff850d14e2 <filldir64+0x102> 0xffffffff850d1457 <filldir64+0x77>: stac 0xffffffff850d145a <filldir64+0x7a>: movq $0x0,0x8(%r12) 0xffffffff850d1463 <filldir64+0x83>: clac 0xffffffff850d1466 <filldir64+0x86>: test %eax,%eax 0xffffffff850d1468 <filldir64+0x88>: jne 0xffffffff850d14e2 <filldir64+0x102> 0xffffffff850d146a <filldir64+0x8a>: stac 0xffffffff850d146d <filldir64+0x8d>: mov %r13w,0x10(%r12) 0xffffffff850d1473 <filldir64+0x93>: clac 0xffffffff850d1476 <filldir64+0x96>: test %eax,%eax 0xffffffff850d1478 <filldir64+0x98>: jne 0xffffffff850d14e2 <filldir64+0x102> 0xffffffff850d147a <filldir64+0x9a>: stac 0xffffffff850d147d <filldir64+0x9d>: mov %eax,%ebx 0xffffffff850d147f <filldir64+0x9f>: mov %r9b,0x12(%r12) 0xffffffff850d1484 <filldir64+0xa4>: clac 0xffffffff850d1487 <filldir64+0xa7>: test %ebx,%ebx 0xffffffff850d1489 <filldir64+0xa9>: jne 0xffffffff850d14e2 <filldir64+0x102> 0xffffffff850d148b <filldir64+0xab>: movslq %edx,%r14 0xffffffff850d148e <filldir64+0xae>: mov %r15,%rdi. ——> r15 is the first parameter when calling __check_object_size 0xffffffff850d1491 <filldir64+0xb1>: mov $0x1,%edx 0xffffffff850d1496 <filldir64+0xb6>: mov %r14,%rsi 0xffffffff850d1499 <filldir64+0xb9>: call 0xffffffff850b47c0 <__check_object_size> 0xffffffff850d149e <filldir64+0xbe>: mov %r14,%rdx Up to this point, r15 is the given object for __check_object_size to validate. And no one change the r15 until the end, so we can get the given object from stack of usercopy_abort i.e., R15: ff3bb03416f1c4fd. I check “sfep” and the give object by using “kmem” crash-8.0.2> kmem ff3bb03416f1c4fa CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME ff3bb00107c0fa00 16 2074327 2860288 11173 4k kmalloc-16 SLAB MEMORY NODE TOTAL ALLOCATED FREE ff8faa00d05bc700 ff3bb03416f1c000 0 256 76 180 FREE / [ALLOCATED] ff3bb03416f1c4f0 (cpu 4 cache) PAGE PHYSICAL MAPPING INDEX CNT FLAGS ff8faa00d05bc700 3416f1c000 ff3bb00107c0fa00 ff3bb03416f1ca00 1 17ffffc0000100 slab crash-8.0.2> kmem ff3bb03416f1c4fd CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME ff3bb00107c0fa00 16 2074327 2860288 11173 4k kmalloc-16 SLAB MEMORY NODE TOTAL ALLOCATED FREE ff8faa00d05bc700 ff3bb03416f1c000 0 256 76 180 FREE / [ALLOCATED] ff3bb03416f1c4f0 (cpu 4 cache) PAGE PHYSICAL MAPPING INDEX CNT FLAGS ff8faa00d05bc700 3416f1c000 ff3bb00107c0fa00 ff3bb03416f1ca00 1 17ffffc0000100 slab It seemed the object have been freed. I have no idea what is going on here. Sincerely hope that you lot give me some help. Below is some information about my server. # xfs_db -r /dev/nvme0n1 xfs_db> version versionnum [0xbcb5+0x18a] = V5,NLINK,DIRV2,ATTR,ALIGN,LOGV2,EXTFLG,SECTOR,MOREBITS,ATTR2,LAZYSBCOUNT,PROJID32BIT,CRC,FTYPE Linux kernel version is v4.18 and the OS is Centos7. Best regards. |
Personally, I do not consider XFS stable enough for serious work. In an enterprise environment with hot failover, hot swap storage, and rotational backups I did test it to see if the performance would justify the risk, and found that on that storage EXT4 on LVM on MDADM RAID-5 performed better than XFS and was far less prone to corruption. (BTRFS RAID was not ready for prime time, but I included it for completeness and found it performed worse than either and was no more stable or safe. It has improved a LOT since then, but not enough to catch up to EXT4.)
I never got to test ZFS, and I really would like to know how those tests would have compared. |
Probably if you can you should try a newer Kernel version? This seems like the type of issue that might have been fixed at some point. If you have to stick with 4.xx, then try with a patched super LTS version.
Like one of these: 4.19.298 4.14.329 |
All times are GMT -5. The time now is 05:47 AM. |