NFS Problems, dropped connections, errors on client.
Pages: 1 2
wid
47 Posts
June 13, 2022, 8:16 pmQuote from wid on June 13, 2022, 8:16 pmHi,
newest Petasan 3.1.0
NFS Cluster,
All NFS customers, regularly have problems accessing NFS. IO errors pop up, connections are broken.
Instance #1:
NFS4 status code: [0], nfs41_write failed on offset 9457664 with Hard I/O error
Error: nfs41_commit failed with
Instance #2:
NFS4 status code: [0]
Reconnectable protocol device was closed.
Failed to upload disk. Skipped arguments: [>];
Agent failed to process method {DataTransfer.SyncDisk}.
The problems started after the last update.
Petasan.log -> nothing there
ganesha.log -> shows the following errors:
10/06/2022 02:23:21 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26493] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4329a77370 fd 126 recv errno 104 (will set dead)
10/06/2022 02:23:21 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26486] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f437241b9d0 fd 126 recv errno 104 (will set dead)
10/06/2022 06:23:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26472] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436a6ff7e0 fd 113 recv errno 104 (will set dead)
10/06/2022 06:23:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26491] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43c0ccc860 fd 113 recv errno 104 (will set dead)
10/06/2022 08:41:45 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 50424171, old: 50424170; deq new: 50424170, old: 50424170
10/06/2022 10:24:04 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26483] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42e4623710 fd 110 recv errno 104 (will set dead)
10/06/2022 10:24:04 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26484] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43296be240 fd 110 recv errno 104 (will set dead)
10/06/2022 14:24:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26473] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4324c44840 fd 113 recv errno 104 (will set dead)
10/06/2022 14:24:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26493] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4329a77370 fd 113 recv errno 104 (will set dead)
10/06/2022 15:50:32 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53390976, old: 53390974; deq new: 53390974, old: 53390974
10/06/2022 16:15:10 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53411259, old: 53411258; deq new: 53411258, old: 53411258
10/06/2022 16:25:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53416913, old: 53416909; deq new: 53416909, old: 53416909
10/06/2022 16:33:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53422261, old: 53422257; deq new: 53422257, old: 53422257
10/06/2022 17:44:24 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53545156, old: 53545152; deq new: 53545150, old: 53545150
10/06/2022 17:46:30 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53549612, old: 53549608; deq new: 53549604, old: 53549604
10/06/2022 18:24:07 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26487] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4353926310 fd 110 recv errno 104 (will set dead)
10/06/2022 18:24:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26480] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42a48f1020 fd 113 recv errno 104 (will set dead)
10/06/2022 18:48:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53629651, old: 53629650; deq new: 53629650, old: 53629650
10/06/2022 19:08:58 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53652902, old: 53652898; deq new: 53652898, old: 53652898
10/06/2022 20:06:15 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53724992, old: 53724991; deq new: 53724991, old: 53724991
10/06/2022 20:41:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53776185, old: 53776184; deq new: 53776184, old: 53776184
10/06/2022 21:32:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53849148, old: 53849146; deq new: 53849146, old: 53849146
10/06/2022 21:44:46 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53891215, old: 53891211; deq new: 53891211, old: 53891211
10/06/2022 21:47:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53894738, old: 53894734; deq new: 53894734, old: 53894734
10/06/2022 21:48:39 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53896871, old: 53896867; deq new: 53896867, old: 53896867
10/06/2022 22:24:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26495] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4354031ef0 fd 113 recv errno 104 (will set dead)
10/06/2022 22:24:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26486] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4324c413c0 fd 113 recv errno 104 (will set dead)
11/06/2022 01:21:38 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 55741738, old: 55741737; deq new: 55741737, old: 55741737
11/06/2022 02:24:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26465] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4359f58750 fd 113 recv errno 104 (will set dead)
11/06/2022 02:24:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26486] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4373137010 fd 113 recv errno 104 (will set dead)
11/06/2022 03:10:03 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 56642823, old: 56642822; deq new: 56642821, old: 56642821
11/06/2022 05:25:01 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57567777, old: 57567776; deq new: 57567772, old: 57567772
11/06/2022 05:53:20 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57597290, old: 57597289; deq new: 57597285, old: 57597285
11/06/2022 06:24:46 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26659] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4351343110 fd 109 recv errno 104 (will set dead)
11/06/2022 06:24:46 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26656] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f429a68d940 fd 109 recv errno 104 (will set dead)
11/06/2022 07:47:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57730421, old: 57730417; deq new: 57730417, old: 57730417
11/06/2022 08:27:29 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57791396, old: 57791392; deq new: 57791392, old: 57791392
11/06/2022 08:36:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57803275, old: 57803272; deq new: 57803271, old: 57803271
11/06/2022 09:31:13 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57869356, old: 57869352; deq new: 57869352, old: 57869352
11/06/2022 10:24:53 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26898] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43091a14c0 fd 110 recv errno 104 (will set dead)
11/06/2022 10:24:53 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26883] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c581e6c0 fd 110 recv errno 104 (will set dead)
11/06/2022 10:54:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26961] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42fcfba4e0 fd 129 recv errno 104 (will set dead)
11/06/2022 13:41:04 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27140] nfs4_Compound :NFS4 :CRIT :Bad Minor Version 29801
11/06/2022 14:24:57 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27117] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c930b980 fd 110 recv errno 104 (will set dead)
11/06/2022 14:24:57 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27174] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42a9e2cfa0 fd 110 recv errno 104 (will set dead)
11/06/2022 14:37:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58124662, old: 58124658; deq new: 58124652, old: 58124652
11/06/2022 18:01:32 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58476288, old: 58476284; deq new: 58476284, old: 58476284
11/06/2022 18:25:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27280] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42abfb2900 fd 113 recv errno 104 (will set dead)
11/06/2022 18:25:14 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27210] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42fca68f70 fd 113 recv errno 104 (will set dead)
11/06/2022 18:27:53 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58525148, old: 58525147; deq new: 58525143, old: 58525143
11/06/2022 19:06:25 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58579809, old: 58579805; deq new: 58579805, old: 58579805
11/06/2022 19:24:56 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58610210, old: 58610206; deq new: 58610206, old: 58610206
11/06/2022 22:25:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27440] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43260ac5a0 fd 132 recv errno 104 (will set dead)
11/06/2022 22:25:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27436] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f432806e130 fd 132 recv errno 104 (will set dead)
12/06/2022 02:25:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27426] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436a9cbf30 fd 129 recv errno 104 (will set dead)
12/06/2022 02:25:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27455] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f434d6c9250 fd 129 recv errno 104 (will set dead)
12/06/2022 06:25:25 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27417] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f429b2baa20 fd 113 recv errno 104 (will set dead)
12/06/2022 06:25:25 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27457] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f432447f090 fd 113 recv errno 104 (will set dead)
12/06/2022 10:25:52 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27528] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4356a9d970 fd 113 recv errno 104 (will set dead)
12/06/2022 10:25:54 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27528] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42e800ae30 fd 113 recv errno 104 (will set dead)
12/06/2022 14:25:55 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27585] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43091a14c0 fd 110 recv errno 104 (will set dead)
12/06/2022 14:25:55 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27586] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436c3b38d0 fd 110 recv errno 104 (will set dead)
12/06/2022 16:10:30 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 66695226, old: 66695225; deq new: 66695224, old: 66695224
12/06/2022 18:26:17 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27629] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436a6ff7e0 fd 124 recv errno 104 (will set dead)
12/06/2022 18:26:17 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27549] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42a621db00 fd 124 recv errno 104 (will set dead)
12/06/2022 19:24:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 67506309, old: 67506308; deq new: 67506307, old: 67506307
12/06/2022 20:24:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 67627961, old: 67627960; deq new: 67627960, old: 67627960
12/06/2022 22:26:20 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27707] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c4fd6fe0 fd 129 recv errno 104 (will set dead)
12/06/2022 22:26:20 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27702] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c930b980 fd 129 recv errno 104 (will set dead)
13/06/2022 00:35:38 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69302078, old: 69302074; deq new: 69302074, old: 69302074
13/06/2022 01:13:03 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27917] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f435499f060 fd 131 recv errno 104 (will set dead)
13/06/2022 02:26:37 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28044] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43b645e010 fd 126 recv errno 104 (will set dead)
13/06/2022 02:26:37 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28038] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a426f180 fd 126 recv errno 104 (will set dead)
13/06/2022 06:26:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28406] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a426f180 fd 126 recv errno 104 (will set dead)
13/06/2022 06:26:34 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28400] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4347816570 fd 126 recv errno 104 (will set dead)
13/06/2022 10:26:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28717] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a426f180 fd 126 recv errno 104 (will set dead)
13/06/2022 10:26:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28767] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4373137010 fd 126 recv errno 104 (will set dead)
13/06/2022 14:27:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29133] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4373137010 fd 124 recv errno 104 (will set dead)
13/06/2022 14:27:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29129] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c930b980 fd 124 recv errno 104 (will set dead)
13/06/2022 15:39:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29237] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42d4006c00 fd 114 recv errno 104 (will set dead)
13/06/2022 15:39:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29235] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f435499f060 fd 110 recv errno 104 (will set dead)
13/06/2022 15:39:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29231] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c5eab500 fd 58 recv errno 104 (will set dead)
13/06/2022 15:39:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29241] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4324c44840 fd 124 recv errno 104 (will set dead)
13/06/2022 15:39:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29235] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43730d4a80 fd 124 recv errno 104 (will set dead)
13/06/2022 16:02:10 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69335717, old: 69335713; deq new: 69335713, old: 69335713
13/06/2022 17:06:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69409040, old: 69409036; deq new: 69409036, old: 69409036
13/06/2022 17:35:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69447453, old: 69447449; deq new: 69447449, old: 69447449
13/06/2022 17:36:52 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69448544, old: 69448540; deq new: 69448540, old: 69448540
13/06/2022 19:40:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29458] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f434726c630 fd 108 recv errno 104 (will set dead)
13/06/2022 19:40:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29642] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a90a9390 fd 108 recv errno 104 (will set dead)
Hi,
newest Petasan 3.1.0
NFS Cluster,
All NFS customers, regularly have problems accessing NFS. IO errors pop up, connections are broken.
Instance #1:
NFS4 status code: [0], nfs41_write failed on offset 9457664 with Hard I/O error
Error: nfs41_commit failed with
Instance #2:
NFS4 status code: [0]
Reconnectable protocol device was closed.
Failed to upload disk. Skipped arguments: [>];
Agent failed to process method {DataTransfer.SyncDisk}.
The problems started after the last update.
Petasan.log -> nothing there
ganesha.log -> shows the following errors:
10/06/2022 02:23:21 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26493] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4329a77370 fd 126 recv errno 104 (will set dead)
10/06/2022 02:23:21 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26486] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f437241b9d0 fd 126 recv errno 104 (will set dead)
10/06/2022 06:23:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26472] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436a6ff7e0 fd 113 recv errno 104 (will set dead)
10/06/2022 06:23:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26491] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43c0ccc860 fd 113 recv errno 104 (will set dead)
10/06/2022 08:41:45 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 50424171, old: 50424170; deq new: 50424170, old: 50424170
10/06/2022 10:24:04 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26483] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42e4623710 fd 110 recv errno 104 (will set dead)
10/06/2022 10:24:04 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26484] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43296be240 fd 110 recv errno 104 (will set dead)
10/06/2022 14:24:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26473] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4324c44840 fd 113 recv errno 104 (will set dead)
10/06/2022 14:24:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26493] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4329a77370 fd 113 recv errno 104 (will set dead)
10/06/2022 15:50:32 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53390976, old: 53390974; deq new: 53390974, old: 53390974
10/06/2022 16:15:10 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53411259, old: 53411258; deq new: 53411258, old: 53411258
10/06/2022 16:25:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53416913, old: 53416909; deq new: 53416909, old: 53416909
10/06/2022 16:33:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53422261, old: 53422257; deq new: 53422257, old: 53422257
10/06/2022 17:44:24 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53545156, old: 53545152; deq new: 53545150, old: 53545150
10/06/2022 17:46:30 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53549612, old: 53549608; deq new: 53549604, old: 53549604
10/06/2022 18:24:07 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26487] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4353926310 fd 110 recv errno 104 (will set dead)
10/06/2022 18:24:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26480] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42a48f1020 fd 113 recv errno 104 (will set dead)
10/06/2022 18:48:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53629651, old: 53629650; deq new: 53629650, old: 53629650
10/06/2022 19:08:58 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53652902, old: 53652898; deq new: 53652898, old: 53652898
10/06/2022 20:06:15 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53724992, old: 53724991; deq new: 53724991, old: 53724991
10/06/2022 20:41:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53776185, old: 53776184; deq new: 53776184, old: 53776184
10/06/2022 21:32:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53849148, old: 53849146; deq new: 53849146, old: 53849146
10/06/2022 21:44:46 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53891215, old: 53891211; deq new: 53891211, old: 53891211
10/06/2022 21:47:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53894738, old: 53894734; deq new: 53894734, old: 53894734
10/06/2022 21:48:39 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53896871, old: 53896867; deq new: 53896867, old: 53896867
10/06/2022 22:24:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26495] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4354031ef0 fd 113 recv errno 104 (will set dead)
10/06/2022 22:24:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26486] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4324c413c0 fd 113 recv errno 104 (will set dead)
11/06/2022 01:21:38 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 55741738, old: 55741737; deq new: 55741737, old: 55741737
11/06/2022 02:24:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26465] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4359f58750 fd 113 recv errno 104 (will set dead)
11/06/2022 02:24:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26486] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4373137010 fd 113 recv errno 104 (will set dead)
11/06/2022 03:10:03 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 56642823, old: 56642822; deq new: 56642821, old: 56642821
11/06/2022 05:25:01 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57567777, old: 57567776; deq new: 57567772, old: 57567772
11/06/2022 05:53:20 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57597290, old: 57597289; deq new: 57597285, old: 57597285
11/06/2022 06:24:46 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26659] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4351343110 fd 109 recv errno 104 (will set dead)
11/06/2022 06:24:46 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26656] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f429a68d940 fd 109 recv errno 104 (will set dead)
11/06/2022 07:47:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57730421, old: 57730417; deq new: 57730417, old: 57730417
11/06/2022 08:27:29 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57791396, old: 57791392; deq new: 57791392, old: 57791392
11/06/2022 08:36:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57803275, old: 57803272; deq new: 57803271, old: 57803271
11/06/2022 09:31:13 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57869356, old: 57869352; deq new: 57869352, old: 57869352
11/06/2022 10:24:53 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26898] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43091a14c0 fd 110 recv errno 104 (will set dead)
11/06/2022 10:24:53 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26883] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c581e6c0 fd 110 recv errno 104 (will set dead)
11/06/2022 10:54:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26961] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42fcfba4e0 fd 129 recv errno 104 (will set dead)
11/06/2022 13:41:04 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27140] nfs4_Compound :NFS4 :CRIT :Bad Minor Version 29801
11/06/2022 14:24:57 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27117] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c930b980 fd 110 recv errno 104 (will set dead)
11/06/2022 14:24:57 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27174] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42a9e2cfa0 fd 110 recv errno 104 (will set dead)
11/06/2022 14:37:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58124662, old: 58124658; deq new: 58124652, old: 58124652
11/06/2022 18:01:32 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58476288, old: 58476284; deq new: 58476284, old: 58476284
11/06/2022 18:25:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27280] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42abfb2900 fd 113 recv errno 104 (will set dead)
11/06/2022 18:25:14 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27210] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42fca68f70 fd 113 recv errno 104 (will set dead)
11/06/2022 18:27:53 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58525148, old: 58525147; deq new: 58525143, old: 58525143
11/06/2022 19:06:25 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58579809, old: 58579805; deq new: 58579805, old: 58579805
11/06/2022 19:24:56 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58610210, old: 58610206; deq new: 58610206, old: 58610206
11/06/2022 22:25:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27440] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43260ac5a0 fd 132 recv errno 104 (will set dead)
11/06/2022 22:25:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27436] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f432806e130 fd 132 recv errno 104 (will set dead)
12/06/2022 02:25:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27426] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436a9cbf30 fd 129 recv errno 104 (will set dead)
12/06/2022 02:25:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27455] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f434d6c9250 fd 129 recv errno 104 (will set dead)
12/06/2022 06:25:25 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27417] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f429b2baa20 fd 113 recv errno 104 (will set dead)
12/06/2022 06:25:25 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27457] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f432447f090 fd 113 recv errno 104 (will set dead)
12/06/2022 10:25:52 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27528] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4356a9d970 fd 113 recv errno 104 (will set dead)
12/06/2022 10:25:54 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27528] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42e800ae30 fd 113 recv errno 104 (will set dead)
12/06/2022 14:25:55 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27585] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43091a14c0 fd 110 recv errno 104 (will set dead)
12/06/2022 14:25:55 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27586] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436c3b38d0 fd 110 recv errno 104 (will set dead)
12/06/2022 16:10:30 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 66695226, old: 66695225; deq new: 66695224, old: 66695224
12/06/2022 18:26:17 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27629] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436a6ff7e0 fd 124 recv errno 104 (will set dead)
12/06/2022 18:26:17 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27549] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42a621db00 fd 124 recv errno 104 (will set dead)
12/06/2022 19:24:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 67506309, old: 67506308; deq new: 67506307, old: 67506307
12/06/2022 20:24:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 67627961, old: 67627960; deq new: 67627960, old: 67627960
12/06/2022 22:26:20 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27707] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c4fd6fe0 fd 129 recv errno 104 (will set dead)
12/06/2022 22:26:20 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27702] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c930b980 fd 129 recv errno 104 (will set dead)
13/06/2022 00:35:38 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69302078, old: 69302074; deq new: 69302074, old: 69302074
13/06/2022 01:13:03 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27917] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f435499f060 fd 131 recv errno 104 (will set dead)
13/06/2022 02:26:37 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28044] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43b645e010 fd 126 recv errno 104 (will set dead)
13/06/2022 02:26:37 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28038] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a426f180 fd 126 recv errno 104 (will set dead)
13/06/2022 06:26:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28406] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a426f180 fd 126 recv errno 104 (will set dead)
13/06/2022 06:26:34 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28400] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4347816570 fd 126 recv errno 104 (will set dead)
13/06/2022 10:26:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28717] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a426f180 fd 126 recv errno 104 (will set dead)
13/06/2022 10:26:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28767] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4373137010 fd 126 recv errno 104 (will set dead)
13/06/2022 14:27:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29133] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4373137010 fd 124 recv errno 104 (will set dead)
13/06/2022 14:27:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29129] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c930b980 fd 124 recv errno 104 (will set dead)
13/06/2022 15:39:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29237] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42d4006c00 fd 114 recv errno 104 (will set dead)
13/06/2022 15:39:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29235] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f435499f060 fd 110 recv errno 104 (will set dead)
13/06/2022 15:39:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29231] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c5eab500 fd 58 recv errno 104 (will set dead)
13/06/2022 15:39:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29241] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4324c44840 fd 124 recv errno 104 (will set dead)
13/06/2022 15:39:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29235] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43730d4a80 fd 124 recv errno 104 (will set dead)
13/06/2022 16:02:10 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69335717, old: 69335713; deq new: 69335713, old: 69335713
13/06/2022 17:06:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69409040, old: 69409036; deq new: 69409036, old: 69409036
13/06/2022 17:35:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69447453, old: 69447449; deq new: 69447449, old: 69447449
13/06/2022 17:36:52 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69448544, old: 69448540; deq new: 69448540, old: 69448540
13/06/2022 19:40:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29458] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f434726c630 fd 108 recv errno 104 (will set dead)
13/06/2022 19:40:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29642] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a90a9390 fd 108 recv errno 104 (will set dead)
admin
2,930 Posts
June 14, 2022, 9:36 amQuote from admin on June 14, 2022, 9:36 amDo you see any errors in syslog
Was the cluster healthy ? from PG status charts were any PGs not clean ?
Do you see high utlization in node stats for % disk or % cpu ?
Is the NFS Status always up in UI ? does it show down or grace status ?
Do clients connect using the public NFS ips ? can you specify -o nfsvers=4.1 when mounting the clients.
Do you see any errors in syslog
Was the cluster healthy ? from PG status charts were any PGs not clean ?
Do you see high utlization in node stats for % disk or % cpu ?
Is the NFS Status always up in UI ? does it show down or grace status ?
Do clients connect using the public NFS ips ? can you specify -o nfsvers=4.1 when mounting the clients.
wid
47 Posts
June 14, 2022, 7:53 pmQuote from wid on June 14, 2022, 7:53 pmNo errors in syslog,
cluster is healthy,
PG Was moved (due hdd prepared to be removed)
CPU load 1/16, Mem used 50%.
NFS Status - probably ok, not ctched any errors.
Clients (2 veeam backup servers) are connected as 4.1 by default.
cluster:
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph03,ceph01,ceph02 (age 3d)
mgr: ceph02(active, since 2w), standbys: ceph01, ceph03
mds: fs-mit:1 {0=ceph01=up:active}
osd: 26 osds: 26 up (since 25h), 26 in (since 4d)
data:
pools: 7 pools, 697 pgs
objects: 15.41M objects, 34 TiB
usage: 62 TiB used, 66 TiB / 128 TiB avail
pgs: 690 active+clean
7 active+clean+scrubbing+deep
No errors in syslog,
cluster is healthy,
PG Was moved (due hdd prepared to be removed)
CPU load 1/16, Mem used 50%.
NFS Status - probably ok, not ctched any errors.
Clients (2 veeam backup servers) are connected as 4.1 by default.
cluster:
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph03,ceph01,ceph02 (age 3d)
mgr: ceph02(active, since 2w), standbys: ceph01, ceph03
mds: fs-mit:1 {0=ceph01=up:active}
osd: 26 osds: 26 up (since 25h), 26 in (since 4d)
data:
pools: 7 pools, 697 pgs
objects: 15.41M objects, 34 TiB
usage: 62 TiB used, 66 TiB / 128 TiB avail
pgs: 690 active+clean
7 active+clean+scrubbing+deep
wid
47 Posts
June 17, 2022, 10:44 amQuote from wid on June 17, 2022, 10:44 amNew type of error from veeam:
16.06.2022 23:43:35 :: Processing XXX Error: NFS4 status code: [0]
nfs41_write failed on offset 1099511627264 with The file size exceeds the limit allowed and cannot be saved
Failed to download disk 'XXXXXX'.
Reconnectable protocol device was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
New type of error from veeam:
16.06.2022 23:43:35 :: Processing XXX Error: NFS4 status code: [0]
nfs41_write failed on offset 1099511627264 with The file size exceeds the limit allowed and cannot be saved
Failed to download disk 'XXXXXX'.
Reconnectable protocol device was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
admin
2,930 Posts
June 17, 2022, 12:38 pmQuote from admin on June 17, 2022, 12:38 pm"The file size exceeds the limit allowed and cannot be saved" looks more like a preset file size limit of 1099511627264 offset ( 1TB ) and not an error at any random offset. What OS is the client ( Windows / Linux / kernel version ?). Also what is the file size you are writing ?
"The file size exceeds the limit allowed and cannot be saved" looks more like a preset file size limit of 1099511627264 offset ( 1TB ) and not an error at any random offset. What OS is the client ( Windows / Linux / kernel version ?). Also what is the file size you are writing ?
wid
47 Posts
June 17, 2022, 12:42 pmQuote from wid on June 17, 2022, 12:42 pmWindows 2019 std,
Writing software is Veeam (10.x) planned upgrade to latest version.
After iSCSI there were no write issues and no errors shown (there were however issues with performance) NFS is about 25x faster than iSCSI was on Petasan.
Windows 2019 std,
Writing software is Veeam (10.x) planned upgrade to latest version.
After iSCSI there were no write issues and no errors shown (there were however issues with performance) NFS is about 25x faster than iSCSI was on Petasan.
admin
2,930 Posts
June 17, 2022, 12:57 pmQuote from admin on June 17, 2022, 12:57 pmWhat is the file size you are writing ?
What speeds were you getting with iSCSI ?
What is the hardware : SSDs/HDDs ?
What is the file size you are writing ?
What speeds were you getting with iSCSI ?
What is the hardware : SSDs/HDDs ?
wid
47 Posts
June 17, 2022, 1:01 pmQuote from wid on June 17, 2022, 1:01 pmiSCSI - 20 MB/s
NFS - 100-200 MB/s
Hardware:
Dell 2 x R510 / 1 x R730Xd (32/64 GB RAM) not used 100%
26 x HDD WD RED PRO 6TB.
iSCSI - 20 MB/s
NFS - 100-200 MB/s
Hardware:
Dell 2 x R510 / 1 x R730Xd (32/64 GB RAM) not used 100%
26 x HDD WD RED PRO 6TB.
admin
2,930 Posts
June 17, 2022, 1:40 pmQuote from admin on June 17, 2022, 1:40 pmThanks for the info. Generally iSCSI is always faster, in your case NFS was 5 to 10 times faster! Most probable reason is because you are using pure HDDs and doing Veeam backups: pure HDD give low performance with small block size but are good at large sizes, by default Windows iSCSI initiator does have a 256KB block size limit by default, Veeam blocksize is 4 MB so it gets chopped in iSCSI but not in NFS, you can change the Windows iSCSI value from 256KB to 4MB and it will give similar performance to NFS. At small sizes on both iSCSI and NFS, iSCSI will be faster.
To your issue: It looks like a file size limitation error which is imposing a 1 TB limit on the file, i suggest you search Windows or Veeam they have some limit with NFS.
For your earlier issue when you did an upgrade, i believe it is not related to your recent Veeam error. If this happens again i would suggest looking at the syslog and ganesha log as well as any Ceph errors at that time. Also as suggested before to look into load on the system, like % disk utilisation to not be too loaded, this is specially try for pure HDD setups as it could be loaded specially when doing online upgrades and restarting services, also make sure your scrub speed and recovery speeds in the maintenance tab are not set above default as these can further strain pure HDD OSDs.
As i understand from your post, this is commercial setup serving your customers, please do consider our commercial support as we can dedicate more time to more involved issues. Good luck 🙂
Thanks for the info. Generally iSCSI is always faster, in your case NFS was 5 to 10 times faster! Most probable reason is because you are using pure HDDs and doing Veeam backups: pure HDD give low performance with small block size but are good at large sizes, by default Windows iSCSI initiator does have a 256KB block size limit by default, Veeam blocksize is 4 MB so it gets chopped in iSCSI but not in NFS, you can change the Windows iSCSI value from 256KB to 4MB and it will give similar performance to NFS. At small sizes on both iSCSI and NFS, iSCSI will be faster.
To your issue: It looks like a file size limitation error which is imposing a 1 TB limit on the file, i suggest you search Windows or Veeam they have some limit with NFS.
For your earlier issue when you did an upgrade, i believe it is not related to your recent Veeam error. If this happens again i would suggest looking at the syslog and ganesha log as well as any Ceph errors at that time. Also as suggested before to look into load on the system, like % disk utilisation to not be too loaded, this is specially try for pure HDD setups as it could be loaded specially when doing online upgrades and restarting services, also make sure your scrub speed and recovery speeds in the maintenance tab are not set above default as these can further strain pure HDD OSDs.
As i understand from your post, this is commercial setup serving your customers, please do consider our commercial support as we can dedicate more time to more involved issues. Good luck 🙂
Last edited on June 17, 2022, 1:42 pm by admin · #9
wid
47 Posts
June 19, 2022, 5:30 pmQuote from wid on June 19, 2022, 5:30 pmThanks for the tips, I will check iSCSI again. I also admit that I have searched and cannot find information on how to increase the blocksize to 4MB.
Back to the error:
The 1TB file limit is not a windows issue:
1) On Linux (clinet) using NFS with the 'dd' command I made a file that was supposed to be 2 TB. It didn't make the whole thing:
ls -la test-2tb.bin
-rw-r--r-- 1 nobody 4294967294 1099511627776 Jun 19:17 test-2tb.bin
ls -lah test-2tb.bin
-rw-r--r-- 1 nobody 4294967294 1.0T Jun 19:17:17 test-2tb.bin
2) Testing adding something to this file from the client, did not work:
dd if=/dev/zero bs=1M count=1024 >> test-2tb.bin
dd: error writing 'standard output': Input/output error
dd: closing output file 'standard output': File too large
3) Tests adding data from ceph cluster (automatically mounted in /mnt) also failed:
dd if=/dev/zero bs=1M count=1024 >> test-2tb.bin
dd: error writing 'standard output': File too large
1+0 records in
0+0 records out
61440 bytes (61 kB, 60 KiB) copied, 0.0652734 s, 941 kB/s
I'm pretty sure ceph after NFS doesn't allow you to make a file larger than 1 TB.
The cluster was created as a LAB for learning, sometimes I throw commercial stuff in there,
I wrote once about support, I got no answer.
However I'm willing to try again 🙂
Thanks for the tips, I will check iSCSI again. I also admit that I have searched and cannot find information on how to increase the blocksize to 4MB.
Back to the error:
The 1TB file limit is not a windows issue:
1) On Linux (clinet) using NFS with the 'dd' command I made a file that was supposed to be 2 TB. It didn't make the whole thing:
ls -la test-2tb.bin
-rw-r--r-- 1 nobody 4294967294 1099511627776 Jun 19:17 test-2tb.bin
ls -lah test-2tb.bin
-rw-r--r-- 1 nobody 4294967294 1.0T Jun 19:17:17 test-2tb.bin
2) Testing adding something to this file from the client, did not work:
dd if=/dev/zero bs=1M count=1024 >> test-2tb.bin
dd: error writing 'standard output': Input/output error
dd: closing output file 'standard output': File too large
3) Tests adding data from ceph cluster (automatically mounted in /mnt) also failed:
dd if=/dev/zero bs=1M count=1024 >> test-2tb.bin
dd: error writing 'standard output': File too large
1+0 records in
0+0 records out
61440 bytes (61 kB, 60 KiB) copied, 0.0652734 s, 941 kB/s
I'm pretty sure ceph after NFS doesn't allow you to make a file larger than 1 TB.
The cluster was created as a LAB for learning, sometimes I throw commercial stuff in there,
I wrote once about support, I got no answer.
However I'm willing to try again 🙂
Last edited on June 19, 2022, 5:31 pm by wid · #10
Pages: 1 2
NFS Problems, dropped connections, errors on client.
wid
47 Posts
Quote from wid on June 13, 2022, 8:16 pmHi,
newest Petasan 3.1.0
NFS Cluster,
All NFS customers, regularly have problems accessing NFS. IO errors pop up, connections are broken.Instance #1:
NFS4 status code: [0], nfs41_write failed on offset 9457664 with Hard I/O error
Error: nfs41_commit failed withInstance #2:
NFS4 status code: [0]
Reconnectable protocol device was closed.
Failed to upload disk. Skipped arguments: [>];
Agent failed to process method {DataTransfer.SyncDisk}.The problems started after the last update.
Petasan.log -> nothing there
ganesha.log -> shows the following errors:
10/06/2022 02:23:21 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26493] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4329a77370 fd 126 recv errno 104 (will set dead)
10/06/2022 02:23:21 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26486] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f437241b9d0 fd 126 recv errno 104 (will set dead)
10/06/2022 06:23:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26472] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436a6ff7e0 fd 113 recv errno 104 (will set dead)
10/06/2022 06:23:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26491] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43c0ccc860 fd 113 recv errno 104 (will set dead)
10/06/2022 08:41:45 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 50424171, old: 50424170; deq new: 50424170, old: 50424170
10/06/2022 10:24:04 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26483] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42e4623710 fd 110 recv errno 104 (will set dead)
10/06/2022 10:24:04 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26484] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43296be240 fd 110 recv errno 104 (will set dead)
10/06/2022 14:24:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26473] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4324c44840 fd 113 recv errno 104 (will set dead)
10/06/2022 14:24:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26493] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4329a77370 fd 113 recv errno 104 (will set dead)
10/06/2022 15:50:32 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53390976, old: 53390974; deq new: 53390974, old: 53390974
10/06/2022 16:15:10 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53411259, old: 53411258; deq new: 53411258, old: 53411258
10/06/2022 16:25:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53416913, old: 53416909; deq new: 53416909, old: 53416909
10/06/2022 16:33:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53422261, old: 53422257; deq new: 53422257, old: 53422257
10/06/2022 17:44:24 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53545156, old: 53545152; deq new: 53545150, old: 53545150
10/06/2022 17:46:30 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53549612, old: 53549608; deq new: 53549604, old: 53549604
10/06/2022 18:24:07 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26487] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4353926310 fd 110 recv errno 104 (will set dead)
10/06/2022 18:24:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26480] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42a48f1020 fd 113 recv errno 104 (will set dead)
10/06/2022 18:48:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53629651, old: 53629650; deq new: 53629650, old: 53629650
10/06/2022 19:08:58 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53652902, old: 53652898; deq new: 53652898, old: 53652898
10/06/2022 20:06:15 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53724992, old: 53724991; deq new: 53724991, old: 53724991
10/06/2022 20:41:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53776185, old: 53776184; deq new: 53776184, old: 53776184
10/06/2022 21:32:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53849148, old: 53849146; deq new: 53849146, old: 53849146
10/06/2022 21:44:46 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53891215, old: 53891211; deq new: 53891211, old: 53891211
10/06/2022 21:47:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53894738, old: 53894734; deq new: 53894734, old: 53894734
10/06/2022 21:48:39 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53896871, old: 53896867; deq new: 53896867, old: 53896867
10/06/2022 22:24:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26495] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4354031ef0 fd 113 recv errno 104 (will set dead)
10/06/2022 22:24:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26486] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4324c413c0 fd 113 recv errno 104 (will set dead)
11/06/2022 01:21:38 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 55741738, old: 55741737; deq new: 55741737, old: 55741737
11/06/2022 02:24:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26465] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4359f58750 fd 113 recv errno 104 (will set dead)
11/06/2022 02:24:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26486] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4373137010 fd 113 recv errno 104 (will set dead)
11/06/2022 03:10:03 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 56642823, old: 56642822; deq new: 56642821, old: 56642821
11/06/2022 05:25:01 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57567777, old: 57567776; deq new: 57567772, old: 57567772
11/06/2022 05:53:20 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57597290, old: 57597289; deq new: 57597285, old: 57597285
11/06/2022 06:24:46 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26659] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4351343110 fd 109 recv errno 104 (will set dead)
11/06/2022 06:24:46 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26656] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f429a68d940 fd 109 recv errno 104 (will set dead)
11/06/2022 07:47:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57730421, old: 57730417; deq new: 57730417, old: 57730417
11/06/2022 08:27:29 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57791396, old: 57791392; deq new: 57791392, old: 57791392
11/06/2022 08:36:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57803275, old: 57803272; deq new: 57803271, old: 57803271
11/06/2022 09:31:13 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57869356, old: 57869352; deq new: 57869352, old: 57869352
11/06/2022 10:24:53 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26898] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43091a14c0 fd 110 recv errno 104 (will set dead)
11/06/2022 10:24:53 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26883] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c581e6c0 fd 110 recv errno 104 (will set dead)
11/06/2022 10:54:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26961] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42fcfba4e0 fd 129 recv errno 104 (will set dead)
11/06/2022 13:41:04 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27140] nfs4_Compound :NFS4 :CRIT :Bad Minor Version 29801
11/06/2022 14:24:57 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27117] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c930b980 fd 110 recv errno 104 (will set dead)
11/06/2022 14:24:57 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27174] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42a9e2cfa0 fd 110 recv errno 104 (will set dead)
11/06/2022 14:37:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58124662, old: 58124658; deq new: 58124652, old: 58124652
11/06/2022 18:01:32 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58476288, old: 58476284; deq new: 58476284, old: 58476284
11/06/2022 18:25:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27280] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42abfb2900 fd 113 recv errno 104 (will set dead)
11/06/2022 18:25:14 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27210] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42fca68f70 fd 113 recv errno 104 (will set dead)
11/06/2022 18:27:53 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58525148, old: 58525147; deq new: 58525143, old: 58525143
11/06/2022 19:06:25 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58579809, old: 58579805; deq new: 58579805, old: 58579805
11/06/2022 19:24:56 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58610210, old: 58610206; deq new: 58610206, old: 58610206
11/06/2022 22:25:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27440] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43260ac5a0 fd 132 recv errno 104 (will set dead)
11/06/2022 22:25:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27436] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f432806e130 fd 132 recv errno 104 (will set dead)
12/06/2022 02:25:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27426] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436a9cbf30 fd 129 recv errno 104 (will set dead)
12/06/2022 02:25:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27455] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f434d6c9250 fd 129 recv errno 104 (will set dead)
12/06/2022 06:25:25 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27417] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f429b2baa20 fd 113 recv errno 104 (will set dead)
12/06/2022 06:25:25 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27457] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f432447f090 fd 113 recv errno 104 (will set dead)
12/06/2022 10:25:52 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27528] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4356a9d970 fd 113 recv errno 104 (will set dead)
12/06/2022 10:25:54 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27528] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42e800ae30 fd 113 recv errno 104 (will set dead)
12/06/2022 14:25:55 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27585] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43091a14c0 fd 110 recv errno 104 (will set dead)
12/06/2022 14:25:55 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27586] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436c3b38d0 fd 110 recv errno 104 (will set dead)
12/06/2022 16:10:30 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 66695226, old: 66695225; deq new: 66695224, old: 66695224
12/06/2022 18:26:17 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27629] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436a6ff7e0 fd 124 recv errno 104 (will set dead)
12/06/2022 18:26:17 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27549] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42a621db00 fd 124 recv errno 104 (will set dead)
12/06/2022 19:24:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 67506309, old: 67506308; deq new: 67506307, old: 67506307
12/06/2022 20:24:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 67627961, old: 67627960; deq new: 67627960, old: 67627960
12/06/2022 22:26:20 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27707] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c4fd6fe0 fd 129 recv errno 104 (will set dead)
12/06/2022 22:26:20 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27702] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c930b980 fd 129 recv errno 104 (will set dead)
13/06/2022 00:35:38 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69302078, old: 69302074; deq new: 69302074, old: 69302074
13/06/2022 01:13:03 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27917] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f435499f060 fd 131 recv errno 104 (will set dead)
13/06/2022 02:26:37 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28044] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43b645e010 fd 126 recv errno 104 (will set dead)
13/06/2022 02:26:37 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28038] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a426f180 fd 126 recv errno 104 (will set dead)
13/06/2022 06:26:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28406] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a426f180 fd 126 recv errno 104 (will set dead)
13/06/2022 06:26:34 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28400] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4347816570 fd 126 recv errno 104 (will set dead)
13/06/2022 10:26:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28717] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a426f180 fd 126 recv errno 104 (will set dead)
13/06/2022 10:26:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28767] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4373137010 fd 126 recv errno 104 (will set dead)
13/06/2022 14:27:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29133] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4373137010 fd 124 recv errno 104 (will set dead)
13/06/2022 14:27:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29129] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c930b980 fd 124 recv errno 104 (will set dead)
13/06/2022 15:39:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29237] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42d4006c00 fd 114 recv errno 104 (will set dead)
13/06/2022 15:39:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29235] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f435499f060 fd 110 recv errno 104 (will set dead)
13/06/2022 15:39:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29231] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c5eab500 fd 58 recv errno 104 (will set dead)
13/06/2022 15:39:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29241] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4324c44840 fd 124 recv errno 104 (will set dead)
13/06/2022 15:39:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29235] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43730d4a80 fd 124 recv errno 104 (will set dead)
13/06/2022 16:02:10 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69335717, old: 69335713; deq new: 69335713, old: 69335713
13/06/2022 17:06:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69409040, old: 69409036; deq new: 69409036, old: 69409036
13/06/2022 17:35:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69447453, old: 69447449; deq new: 69447449, old: 69447449
13/06/2022 17:36:52 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69448544, old: 69448540; deq new: 69448540, old: 69448540
13/06/2022 19:40:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29458] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f434726c630 fd 108 recv errno 104 (will set dead)
13/06/2022 19:40:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29642] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a90a9390 fd 108 recv errno 104 (will set dead)
Hi,
newest Petasan 3.1.0
NFS Cluster,
All NFS customers, regularly have problems accessing NFS. IO errors pop up, connections are broken.
Instance #1:
NFS4 status code: [0], nfs41_write failed on offset 9457664 with Hard I/O error
Error: nfs41_commit failed with
Instance #2:
NFS4 status code: [0]
Reconnectable protocol device was closed.
Failed to upload disk. Skipped arguments: [>];
Agent failed to process method {DataTransfer.SyncDisk}.
The problems started after the last update.
Petasan.log -> nothing there
ganesha.log -> shows the following errors:
10/06/2022 02:23:21 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26493] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4329a77370 fd 126 recv errno 104 (will set dead)
10/06/2022 02:23:21 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26486] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f437241b9d0 fd 126 recv errno 104 (will set dead)
10/06/2022 06:23:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26472] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436a6ff7e0 fd 113 recv errno 104 (will set dead)
10/06/2022 06:23:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26491] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43c0ccc860 fd 113 recv errno 104 (will set dead)
10/06/2022 08:41:45 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 50424171, old: 50424170; deq new: 50424170, old: 50424170
10/06/2022 10:24:04 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26483] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42e4623710 fd 110 recv errno 104 (will set dead)
10/06/2022 10:24:04 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26484] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43296be240 fd 110 recv errno 104 (will set dead)
10/06/2022 14:24:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26473] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4324c44840 fd 113 recv errno 104 (will set dead)
10/06/2022 14:24:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26493] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4329a77370 fd 113 recv errno 104 (will set dead)
10/06/2022 15:50:32 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53390976, old: 53390974; deq new: 53390974, old: 53390974
10/06/2022 16:15:10 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53411259, old: 53411258; deq new: 53411258, old: 53411258
10/06/2022 16:25:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53416913, old: 53416909; deq new: 53416909, old: 53416909
10/06/2022 16:33:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53422261, old: 53422257; deq new: 53422257, old: 53422257
10/06/2022 17:44:24 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53545156, old: 53545152; deq new: 53545150, old: 53545150
10/06/2022 17:46:30 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53549612, old: 53549608; deq new: 53549604, old: 53549604
10/06/2022 18:24:07 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26487] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4353926310 fd 110 recv errno 104 (will set dead)
10/06/2022 18:24:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26480] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42a48f1020 fd 113 recv errno 104 (will set dead)
10/06/2022 18:48:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53629651, old: 53629650; deq new: 53629650, old: 53629650
10/06/2022 19:08:58 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53652902, old: 53652898; deq new: 53652898, old: 53652898
10/06/2022 20:06:15 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53724992, old: 53724991; deq new: 53724991, old: 53724991
10/06/2022 20:41:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53776185, old: 53776184; deq new: 53776184, old: 53776184
10/06/2022 21:32:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53849148, old: 53849146; deq new: 53849146, old: 53849146
10/06/2022 21:44:46 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53891215, old: 53891211; deq new: 53891211, old: 53891211
10/06/2022 21:47:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53894738, old: 53894734; deq new: 53894734, old: 53894734
10/06/2022 21:48:39 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 53896871, old: 53896867; deq new: 53896867, old: 53896867
10/06/2022 22:24:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26495] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4354031ef0 fd 113 recv errno 104 (will set dead)
10/06/2022 22:24:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26486] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4324c413c0 fd 113 recv errno 104 (will set dead)
11/06/2022 01:21:38 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 55741738, old: 55741737; deq new: 55741737, old: 55741737
11/06/2022 02:24:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26465] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4359f58750 fd 113 recv errno 104 (will set dead)
11/06/2022 02:24:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26486] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4373137010 fd 113 recv errno 104 (will set dead)
11/06/2022 03:10:03 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 56642823, old: 56642822; deq new: 56642821, old: 56642821
11/06/2022 05:25:01 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57567777, old: 57567776; deq new: 57567772, old: 57567772
11/06/2022 05:53:20 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57597290, old: 57597289; deq new: 57597285, old: 57597285
11/06/2022 06:24:46 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26659] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4351343110 fd 109 recv errno 104 (will set dead)
11/06/2022 06:24:46 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26656] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f429a68d940 fd 109 recv errno 104 (will set dead)
11/06/2022 07:47:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57730421, old: 57730417; deq new: 57730417, old: 57730417
11/06/2022 08:27:29 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57791396, old: 57791392; deq new: 57791392, old: 57791392
11/06/2022 08:36:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57803275, old: 57803272; deq new: 57803271, old: 57803271
11/06/2022 09:31:13 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 57869356, old: 57869352; deq new: 57869352, old: 57869352
11/06/2022 10:24:53 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26898] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43091a14c0 fd 110 recv errno 104 (will set dead)
11/06/2022 10:24:53 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26883] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c581e6c0 fd 110 recv errno 104 (will set dead)
11/06/2022 10:54:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_26961] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42fcfba4e0 fd 129 recv errno 104 (will set dead)
11/06/2022 13:41:04 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27140] nfs4_Compound :NFS4 :CRIT :Bad Minor Version 29801
11/06/2022 14:24:57 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27117] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c930b980 fd 110 recv errno 104 (will set dead)
11/06/2022 14:24:57 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27174] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42a9e2cfa0 fd 110 recv errno 104 (will set dead)
11/06/2022 14:37:08 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58124662, old: 58124658; deq new: 58124652, old: 58124652
11/06/2022 18:01:32 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58476288, old: 58476284; deq new: 58476284, old: 58476284
11/06/2022 18:25:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27280] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42abfb2900 fd 113 recv errno 104 (will set dead)
11/06/2022 18:25:14 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27210] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42fca68f70 fd 113 recv errno 104 (will set dead)
11/06/2022 18:27:53 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58525148, old: 58525147; deq new: 58525143, old: 58525143
11/06/2022 19:06:25 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58579809, old: 58579805; deq new: 58579805, old: 58579805
11/06/2022 19:24:56 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 58610210, old: 58610206; deq new: 58610206, old: 58610206
11/06/2022 22:25:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27440] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43260ac5a0 fd 132 recv errno 104 (will set dead)
11/06/2022 22:25:16 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27436] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f432806e130 fd 132 recv errno 104 (will set dead)
12/06/2022 02:25:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27426] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436a9cbf30 fd 129 recv errno 104 (will set dead)
12/06/2022 02:25:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27455] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f434d6c9250 fd 129 recv errno 104 (will set dead)
12/06/2022 06:25:25 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27417] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f429b2baa20 fd 113 recv errno 104 (will set dead)
12/06/2022 06:25:25 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27457] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f432447f090 fd 113 recv errno 104 (will set dead)
12/06/2022 10:25:52 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27528] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4356a9d970 fd 113 recv errno 104 (will set dead)
12/06/2022 10:25:54 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27528] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42e800ae30 fd 113 recv errno 104 (will set dead)
12/06/2022 14:25:55 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27585] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43091a14c0 fd 110 recv errno 104 (will set dead)
12/06/2022 14:25:55 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27586] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436c3b38d0 fd 110 recv errno 104 (will set dead)
12/06/2022 16:10:30 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 66695226, old: 66695225; deq new: 66695224, old: 66695224
12/06/2022 18:26:17 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27629] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f436a6ff7e0 fd 124 recv errno 104 (will set dead)
12/06/2022 18:26:17 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27549] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42a621db00 fd 124 recv errno 104 (will set dead)
12/06/2022 19:24:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 67506309, old: 67506308; deq new: 67506307, old: 67506307
12/06/2022 20:24:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 67627961, old: 67627960; deq new: 67627960, old: 67627960
12/06/2022 22:26:20 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27707] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c4fd6fe0 fd 129 recv errno 104 (will set dead)
12/06/2022 22:26:20 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27702] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c930b980 fd 129 recv errno 104 (will set dead)
13/06/2022 00:35:38 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69302078, old: 69302074; deq new: 69302074, old: 69302074
13/06/2022 01:13:03 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_27917] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f435499f060 fd 131 recv errno 104 (will set dead)
13/06/2022 02:26:37 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28044] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43b645e010 fd 126 recv errno 104 (will set dead)
13/06/2022 02:26:37 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28038] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a426f180 fd 126 recv errno 104 (will set dead)
13/06/2022 06:26:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28406] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a426f180 fd 126 recv errno 104 (will set dead)
13/06/2022 06:26:34 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28400] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4347816570 fd 126 recv errno 104 (will set dead)
13/06/2022 10:26:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28717] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a426f180 fd 126 recv errno 104 (will set dead)
13/06/2022 10:26:49 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_28767] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4373137010 fd 126 recv errno 104 (will set dead)
13/06/2022 14:27:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29133] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4373137010 fd 124 recv errno 104 (will set dead)
13/06/2022 14:27:11 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29129] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c930b980 fd 124 recv errno 104 (will set dead)
13/06/2022 15:39:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29237] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42d4006c00 fd 114 recv errno 104 (will set dead)
13/06/2022 15:39:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29235] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f435499f060 fd 110 recv errno 104 (will set dead)
13/06/2022 15:39:00 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29231] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f42c5eab500 fd 58 recv errno 104 (will set dead)
13/06/2022 15:39:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29241] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f4324c44840 fd 124 recv errno 104 (will set dead)
13/06/2022 15:39:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29235] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43730d4a80 fd 124 recv errno 104 (will set dead)
13/06/2022 16:02:10 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69335717, old: 69335713; deq new: 69335713, old: 69335713
13/06/2022 17:06:33 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69409040, old: 69409036; deq new: 69409036, old: 69409036
13/06/2022 17:35:18 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69447453, old: 69447449; deq new: 69447449, old: 69447449
13/06/2022 17:36:52 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[dbus_heartbeat] nfs_health :DBUS :WARN :Health status is unhealthy. enq new: 69448544, old: 69448540; deq new: 69448540, old: 69448540
13/06/2022 19:40:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29458] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f434726c630 fd 108 recv errno 104 (will set dead)
13/06/2022 19:40:41 : epoch 62933cab : NFS-172-30-0-172 : ganesha.nfsd-152[svc_29642] rpc :TIRPC :EVENT :svc_vc_wait: 0x7f43a90a9390 fd 108 recv errno 104 (will set dead)
admin
2,930 Posts
Quote from admin on June 14, 2022, 9:36 amDo you see any errors in syslog
Was the cluster healthy ? from PG status charts were any PGs not clean ?
Do you see high utlization in node stats for % disk or % cpu ?
Is the NFS Status always up in UI ? does it show down or grace status ?
Do clients connect using the public NFS ips ? can you specify -o nfsvers=4.1 when mounting the clients.
Do you see any errors in syslog
Was the cluster healthy ? from PG status charts were any PGs not clean ?
Do you see high utlization in node stats for % disk or % cpu ?
Is the NFS Status always up in UI ? does it show down or grace status ?
Do clients connect using the public NFS ips ? can you specify -o nfsvers=4.1 when mounting the clients.
wid
47 Posts
Quote from wid on June 14, 2022, 7:53 pmNo errors in syslog,
cluster is healthy,
PG Was moved (due hdd prepared to be removed)
CPU load 1/16, Mem used 50%.
NFS Status - probably ok, not ctched any errors.
Clients (2 veeam backup servers) are connected as 4.1 by default.
cluster:
health: HEALTH_OKservices:
mon: 3 daemons, quorum ceph03,ceph01,ceph02 (age 3d)
mgr: ceph02(active, since 2w), standbys: ceph01, ceph03
mds: fs-mit:1 {0=ceph01=up:active}
osd: 26 osds: 26 up (since 25h), 26 in (since 4d)data:
pools: 7 pools, 697 pgs
objects: 15.41M objects, 34 TiB
usage: 62 TiB used, 66 TiB / 128 TiB avail
pgs: 690 active+clean
7 active+clean+scrubbing+deep
No errors in syslog,
cluster is healthy,
PG Was moved (due hdd prepared to be removed)
CPU load 1/16, Mem used 50%.
NFS Status - probably ok, not ctched any errors.
Clients (2 veeam backup servers) are connected as 4.1 by default.
cluster:
health: HEALTH_OKservices:
mon: 3 daemons, quorum ceph03,ceph01,ceph02 (age 3d)
mgr: ceph02(active, since 2w), standbys: ceph01, ceph03
mds: fs-mit:1 {0=ceph01=up:active}
osd: 26 osds: 26 up (since 25h), 26 in (since 4d)data:
pools: 7 pools, 697 pgs
objects: 15.41M objects, 34 TiB
usage: 62 TiB used, 66 TiB / 128 TiB avail
pgs: 690 active+clean
7 active+clean+scrubbing+deep
wid
47 Posts
Quote from wid on June 17, 2022, 10:44 amNew type of error from veeam:
16.06.2022 23:43:35 :: Processing XXX Error: NFS4 status code: [0]
nfs41_write failed on offset 1099511627264 with The file size exceeds the limit allowed and cannot be saved
Failed to download disk 'XXXXXX'.
Reconnectable protocol device was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
New type of error from veeam:
16.06.2022 23:43:35 :: Processing XXX Error: NFS4 status code: [0]
nfs41_write failed on offset 1099511627264 with The file size exceeds the limit allowed and cannot be saved
Failed to download disk 'XXXXXX'.
Reconnectable protocol device was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
admin
2,930 Posts
Quote from admin on June 17, 2022, 12:38 pm"The file size exceeds the limit allowed and cannot be saved" looks more like a preset file size limit of 1099511627264 offset ( 1TB ) and not an error at any random offset. What OS is the client ( Windows / Linux / kernel version ?). Also what is the file size you are writing ?
"The file size exceeds the limit allowed and cannot be saved" looks more like a preset file size limit of 1099511627264 offset ( 1TB ) and not an error at any random offset. What OS is the client ( Windows / Linux / kernel version ?). Also what is the file size you are writing ?
wid
47 Posts
Quote from wid on June 17, 2022, 12:42 pmWindows 2019 std,
Writing software is Veeam (10.x) planned upgrade to latest version.After iSCSI there were no write issues and no errors shown (there were however issues with performance) NFS is about 25x faster than iSCSI was on Petasan.
Windows 2019 std,
Writing software is Veeam (10.x) planned upgrade to latest version.
After iSCSI there were no write issues and no errors shown (there were however issues with performance) NFS is about 25x faster than iSCSI was on Petasan.
admin
2,930 Posts
Quote from admin on June 17, 2022, 12:57 pmWhat is the file size you are writing ?
What speeds were you getting with iSCSI ?
What is the hardware : SSDs/HDDs ?
What is the file size you are writing ?
What speeds were you getting with iSCSI ?
What is the hardware : SSDs/HDDs ?
wid
47 Posts
Quote from wid on June 17, 2022, 1:01 pmiSCSI - 20 MB/s
NFS - 100-200 MB/sHardware:
Dell 2 x R510 / 1 x R730Xd (32/64 GB RAM) not used 100%
26 x HDD WD RED PRO 6TB.
iSCSI - 20 MB/s
NFS - 100-200 MB/s
Hardware:
Dell 2 x R510 / 1 x R730Xd (32/64 GB RAM) not used 100%
26 x HDD WD RED PRO 6TB.
admin
2,930 Posts
Quote from admin on June 17, 2022, 1:40 pmThanks for the info. Generally iSCSI is always faster, in your case NFS was 5 to 10 times faster! Most probable reason is because you are using pure HDDs and doing Veeam backups: pure HDD give low performance with small block size but are good at large sizes, by default Windows iSCSI initiator does have a 256KB block size limit by default, Veeam blocksize is 4 MB so it gets chopped in iSCSI but not in NFS, you can change the Windows iSCSI value from 256KB to 4MB and it will give similar performance to NFS. At small sizes on both iSCSI and NFS, iSCSI will be faster.
To your issue: It looks like a file size limitation error which is imposing a 1 TB limit on the file, i suggest you search Windows or Veeam they have some limit with NFS.
For your earlier issue when you did an upgrade, i believe it is not related to your recent Veeam error. If this happens again i would suggest looking at the syslog and ganesha log as well as any Ceph errors at that time. Also as suggested before to look into load on the system, like % disk utilisation to not be too loaded, this is specially try for pure HDD setups as it could be loaded specially when doing online upgrades and restarting services, also make sure your scrub speed and recovery speeds in the maintenance tab are not set above default as these can further strain pure HDD OSDs.
As i understand from your post, this is commercial setup serving your customers, please do consider our commercial support as we can dedicate more time to more involved issues. Good luck 🙂
Thanks for the info. Generally iSCSI is always faster, in your case NFS was 5 to 10 times faster! Most probable reason is because you are using pure HDDs and doing Veeam backups: pure HDD give low performance with small block size but are good at large sizes, by default Windows iSCSI initiator does have a 256KB block size limit by default, Veeam blocksize is 4 MB so it gets chopped in iSCSI but not in NFS, you can change the Windows iSCSI value from 256KB to 4MB and it will give similar performance to NFS. At small sizes on both iSCSI and NFS, iSCSI will be faster.
To your issue: It looks like a file size limitation error which is imposing a 1 TB limit on the file, i suggest you search Windows or Veeam they have some limit with NFS.
For your earlier issue when you did an upgrade, i believe it is not related to your recent Veeam error. If this happens again i would suggest looking at the syslog and ganesha log as well as any Ceph errors at that time. Also as suggested before to look into load on the system, like % disk utilisation to not be too loaded, this is specially try for pure HDD setups as it could be loaded specially when doing online upgrades and restarting services, also make sure your scrub speed and recovery speeds in the maintenance tab are not set above default as these can further strain pure HDD OSDs.
As i understand from your post, this is commercial setup serving your customers, please do consider our commercial support as we can dedicate more time to more involved issues. Good luck 🙂
wid
47 Posts
Quote from wid on June 19, 2022, 5:30 pmThanks for the tips, I will check iSCSI again. I also admit that I have searched and cannot find information on how to increase the blocksize to 4MB.
Back to the error:
The 1TB file limit is not a windows issue:
1) On Linux (clinet) using NFS with the 'dd' command I made a file that was supposed to be 2 TB. It didn't make the whole thing:ls -la test-2tb.bin
-rw-r--r-- 1 nobody 4294967294 1099511627776 Jun 19:17 test-2tb.bin
ls -lah test-2tb.bin
-rw-r--r-- 1 nobody 4294967294 1.0T Jun 19:17:17 test-2tb.bin2) Testing adding something to this file from the client, did not work:
dd if=/dev/zero bs=1M count=1024 >> test-2tb.bin
dd: error writing 'standard output': Input/output error
dd: closing output file 'standard output': File too large3) Tests adding data from ceph cluster (automatically mounted in /mnt) also failed:
dd if=/dev/zero bs=1M count=1024 >> test-2tb.bin
dd: error writing 'standard output': File too large
1+0 records in
0+0 records out
61440 bytes (61 kB, 60 KiB) copied, 0.0652734 s, 941 kB/sI'm pretty sure ceph after NFS doesn't allow you to make a file larger than 1 TB.
The cluster was created as a LAB for learning, sometimes I throw commercial stuff in there,
I wrote once about support, I got no answer.
However I'm willing to try again 🙂
Thanks for the tips, I will check iSCSI again. I also admit that I have searched and cannot find information on how to increase the blocksize to 4MB.
Back to the error:
The 1TB file limit is not a windows issue:
1) On Linux (clinet) using NFS with the 'dd' command I made a file that was supposed to be 2 TB. It didn't make the whole thing:
ls -la test-2tb.bin
-rw-r--r-- 1 nobody 4294967294 1099511627776 Jun 19:17 test-2tb.bin
ls -lah test-2tb.bin
-rw-r--r-- 1 nobody 4294967294 1.0T Jun 19:17:17 test-2tb.bin
2) Testing adding something to this file from the client, did not work:
dd if=/dev/zero bs=1M count=1024 >> test-2tb.bin
dd: error writing 'standard output': Input/output error
dd: closing output file 'standard output': File too large
3) Tests adding data from ceph cluster (automatically mounted in /mnt) also failed:
dd if=/dev/zero bs=1M count=1024 >> test-2tb.bin
dd: error writing 'standard output': File too large
1+0 records in
0+0 records out
61440 bytes (61 kB, 60 KiB) copied, 0.0652734 s, 941 kB/s
I'm pretty sure ceph after NFS doesn't allow you to make a file larger than 1 TB.
The cluster was created as a LAB for learning, sometimes I throw commercial stuff in there,
I wrote once about support, I got no answer.
However I'm willing to try again 🙂