Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

one of the node has been failed and then cannot start the old iscsi disk


Yipkaiwing
18 Posts
October 10, 2018, 1:40 am

Dear Admin

Please help me, my storage has been crashed.

Server info:

Management Node 1-3 are VM guest. There are 4 units of storage nodes,  each has 7 SSD,  a total of 7TB in each node.

On last Sunday, on Node 5, system OS disk is failed.  After the rebuilding, it seems that the storage pool is normal.

https://ibb.co/c4hub9

https://ibb.co/mQ2Sw9

But I cannot start the iscsi disk properly. After all node reboot, it can be started. But The VM server could not connect it functionally. In addition, when I try to stop and start the iscsi disk again, it will be hangs up.

https://ibb.co/bC1y3p

I check with the syslog, I found the below error:

70999.311179] Unable to locate Target IQN: iqn.2016-05.com.petasan:00002 in Storage Node
Oct 10 06:25:49 node4 kernel: [70999.311187] iSCSI Login negotiation failed.
Oct 10 06:25:52 node4 kernel: [71002.329318] Unable to locate Target IQN: iqn.2016-05.com.petasan:00002 in Storage Node
Oct 10 06:25:52 node4 kernel: [71002.329326] iSCSI Login negotiation failed.
Oct 10 06:25:55 node4 kernel: [71005.341196] Unable to locate Target IQN: iqn.2016-05.com.petasan:00002 in Storage Node
Oct 10 06:25:55 node4 kernel: [71005.341202] iSCSI Login negotiation failed.
Oct 10 06:25:58 node4 kernel: [71008.354784] Unable to locate Target IQN: iqn.2016-05.com.petasan:00002 in Storage Node
Oct 10 06:25:58 node4 kernel: [71008.354792] iSCSI Login negotiation failed.

https://ibb.co/cCWZ9U

https://ibb.co/gHC7UU

After this incident, I try to create the new testing disk which is 00003. All operation is normal. It can connect it via Windows 2016 server iscci initiator.

disk 00001 and 00002 are the old production disks. I need to retrieve the production data, please help me.

Ben

#1

admin
2,930 Posts
October 10, 2018, 6:13 am

What version of PetaSAN are you using ?

What is the current cluster status

ceph status --cluster  CLUSTER_NAME

rbd ls  --cluster  CLUSTER_NAME

You can connect and read/write to disk 3  but not 1 and 2 ?

In the iSCSI disk list : what is the status of disk 1/2 (started/starting/stopping)  ? disk 3 is started ?

#2

Yipkaiwing
18 Posts
October 10, 2018, 7:21 am

Dear Admin

The version is 2.1

The current cluster status is

root@node1:~# ceph status --cluster ceph
cluster:
id: 2239b97c-01e2-47f7-af5d-16bc69e312d3
health: HEALTH_WARN
Reduced data availability: 166 pgs inactive, 166 pgs down

 

root@node1:~# rbd ls --cluster ceph
parse error setting 'bluestore_cache_size_ssd' to '1GB' (The option value '1G' contains invalid digits)
image-00001
image-00002
image-00003
root@node1:~#

services:
mon: 3 daemons, quorum node1,node2,node3
mgr: node1(active), standbys: node2
osd: 28 osds: 21 up, 21 in

data:
pools: 1 pools, 1024 pgs
objects: 661k objects, 2644 GB
usage: 4882 GB used, 15146 GB / 20029 GB avail
pgs: 16.211% pgs not active
858 active+clean
166 down

io:
client: 255 B/s rd, 0 op/s rd, 0 op/s wr

You can connect and read/write to disk 3  but not 1 and 2 ?

The answer is yes. Disk 3 is more stable and running normally. But disk 1 and disk 2 are always hangs, not function normally.

In the iSCSI disk list : what is the status of disk 1/2 (started/starting/stopping)  ? disk 3 is started ?

The status (started/starting/stopping) are hangs up, no response. Unless I reboot the server manually.

Please help.

Ben

 

Last edited on October 10, 2018, 7:30 am by Yipkaiwing · #3

admin
2,930 Posts
October 10, 2018, 8:07 am

The main problem is the 166 down pgs. Under normal case they would be degraded for a while and Ceph would start its recovery until they become active/clean again.

Do you see any maxed out cpu/ram resources on any node going back to when the failure ocured or after you rebooted ?

What is the output of:

ceph osd pool get rbd size --cluster CLUSTER_NAME
ceph osd pool get rbd min_size --cluster CLUSTER_NAME

ceph pg dump --cluster CLUSTER_NAME | grep down

 

The recovery should happen even if node 5  and all its OSDs are down, but it is also strange why its OSDs are still not up since they could help with the recovery,  try to bring them up  manually  on node 5 via

/usr/lib/ceph/ceph-osd-prestart.sh --cluster CLUSTER_NAME --id OSD_ID
/usr/bin/ceph-osd -f --cluster CLUSTER_NAME --id OSD_ID --setuser ceph --setgroup ceph

where  OSD_ID is the osd id, do you get any errors on the console screen ?

 

Last edited on October 10, 2018, 8:40 am by admin · #4

Yipkaiwing
18 Posts
October 10, 2018, 8:52 am

Dear Admin

Thanks for your help.

Initially, the recovery is very unstable. The node are up and down frequently. Then I change the RAM setting "bluestore_cache_size_ssd" to 1 GB Ram. The storage rebuilding is become normal and smooth.

root@node1:~# ceph osd pool get rbd min_size --cluster ceph
min_size: 1
root@node1:~# ceph osd pool get rbd size --cluster ceph
size: 2

root@node1:~# ceph pg dump --cluster ceph | grep down
dumped all
1.3fe 695 0 0 0 0 2913992704 1587 1587 down 2018-10-10 08:59:52.396458 209'285987 4325:525798 [6,15] 6 [6,15] 6 206'285660 2018-10-07 02:42:09.022545 206'251240 2018-10-04 20:36:18.485813 0
1.3fa 621 0 0 0 0 2601517056 1557 1557 down 2018-10-10 08:59:52.365265 209'217058 4325:434535 [3,15] 3 [3,15] 3 206'208969 2018-10-06 22:54:47.044933 206'193807 2018-10-04 20:43:47.645417 0
1.3f9 678 0 0 0 0 2843738112 1531 1531 down 2018-10-10 08:59:55.442540 209'478732 4325:736308 [6,14] 6 [6,14] 6 206'464175 2018-10-06 00:35:45.216180 206'464175 2018-10-06 00:35:45.216180 0
1.3f5 669 0 0 0 0 2802843648 1551 1551 down 2018-10-10 08:59:55.458117 209'231051 4325:704699 [3,22] 3 [3,22] 3 206'211586 2018-10-06 20:36:44.597653 206'211586 2018-10-06 20:36:44.597653 0
1.3f2 668 0 0 0 0 2796814336 1532 1532 down 2018-10-10 08:59:55.462709 206'116432 4325:289516 [5,14] 5 [5,14] 5 206'113196 2018-10-05 21:41:00.453208 206'113196 2018-10-05 21:41:00.453208 0
1.3ef 633 0 0 0 0 2654994432 1594 1594 down 2018-10-10 08:59:55.444967 206'234394 4325:447837 [0,24] 0 [0,24] 0 206'234067 2018-10-06 20:28:46.715083 206'227053 2018-10-04 20:10:37.994817 0
1.3e8 659 0 0 0 0 2764046336 1547 1547 down 2018-10-10 08:59:55.468022 206'110948 4325:270651 [2,19] 2 [2,19] 2 206'107100 2018-10-06 01:36:18.857828 206'107100 2018-10-06 01:36:18.857828 0
1.3e7 661 0 0 0 0 2772434944 1524 1524 down 2018-10-10 08:59:55.453708 209'240424 4325:399110 [1,22] 1 [1,22] 1 206'239564 2018-10-06 20:17:25.928839 206'239564 2018-10-06 20:17:25.928839 0
1.3ca 676 0 0 0 0 2832203776 1557 1557 down 2018-10-10 08:59:53.413972 209'213557 4325:511431 [6,19] 6 [6,19] 6 206'212915 2018-10-07 00:49:19.206082 202'192568 2018-10-01 05:42:10.962306 0
1.3c1 675 0 0 0 0 2831155200 1505 1505 down 2018-10-10 08:59:55.441479 206'249405 4325:440935 [6,20] 6 [6,20] 6 206'248254 2018-10-06 20:27:01.332258 206'215681 2018-10-03 23:34:58.343314 0
1.3bd 669 0 0 0 0 2804940800 1559 1559 down 2018-10-10 08:59:54.420952 209'361859 4325:544105 [3,27] 3 [3,27] 3 206'357439 2018-10-06 21:26:03.441301 206'341738 2018-10-05 00:40:47.086800 0
1.3bb 671 0 0 0 0 2811691008 1574 1574 down 2018-10-10 08:59:55.469924 206'270174 4325:460349 [5,14] 5 [5,14] 5 206'269445 2018-10-06 20:58:15.752109 206'248503 2018-10-04 21:18:04.116811 0
1.3b4 684 0 0 0 0 2868903936 1594 1594 down 2018-10-10 08:59:55.442361 206'381894 4325:777333 [6,25] 6 [6,25] 6 206'377008 2018-10-06 21:27:38.752020 206'329973 2018-10-01 20:43:26.246690 0
1.3a2 677 0 0 0 0 2839543808 1506 1506 down 2018-10-10 08:59:55.450889 206'294006 4325:580579 [0,26] 0 [0,26] 0 206'293208 2018-10-06 21:20:49.866443 206'256789 2018-10-03 20:17:46.979862 0
1.39f 616 0 0 0 0 2579628032 1551 1551 down 2018-10-10 08:59:55.458286 209'1356752 4325:1629521 [4,20] 4 [4,20] 4 206'1302086 2018-10-06 05:32:31.195837 206'1116370 2018-10-01 20:21:51.722303 0
1.39a 652 0 0 0 0 2730360832 1537 1537 down 2018-10-10 08:59:52.402893 209'230938 4325:401000 [6,22] 6 [6,22] 6 206'226926 2018-10-07 01:45:11.172018 206'226926 2018-10-07 01:45:11.172018 0
1.399 591 0 0 0 0 2478833664 1506 1506 down 2018-10-10 08:59:55.459715 209'205806 4325:348896 [1,20] 1 [1,20] 1 206'205109 2018-10-06 21:11:46.232354 206'205109 2018-10-06 21:11:46.232354 0
1.398 653 0 0 0 0 2732785664 1582 1582 down 2018-10-10 08:59:55.469287 206'241783 4325:476350 [5,24] 5 [5,24] 5 206'241319 2018-10-07 05:12:44.625236 206'232818 2018-10-04 21:59:57.182136 0
1.391 659 0 0 0 0 2764046336 1566 1566 down 2018-10-10 08:59:55.450174 206'469466 4325:730006 [0,27] 0 [0,27] 0 206'455229 2018-10-06 05:47:49.464506 206'436677 2018-10-03 02:58:45.274120 0
1.38a 629 0 0 0 0 2632974336 1511 1511 down 2018-10-10 08:59:55.449919 206'149211 4325:283075 [0,16] 0 [0,16] 0 206'143225 2018-10-05 23:25:05.879394 206'139053 2018-10-04 21:19:10.986677 0
1.37f 628 0 0 0 0 2634022912 1541 1541 down 2018-10-10 08:59:55.452167 209'349843 4325:584459 [3,17] 3 [3,17] 3 206'347692 2018-10-07 03:18:36.796512 206'297229 2018-10-04 05:01:27.787549 0
1.37d 629 0 0 0 0 2630221824 1576 1576 down 2018-10-10 08:59:52.375959 209'271584 4325:446690 [2,26] 2 [2,26] 2 206'261150 2018-10-06 20:43:50.677865 206'237209 2018-10-03 20:44:35.514977 0
1.375 696 0 0 0 0 2918187008 1515 1515 down 2018-10-10 08:59:55.459731 206'154915 4325:343035 [4,20] 4 [4,20] 4 206'154317 2018-10-07 03:32:33.241067 206'144151 2018-10-03 21:20:59.979549 0
1.370 631 0 0 0 0 2645557248 1552 1552 down 2018-10-10 08:59:55.462854 209'277952 4325:453726 [4,19] 4 [4,19] 4 206'272732 2018-10-06 23:38:04.435594 206'261054 2018-10-05 20:15:36.695075 0
1.36c 702 0 0 0 0 2942304256 1538 1538 down 2018-10-10 08:59:55.457534 209'418039 4325:631689 [5,27] 5 [5,27] 5 206'404932 2018-10-06 20:19:36.199050 206'404932 2018-10-06 20:19:36.199050 0
1.36b 676 0 0 0 0 2834300928 1561 1561 down 2018-10-10 08:59:55.461724 206'270063 4325:505237 [4,25] 4 [4,25] 4 206'268777 2018-10-06 21:14:05.063694 206'201143 2018-10-03 01:00:44.826044 0
1.350 653 0 0 0 0 2734686208 1587 1587 down 2018-10-10 08:59:55.477065 206'201687 4325:389911 [5,24] 5 [5,24] 5 206'198724 2018-10-07 01:23:22.892742 206'187095 2018-10-05 21:20:41.222238 0
1.34e 636 0 0 0 0 2667577344 1586 1586 down 2018-10-10 08:59:55.470134 206'144686 4325:370214 [2,23] 2 [2,23] 2 206'141503 2018-10-07 03:51:36.974517 206'92979 2018-10-02 03:12:33.801450 0
1.349 635 0 0 0 0 2662334464 1528 1528 down 2018-10-10 08:59:53.413474 206'197428 4325:608392 [2,19] 2 [2,19] 2 206'194860 2018-10-06 20:34:22.749836 206'194860 2018-10-06 20:34:22.749836 0
1.33f 627 0 0 0 0 2627731456 1546 1546 down 2018-10-10 08:59:55.466546 206'162546 4325:320812 [5,19] 5 [5,19] 5 206'160567 2018-10-06 22:11:53.102227 206'160567 2018-10-06 22:11:53.102227 0
1.195 669 0 0 0 0 2803433472 1511 1511 down 2018-10-10 08:59:55.453415 209'328911 4325:473075 [1,24] 1 [1,24] 1 206'311941 2018-10-06 20:02:16.781906 206'265920 2018-10-02 00:51:09.381016 0
1.18e 633 0 0 0 0 2654994432 1585 1585 down 2018-10-10 08:59:55.450386 206'251586 4325:405613 [5,17] 5 [5,17] 5 206'246250 2018-10-06 04:11:19.917823 206'231255 2018-10-04 21:33:34.577615 0
1.18b 623 0 0 0 0 2613051392 1599 1599 down 2018-10-10 08:59:55.458789 206'139999 4325:376538 [1,17] 1 [1,17] 1 206'138097 2018-10-06 05:48:58.787533 206'134925 2018-10-05 02:54:48.920974 0
1.18a 676 0 0 0 0 2835349504 1538 1538 down 2018-10-10 08:59:55.455242 208'224338 4325:380604 [3,24] 3 [3,24] 3 206'211154 2018-10-06 04:15:53.579765 202'154074 2018-09-30 22:13:07.211234 0
1.178 648 0 0 0 0 2715811840 1529 1529 down 2018-10-10 08:59:55.472145 206'516332 4325:756616 [5,18] 5 [5,18] 5 206'494000 2018-10-05 20:59:21.453411 202'399161 2018-09-29 20:10:19.212349 0
1.174 641 0 0 0 0 2687500288 1576 1576 down 2018-10-10 08:59:55.471683 206'248976 4325:474330 [2,23] 2 [2,23] 2 206'248298 2018-10-06 20:54:17.633003 206'233314 2018-10-03 20:33:52.603212 0
1.173 655 0 0 0 0 2747269120 1509 1509 down 2018-10-10 08:59:52.361309 206'226909 4325:460277 [3,27] 3 [3,27] 3 206'226175 2018-10-06 20:16:56.566793 206'219757 2018-10-04 21:14:56.869046 0
1.172 680 0 0 0 0 2848980992 1504 1504 down 2018-10-10 08:59:55.458930 209'262004 4325:443473 [4,26] 4 [4,26] 4 206'253071 2018-10-06 21:53:24.480440 206'253071 2018-10-06 21:53:24.480440 0
1.16f 644 0 0 0 0 2700083200 1579 1579 down 2018-10-10 08:59:55.459605 206'369079 4325:574328 [4,22] 4 [4,22] 4 206'368077 2018-10-06 21:02:40.489692 206'324686 2018-10-02 20:24:51.799122 0
1.16e 660 0 0 0 0 2768240640 1563 1563 down 2018-10-10 08:59:55.449531 209'297163 4325:573483 [0,27] 0 [0,27] 0 206'295150 2018-10-07 03:47:14.046615 206'295150 2018-10-07 03:47:14.046615 0
1.16b 648 0 0 0 0 2717908992 1541 1541 down 2018-10-10 08:59:55.459329 206'171741 4325:311932 [4,25] 4 [4,25] 4 206'168097 2018-10-06 05:46:24.052709 206'168097 2018-10-06 05:46:24.052709 0
1.168 662 0 0 0 0 2776629248 1517 1517 down 2018-10-10 08:59:55.453602 206'171420 4325:323302 [3,19] 3 [3,19] 3 206'170896 2018-10-07 00:01:15.546793 206'165634 2018-10-05 20:12:59.717628 0
1.160 693 0 0 0 0 2905604096 1519 1519 down 2018-10-10 08:59:55.469006 206'184719 4325:340474 [2,17] 2 [2,17] 2 206'181561 2018-10-06 05:39:16.802899 206'165283 2018-10-01 21:41:20.139977 0
1.15d 640 0 0 0 0 2681602048 1597 1597 down 2018-10-10 08:59:52.367387 209'324998 4325:520920 [4,26] 4 [4,26] 4 206'308183 2018-10-06 04:57:06.943762 206'272175 2018-10-02 03:54:26.148300 0
1.15a 630 0 0 0 0 2637168640 1560 1560 down 2018-10-10 08:59:55.455094 209'203860 4325:399496 [1,19] 1 [1,19] 1 206'203441 2018-10-07 02:21:51.535767 206'178244 2018-10-04 01:14:51.331867 0
1.157 620 0 0 0 0 2600468480 1532 1532 down 2018-10-10 08:59:55.448901 206'387032 4325:607681 [0,27] 0 [0,27] 0 206'385919 2018-10-07 03:11:48.222958 206'357341 2018-10-03 05:19:56.944378 0
1.154 660 0 0 0 0 2766143488 1515 1515 down 2018-10-10 08:59:55.469211 209'271917 4325:516654 [2,25] 2 [2,25] 2 206'270507 2018-10-07 05:10:26.097165 206'262269 2018-10-06 00:28:54.314626 0
1.153 698 0 0 0 0 2927624192 1586 1586 down 2018-10-10 08:59:55.456603 209'307088 4325:474130 [1,18] 1 [1,18] 1 206'304123 2018-10-06 20:33:45.474345 202'264247 2018-09-30 22:19:27.201291 0
1.14a 677 0 0 0 0 2837643264 1597 1597 down 2018-10-10 08:59:55.453051 209'238597 4325:465982 [1,19] 1 [1,19] 1 206'235511 2018-10-06 20:12:30.202237 206'235511 2018-10-06 20:12:30.202237 0
1.13c 650 0 0 0 0 2726297600 1599 1599 down 2018-10-10 08:59:55.450407 206'107599 4325:228150 [0,27] 0 [0,27] 0 206'107471 2018-10-07 04:51:29.898401 206'98749 2018-10-02 20:58:25.855764 0
1.12f 653 0 0 0 0 2735734784 1538 1538 down 2018-10-10 08:59:55.458540 209'193338 4325:350375 [1,19] 1 [1,19] 1 206'192799 2018-10-07 02:37:53.068491 206'177447 2018-10-04 05:08:11.536557 0
1.12a 696 0 0 0 0 2915368960 1592 1592 down 2018-10-10 08:59:55.461862 206'320393 4325:573349 [5,23] 5 [5,23] 5 206'318894 2018-10-06 20:32:04.670605 206'313784 2018-10-05 03:14:07.865451 0
1.125 676 0 0 0 0 2831155200 1580 1580 down 2018-10-10 08:59:55.454502 209'530082 4325:843221 [3,22] 3 [3,22] 3 206'527916 2018-10-06 20:14:33.399545 202'427518 2018-09-30 03:39:50.486210 0
1.11e 621 0 0 0 0 2604662784 1593 1593 down 2018-10-10 08:59:55.446640 209'294693 4325:504510 [0,20] 0 [0,20] 0 206'258596 2018-10-06 04:50:34.111053 206'225581 2018-10-03 20:02:15.958587 0
1.11b 691 0 0 0 0 2894069760 1539 1539 down 2018-10-10 08:59:53.400776 209'298740 4325:570197 [1,16] 1 [1,16] 1 206'293887 2018-10-07 00:05:23.896465 206'287931 2018-10-05 21:37:43.800422 0
1.119 680 0 0 0 0 2843738112 1550 1550 down 2018-10-10 08:59:55.453946 209'373451 4325:596446 [0,18] 0 [0,18] 0 206'369037 2018-10-06 21:16:14.384044 206'339980 2018-10-02 21:05:24.004867 0
1.117 661 0 0 0 0 2772434944 1515 1515 down 2018-10-10 08:59:55.459923 209'995718 4325:1315011 [4,24] 4 [4,24] 4 206'967976 2018-10-05 21:42:54.012937 202'872734 2018-09-30 20:33:27.677955 0
1.111 679 0 0 0 0 2845835264 1557 1557 down 2018-10-10 08:59:55.451296 209'273857 4325:518425 [3,26] 3 [3,26] 3 206'258452 2018-10-06 04:09:30.313146 206'258452 2018-10-06 04:09:30.313146 0
1.109 661 0 0 0 0 2770337792 1597 1597 down 2018-10-10 08:59:55.461695 209'248797 4325:468113 [4,16] 4 [4,16] 4 206'241195 2018-10-06 20:02:14.116240 206'215941 2018-10-03 20:12:21.950039 0
1.105 611 0 0 0 0 2560622592 1540 1540 down 2018-10-10 08:59:55.451838 209'313141 4325:486120 [1,24] 1 [1,24] 1 206'311242 2018-10-06 20:42:43.907469 206'311242 2018-10-06 20:42:43.907469 0
1.101 611 0 0 0 0 2558984192 1540 1540 down 2018-10-10 08:59:52.363205 206'227341 4325:390388 [1,15] 1 [1,15] 1 206'226510 2018-10-07 03:57:18.094441 206'213133 2018-10-04 05:00:00.719768 0
1.ff 649 0 0 0 0 2720989184 1515 1515 down 2018-10-10 08:59:55.465117 206'256715 4325:451421 [5,21] 5 [5,21] 5 206'256029 2018-10-07 01:47:37.412852 206'250642 2018-10-04 21:02:25.510120 0
1.fd 690 0 0 0 0 2893807616 1509 1509 down 2018-10-10 08:59:55.453816 209'589210 4325:862591 [3,23] 3 [3,23] 3 206'577955 2018-10-06 20:39:11.723444 202'454544 2018-09-30 20:07:58.306496 0
1.fb 651 0 0 0 0 2730491904 1562 1562 down 2018-10-10 08:59:55.461337 206'302062 4325:501576 [4,16] 4 [4,16] 4 206'301042 2018-10-06 20:56:03.502744 202'263289 2018-09-30 03:08:02.237865 0
1.f6 638 0 0 0 0 2672820224 1600 1600 down 2018-10-10 08:59:55.475076 209'104200 4325:551700 [5,22] 5 [5,22] 5 206'103762 2018-10-06 20:14:39.976772 206'100607 2018-10-05 00:53:19.089093 0
1.f4 668 0 0 0 0 2800746496 1557 1557 down 2018-10-10 08:59:55.467754 208'259257 4325:410740 [5,18] 5 [5,18] 5 206'257052 2018-10-06 20:25:19.332803 202'215744 2018-09-30 03:46:45.694228 0
1.f2 658 0 0 0 0 2759852032 1550 1550 down 2018-10-10 08:59:55.446924 209'395450 4325:607503 [0,26] 0 [0,26] 0 206'390515 2018-10-07 02:12:07.478944 206'369298 2018-10-05 20:49:18.461986 0
1.f1 637 0 0 0 0 2671771648 1594 1594 down 2018-10-10 08:59:55.447913 209'220895 4325:363355 [0,19] 0 [0,19] 0 206'217232 2018-10-07 04:58:24.857838 206'169743 2018-10-02 22:51:58.465506 0
1.ef 630 0 0 0 0 2642411520 1578 1578 down 2018-10-10 08:59:51.390339 209'165378 4325:286638 [2,22] 2 [2,22] 2 206'155422 2018-10-06 04:57:14.933611 206'155422 2018-10-06 04:57:14.933611 0
1.ee 695 0 0 0 0 2915041280 1532 1532 down 2018-10-10 08:59:55.442718 209'143832 4325:350390 [6,17] 6 [6,17] 6 206'140252 2018-10-06 20:40:32.527888 206'123279 2018-10-03 21:12:45.123198 0
1.dc 644 0 0 0 0 2697986048 1596 1596 down 2018-10-10 08:59:55.440997 206'115497 4325:266486 [6,23] 6 [6,23] 6 206'112175 2018-10-06 02:28:40.964645 206'112175 2018-10-06 02:28:40.964645 0
1.57 620 0 0 0 0 2600468480 1532 1532 down 2018-10-10 08:59:55.472909 206'96032 4325:491632 [5,14] 5 [5,14] 5 206'93370 2018-10-06 04:59:13.805161 206'84919 2018-10-01 20:24:09.216852 0
1.51 662 0 0 0 0 2773483520 1592 1592 down 2018-10-10 08:59:55.440641 206'203392 4325:358452 [6,24] 6 [6,24] 6 206'202805 2018-10-06 21:37:17.133908 206'202805 2018-10-06 21:37:17.133908 0
1.4c 671 0 0 0 0 2814377984 1577 1577 down 2018-10-10 08:59:52.398712 206'204277 4325:359296 [6,26] 6 [6,26] 6 206'199985 2018-10-05 20:49:23.426315 206'171533 2018-10-02 04:58:04.766140 0
1.49 668 0 0 0 0 2796552192 1537 1537 down 2018-10-10 08:59:55.457592 209'207437 4325:355228 [3,22] 3 [3,22] 3 206'207033 2018-10-07 00:52:53.902788 206'200433 2018-10-05 20:35:25.975393 0
1.48 679 0 0 0 0 2847932416 1502 1502 down 2018-10-10 08:59:55.477753 206'260602 4325:510254 [5,15] 5 [5,15] 5 206'258761 2018-10-06 20:52:34.025763 206'217488 2018-10-03 05:25:42.955528 0
1.43 661 0 0 0 0 2772434944 1567 1567 down 2018-10-10 08:59:55.470026 206'396767 4325:666764 [2,19] 2 [2,19] 2 206'382803 2018-10-06 03:33:11.224490 206'349969 2018-10-03 04:51:31.806790 0
1.3c 695 0 0 0 0 2915041280 1503 1503 down 2018-10-10 08:59:55.450904 209'281403 4325:493465 [5,19] 5 [5,19] 5 206'279084 2018-10-07 04:27:04.359298 206'269721 2018-10-05 20:35:11.458189 0
1.39 667 0 0 0 0 2796552192 1514 1514 down 2018-10-10 08:59:55.449563 209'638516 4325:963637 [0,17] 0 [0,17] 0 206'634984 2018-10-07 05:56:21.872587 206'634984 2018-10-07 05:56:21.872587 0
1.f 623 0 0 0 0 2611675136 1577 1577 down 2018-10-10 08:59:52.360651 209'197678 4325:352094 [3,17] 3 [3,17] 3 206'192120 2018-10-06 04:18:25.471509 206'192120 2018-10-06 04:18:25.471509 0
1.e 691 0 0 0 0 2898264064 1545 1545 down 2018-10-10 08:59:55.461009 209'209446 4325:406745 [4,20] 4 [4,20] 4 206'206471 2018-10-06 22:26:37.999849 206'201593 2018-10-05 20:12:42.708296 0
1.3 708 0 0 0 0 2969567232 1526 1526 down 2018-10-10 08:59:55.462317 209'194627 4325:319346 [4,27] 4 [4,27] 4 206'189477 2018-10-07 05:50:13.170565 206'155275 2018-10-05 21:01:24.323734 0
1.8 639 0 0 0 0 2674327552 1591 1591 down 2018-10-10 08:59:55.462086 206'234991 4325:403954 [4,16] 4 [4,16] 4 206'232532 2018-10-06 23:28:16.458068 206'232532 2018-10-06 23:28:16.458068 0
1.28 657 0 0 0 0 2754609152 1564 1564 down 2018-10-10 08:59:55.467905 206'243264 4325:497956 [2,25] 2 [2,25] 2 206'242008 2018-10-06 22:06:08.640692 206'238745 2018-10-05 20:42:46.383445 0
1.2c 655 0 0 0 0 2747269120 1558 1558 down 2018-10-10 08:59:55.452251 206'197360 4325:419965 [0,21] 0 [0,21] 0 206'196233 2018-10-06 20:38:27.757487 202'185153 2018-09-30 20:07:48.918320 0
1.67 659 0 0 0 0 2760441856 1548 1548 down 2018-10-10 08:59:55.445090 209'131148 4325:297278 [0,17] 0 [0,17] 0 206'130202 2018-10-06 20:51:31.863465 206'120007 2018-10-02 05:09:43.222619 0
1.6b 664 0 0 0 0 2785017856 1533 1533 down 2018-10-10 08:59:54.467063 209'270933 4325:482730 [0,26] 0 [0,26] 0 206'268403 2018-10-06 20:45:44.573968 202'186767 2018-09-29 21:40:34.566940 0
1.73 683 0 0 0 0 2862612480 1558 1558 down 2018-10-10 08:59:52.364888 209'102758 4325:226019 [4,16] 4 [4,16] 4 206'99318 2018-10-06 20:38:03.404024 206'89431 2018-10-04 22:31:41.976219 0
1.76 650 0 0 0 0 2723807232 1514 1514 down 2018-10-10 08:59:55.471926 206'207014 4325:389747 [5,22] 5 [5,22] 5 206'166241 2018-10-06 00:28:58.758009 206'166241 2018-10-06 00:28:58.758009 0
1.7a 666 0 0 0 0 2791309312 1502 1502 down 2018-10-10 08:59:53.415244 209'581203 4325:843452 [2,19] 2 [2,19] 2 206'554212 2018-10-05 20:39:59.163180 206'523051 2018-10-03 23:11:29.793363 0
1.8b 634 0 0 0 0 2659188736 1549 1549 down 2018-10-10 08:59:55.460560 209'450768 4325:752318 [4,22] 4 [4,22] 4 206'435747 2018-10-06 03:27:40.315939 202'327980 2018-09-30 21:38:13.843323 0
1.8f 688 0 0 0 0 2882666496 1537 1537 down 2018-10-10 08:59:55.458215 206'206237 4325:661414 [1,15] 1 [1,15] 1 206'201400 2018-10-05 21:34:22.628823 206'201400 2018-10-05 21:34:22.628823 0
1.9c 641 0 0 0 0 2682191872 1528 1528 down 2018-10-10 08:59:55.464428 209'418128 4325:618482 [5,14] 5 [5,14] 5 206'409743 2018-10-06 22:33:26.805066 206'378472 2018-10-04 20:45:28.125700 0
1.a0 642 0 0 0 0 2689597440 1538 1538 down 2018-10-10 08:59:55.475741 206'197138 4325:338578 [5,17] 5 [5,17] 5 206'196747 2018-10-07 01:43:13.706524 206'191891 2018-10-05 21:03:26.054327 0
1.a2 638 0 0 0 0 2674917376 1536 1536 down 2018-10-10 08:59:55.451413 209'142437 4325:292010 [1,23] 1 [1,23] 1 206'133353 2018-10-06 03:02:36.654581 206'118649 2018-10-04 20:53:40.864457 0
1.a9 656 0 0 0 0 2749366272 1531 1531 down 2018-10-10 08:59:55.439591 206'83031 4325:187608 [6,23] 6 [6,23] 6 206'82757 2018-10-07 00:17:05.422538 206'82757 2018-10-07 00:17:05.422538 0
1.ad 639 0 0 0 0 2680160256 1516 1516 down 2018-10-10 08:59:55.450888 209'1066417 4325:1295452 [0,21] 0 [0,21] 0 206'1050217 2018-10-06 21:03:14.855406 206'903835 2018-10-02 23:00:43.301665 0
1.af 629 0 0 0 0 2632318976 1568 1568 down 2018-10-10 08:59:55.439687 206'162368 4325:309010 [6,20] 6 [6,20] 6 206'157985 2018-10-07 05:40:21.115775 206'126341 2018-10-01 20:17:18.944955 0
1.b7 676 0 0 0 0 2832203776 1570 1570 down 2018-10-10 08:59:55.468889 209'336771 4325:487745 [2,21] 2 [2,21] 2 206'335137 2018-10-07 05:33:14.964964 206'305511 2018-10-02 21:24:44.893434 0
1.bc 654 0 0 0 0 2739929088 1503 1503 down 2018-10-10 08:59:55.469941 206'390703 4325:688465 [2,25] 2 [2,25] 2 206'377973 2018-10-06 00:33:24.369814 202'354155 2018-09-30 22:37:43.576966 0
1.c1 668 0 0 0 0 2801795072 1583 1583 down 2018-10-10 08:59:55.442301 209'303183 4325:481916 [6,14] 6 [6,14] 6 206'293871 2018-10-05 21:07:18.808506 202'232714 2018-09-29 01:46:16.854238 0
1.19a 653 0 0 0 0 2738880512 1522 1522 down 2018-10-10 08:59:55.459092 209'177422 4325:363284 [1,26] 1 [1,26] 1 206'176875 2018-10-07 05:40:27.832643 206'149251 2018-10-03 20:49:52.218867 0
1.19d 675 0 0 0 0 2831155200 1509 1509 down 2018-10-10 08:59:55.451218 209'150409 4325:426601 [3,15] 3 [3,15] 3 206'145975 2018-10-05 20:39:49.211832 206'136386 2018-10-04 05:46:29.784318 0
1.1aa 662 0 0 0 0 2776629248 1519 1519 down 2018-10-10 08:59:55.469509 206'161919 4325:340228 [2,14] 2 [2,14] 2 206'156916 2018-10-06 05:31:33.386693 202'114749 2018-09-30 20:10:26.947374 0
1.1ad 696 0 0 0 0 2916089856 1525 1525 down 2018-10-10 08:59:55.456135 206'384826 4325:769371 [1,17] 1 [1,17] 1 206'384125 2018-10-06 20:31:24.378919 206'365240 2018-10-03 21:11:31.116693 0
1.1bf 700 0 0 0 0 2930769920 1514 1514 down 2018-10-10 08:59:52.362722 206'549515 4325:1087461 [3,15] 3 [3,15] 3 206'545303 2018-10-06 01:58:11.671104 202'504065 2018-09-29 21:06:03.248299 0
1.1c4 624 0 0 0 0 2616197120 1543 1543 down 2018-10-10 08:59:55.453087 209'530243 4325:841767 [3,21] 3 [3,21] 3 206'528912 2018-10-07 04:18:26.240877 206'518156 2018-10-06 02:34:07.364140 0
1.1c8 675 0 0 0 0 2831155200 1545 1545 down 2018-10-10 08:59:55.470974 209'161745 4325:326414 [2,19] 2 [2,19] 2 206'154814 2018-10-06 00:31:11.407073 206'149752 2018-10-03 05:49:23.681425 0
1.1cd 661 0 0 0 0 2769289216 1600 1600 down 2018-10-10 08:59:55.459422 209'234101 4325:447498 [1,19] 1 [1,19] 1 206'230896 2018-10-06 22:04:52.795726 206'230896 2018-10-06 22:04:52.795726 0
1.1d7 633 0 0 0 0 2652897280 1527 1527 down 2018-10-10 08:59:52.398291 209'308127 4325:508833 [6,24] 6 [6,24] 6 206'267712 2018-10-05 20:18:26.420550 202'179785 2018-09-28 22:20:13.071621 0
1.1e0 638 0 0 0 0 2675965952 1508 1508 down 2018-10-10 08:59:55.460148 206'206509 4325:351506 [4,21] 4 [4,21] 4 206'201897 2018-10-05 20:47:08.586761 206'191385 2018-10-04 05:55:17.495895 0
1.1e5 696 0 0 0 0 2917138432 1534 1534 down 2018-10-10 08:59:55.441845 206'337335 4325:624677 [6,26] 6 [6,26] 6 206'334588 2018-10-06 22:07:38.106965 202'254901 2018-10-01 02:43:25.987027 0
1.1e9 654 0 0 0 0 2738880512 1573 1573 down 2018-10-10 08:59:52.361802 206'230873 4325:407365 [3,25] 3 [3,25] 3 206'228496 2018-10-06 05:25:08.219635 206'228496 2018-10-06 05:25:08.219635 0
1.1ee 696 0 0 0 0 2919235584 1538 1538 down 2018-10-10 08:59:52.394058 206'545939 4325:847332 [6,22] 6 [6,22] 6 206'544991 2018-10-07 02:59:25.074473 206'505095 2018-10-03 05:25:18.836058 0
1.1f0 617 0 0 0 0 2587885568 1548 1548 down 2018-10-10 08:59:55.445572 206'542148 4325:745790 [0,15] 0 [0,15] 0 206'533696 2018-10-06 20:21:41.358580 206'519863 2018-10-04 21:42:32.787771 0
1.1f7 671 0 0 0 0 2814377984 1514 1514 down 2018-10-10 08:59:55.452645 206'377214 4325:585524 [0,14] 0 [0,14] 0 206'375543 2018-10-07 04:09:09.641371 206'315313 2018-10-03 05:04:10.315940 0
1.200 655 0 0 0 0 2745171968 1533 1533 down 2018-10-10 08:59:55.441624 209'393034 4325:611564 [6,19] 6 [6,19] 6 206'363934 2018-10-06 01:48:17.695788 202'239456 2018-09-29 21:56:07.497892 0
1.207 660 0 0 0 0 2768240640 1570 1570 down 2018-10-10 08:59:52.400833 206'287970 4325:540511 [6,25] 6 [6,25] 6 206'286176 2018-10-07 04:21:27.362987 202'236542 2018-10-01 01:50:13.806136 0
1.21c 669 0 0 0 0 2805989376 1571 1571 down 2018-10-10 08:59:55.440310 209'262872 4325:436348 [6,15] 6 [6,15] 6 206'255305 2018-10-06 20:15:17.237888 202'202516 2018-09-30 20:41:56.690778 0
1.21d 651 0 0 0 0 2725249024 1570 1570 down 2018-10-10 08:59:55.450861 209'287670 4325:495476 [3,25] 3 [3,25] 3 206'280400 2018-10-06 23:00:40.352203 206'252624 2018-10-03 03:44:30.647372 0
1.21e 693 0 0 0 0 2904555520 1554 1554 down 2018-10-10 08:59:55.454836 206'82454 4325:242555 [5,21] 5 [5,21] 5 206'80662 2018-10-06 01:16:24.397026 206'78608 2018-10-04 20:23:26.375228 0
1.226 650 0 0 0 0 2724200448 1600 1600 down 2018-10-10 08:59:55.452297 206'236300 4325:515401 [1,25] 1 [1,25] 1 206'236077 2018-10-07 05:26:18.397494 206'236077 2018-10-07 05:26:18.397494 0
1.22a 709 0 0 0 0 2973761536 1547 1547 down 2018-10-10 08:59:55.453387 209'252449 4325:722370 [3,18] 3 [3,18] 3 206'241651 2018-10-07 01:12:14.760199 206'241651 2018-10-07 01:12:14.760199 0
1.22b 674 0 0 0 0 2821586944 1520 1520 down 2018-10-10 08:59:55.471263 206'555421 4325:838700 [2,26] 2 [2,26] 2 206'535390 2018-10-05 20:24:23.752451 206'513705 2018-10-04 01:09:00.306751 0
1.236 613 0 0 0 0 2569011200 1565 1565 down 2018-10-10 08:59:54.435160 209'121265 4325:361782 [2,26] 2 [2,26] 2 206'120652 2018-10-06 20:38:57.462568 206'112598 2018-10-03 20:19:17.345890 0
1.23a 726 0 0 0 0 3042967552 1597 1597 down 2018-10-10 08:59:55.451724 209'200697 4325:369953 [3,23] 3 [3,23] 3 206'199283 2018-10-06 23:45:52.545410 202'174263 2018-09-30 20:47:27.340478 0
1.23d 636 0 0 0 0 2666987520 1518 1518 down 2018-10-10 08:59:55.452700 207'259718 4325:447869 [1,16] 1 [1,16] 1 206'255475 2018-10-06 05:39:05.744147 206'249884 2018-10-05 04:11:06.283276 0
1.246 604 0 0 0 0 2533359616 1546 1546 down 2018-10-10 08:59:54.432871 209'415848 4325:612415 [2,16] 2 [2,16] 2 206'413504 2018-10-07 03:45:39.498784 206'359969 2018-10-03 04:31:07.789469 0
1.248 643 0 0 0 0 2694840320 1566 1566 down 2018-10-10 08:59:54.459010 209'198567 4325:350809 [0,19] 0 [0,19] 0 206'192484 2018-10-06 21:35:59.947578 206'115459 2018-10-02 04:58:01.523300 0
1.24c 648 0 0 0 0 2715811840 1567 1567 down 2018-10-10 08:59:55.461127 209'218367 4325:361959 [4,14] 4 [4,14] 4 206'207154 2018-10-06 04:22:02.745923 206'174491 2018-10-02 20:28:11.627051 0
1.254 708 0 0 0 0 2968518656 1591 1591 down 2018-10-10 08:59:55.469704 206'85991 4325:216653 [2,16] 2 [2,16] 2 206'85771 2018-10-07 01:07:43.148670 206'85771 2018-10-07 01:07:43.148670 0
1.25c 629 0 0 0 0 2638217216 1531 1531 down 2018-10-10 08:59:55.469346 209'262331 4325:463471 [2,19] 2 [2,19] 2 206'248252 2018-10-06 01:49:09.976032 206'204500 2018-10-03 21:21:04.758627 0
1.266 658 0 0 0 0 2758475776 1595 1595 down 2018-10-10 08:59:55.468699 209'235495 4325:447637 [2,25] 2 [2,25] 2 206'232087 2018-10-06 20:36:50.899585 206'232087 2018-10-06 20:36:50.899585 0
1.26b 637 0 0 0 0 2666528768 1556 1556 down 2018-10-10 08:59:55.454072 209'522857 4325:742831 [5,27] 5 [5,27] 5 206'519365 2018-10-07 00:35:18.556234 206'519365 2018-10-07 00:35:18.556234 0
1.26f 659 0 0 0 0 2764046336 1502 1502 down 2018-10-10 08:59:55.447649 206'263902 4325:672172 [0,18] 0 [0,18] 0 206'261609 2018-10-06 05:27:33.316590 206'258802 2018-10-04 21:05:22.627938 0
1.270 699 0 0 0 0 2930769920 1517 1517 down 2018-10-10 08:59:55.461487 206'400318 4325:621077 [4,23] 4 [4,23] 4 206'397476 2018-10-06 23:30:51.292999 206'397476 2018-10-06 23:30:51.292999 0
1.27c 659 0 0 0 0 2761949184 1539 1539 down 2018-10-10 08:59:55.455544 209'524639 4325:788810 [1,19] 1 [1,19] 1 206'516190 2018-10-06 20:14:54.582319 206'516190 2018-10-06 20:14:54.582319 0
1.28c 670 0 0 0 0 2803892224 1529 1529 down 2018-10-10 08:59:55.452637 209'475529 4325:728618 [5,19] 5 [5,19] 5 206'473238 2018-10-07 02:56:32.746003 206'451679 2018-10-05 21:24:52.109871 0
1.28d 670 0 0 0 0 2806185984 1507 1507 down 2018-10-10 08:59:55.468089 209'346307 4325:909909 [2,16] 2 [2,16] 2 206'342007 2018-10-06 20:02:28.451567 206'342007 2018-10-06 20:02:28.451567 0
1.28e 681 0 0 0 0 2856321024 1551 1551 down 2018-10-10 08:59:55.460438 206'235351 4325:410019 [4,15] 4 [4,15] 4 206'233554 2018-10-06 20:45:34.440332 206'206593 2018-10-02 05:40:28.676720 0
1.291 688 0 0 0 0 2885681152 1584 1584 down 2018-10-10 08:59:55.448837 209'350284 4325:529242 [0,15] 0 [0,15] 0 206'348246 2018-10-07 04:16:18.742082 202'260926 2018-09-30 22:02:56.009676 0
1.29b 679 0 0 0 0 2847932416 1528 1528 down 2018-10-10 08:59:55.459254 206'128028 4325:326064 [4,15] 4 [4,15] 4 206'126951 2018-10-06 23:58:54.924587 206'119806 2018-10-04 01:04:36.036555 0
1.2a3 641 0 0 0 0 2682585088 1526 1526 down 2018-10-10 08:59:55.461978 209'300326 4325:564239 [4,15] 4 [4,15] 4 206'299459 2018-10-07 03:43:34.930761 206'299459 2018-10-07 03:43:34.930761 0
1.2ae 670 0 0 0 0 2807037952 1505 1505 down 2018-10-10 08:59:55.471914 209'456206 4325:624948 [2,17] 2 [2,17] 2 206'449801 2018-10-06 20:31:33.684978 202'362298 2018-10-01 05:35:45.086451 0
1.2b0 653 0 0 0 0 2737831936 1586 1586 down 2018-10-10 08:59:55.473198 209'305387 4325:514553 [5,26] 5 [5,26] 5 206'300699 2018-10-06 20:54:49.006148 206'283928 2018-10-05 05:32:21.024252 0
1.2b2 684 0 0 0 0 2866806784 1530 1530 down 2018-10-10 08:59:55.452823 209'227931 4325:472385 [0,25] 0 [0,25] 0 206'224195 2018-10-06 20:33:52.506908 206'181256 2018-10-02 03:43:27.153472 0
1.2b9 664 0 0 0 0 2781872128 1599 1599 down 2018-10-10 08:59:55.447459 209'162599 4325:305538 [0,22] 0 [0,22] 0 206'152862 2018-10-06 20:12:36.126883 206'146479 2018-10-04 20:52:58.599098 0
1.2c3 707 0 0 0 0 2965372928 1549 1549 down 2018-10-10 08:59:53.415045 209'304449 4325:492284 [2,14] 2 [2,14] 2 206'291948 2018-10-06 20:21:28.411965 206'270665 2018-10-04 21:56:44.658398 0
1.2c8 681 0 0 0 0 2855272448 1557 1557 down 2018-10-10 08:59:54.464418 206'225958 4325:416077 [0,18] 0 [0,18] 0 206'225472 2018-10-07 05:42:50.196552 206'225472 2018-10-07 05:42:50.196552 0
1.2cc 656 0 0 0 0 2747269120 1563 1563 down 2018-10-10 08:59:55.451708 209'133164 4325:318354 [5,20] 5 [5,20] 5 206'121994 2018-10-06 02:53:21.873146 206'120419 2018-10-04 21:41:28.945653 0
1.2da 628 0 0 0 0 2632581120 1600 1600 down 2018-10-10 08:59:55.441022 209'211100 4325:382332 [6,25] 6 [6,25] 6 206'208687 2018-10-07 00:56:08.172802 206'169766 2018-10-01 20:27:58.789449 0
1.2dd 675 0 0 0 0 2831155200 1595 1595 down 2018-10-10 08:59:55.458594 206'139595 4325:295923 [4,17] 4 [4,17] 4 206'139164 2018-10-06 23:07:56.387092 202'127056 2018-10-01 05:27:37.569886 0
1.2ed 670 0 0 0 0 2807758848 1555 1555 down 2018-10-10 08:59:55.451190 206'292756 4325:813354 [5,21] 5 [5,21] 5 206'292261 2018-10-07 01:55:19.328036 206'282217 2018-10-04 20:13:13.369035 0
1.2ee 645 0 0 0 0 2701393920 1588 1588 down 2018-10-10 08:59:55.440108 209'747390 4325:1093722 [6,26] 6 [6,26] 6 206'746164 2018-10-06 20:22:07.392596 202'704854 2018-09-30 20:30:25.717104 0
1.2f6 694 0 0 0 0 2910846976 1588 1588 down 2018-10-10 08:59:53.413242 209'369589 4325:538680 [6,15] 6 [6,15] 6 206'352019 2018-10-05 23:46:16.954979 206'339982 2018-10-04 21:08:52.877578 0
1.2fd 636 0 0 0 0 2664431616 1556 1556 down 2018-10-10 08:59:55.457207 206'564056 4325:777780 [1,15] 1 [1,15] 1 206'562181 2018-10-06 20:58:41.041453 202'513295 2018-09-30 03:32:33.265178 0
1.306 657 0 0 0 0 2754609152 1518 1518 down 2018-10-10 08:59:55.455924 209'274520 4325:508468 [3,16] 3 [3,16] 3 206'269349 2018-10-07 04:09:33.579381 206'269349 2018-10-07 04:09:33.579381 0
1.30b 688 0 0 0 0 2883584000 1535 1535 down 2018-10-10 08:59:55.450104 206'365936 4325:594069 [5,14] 5 [5,14] 5 206'365036 2018-10-07 05:54:16.522870 206'359002 2018-10-05 21:37:47.320211 0
1.319 681 0 0 0 0 2851654656 1598 1598 down 2018-10-10 08:59:55.469263 209'515000 4325:782053 [5,23] 5 [5,23] 5 206'510202 2018-10-07 04:01:42.833564 206'510202 2018-10-07 04:01:42.833564 0
1.31f 685 0 0 0 0 2872049664 1504 1504 down 2018-10-10 08:59:55.471067 209'159004 4325:3341710 [5,27] 5 [5,27] 5 206'157621 2018-10-06 20:34:41.151808 206'157621 2018-10-06 20:34:41.151808 0
1.321 641 0 0 0 0 2688548864 1592 1592 down 2018-10-10 08:59:55.451890 209'239792 4325:438678 [0,20] 0 [0,20] 0 206'231829 2018-10-06 20:43:09.526879 206'217935 2018-10-04 21:02:58.327669 0
1.325 679 0 0 0 0 2846883840 1597 1597 down 2018-10-10 08:59:55.449282 206'168398 4325:351746 [0,22] 0 [0,22] 0 206'166640 2018-10-06 04:18:34.843884 206'161184 2018-10-02 23:45:07.608811 0
1.32b 685 0 0 0 0 2868707328 1566 1566 down 2018-10-10 08:59:55.441745 206'360066 4325:602588 [6,26] 6 [6,26] 6 206'358662 2018-10-06 21:06:43.656690 206'351430 2018-10-05 02:59:22.900438 0
1.32d 653 0 0 0 0 2737831936 1527 1527 down 2018-10-10 08:59:55.454132 209'721028 4325:1040904 [3,24] 3 [3,24] 3 206'713658 2018-10-06 20:21:41.073105 206'668556 2018-10-05 05:25:36.689562 0
1.32f 641 0 0 0 0 2685403136 1600 1600 down 2018-10-10 08:59:55.467446 206'243300 4325:367931 [2,22] 2 [2,22] 2 206'238715 2018-10-05 20:12:39.550637 206'232783 2018-10-04 04:17:53.645740 0
1.335 659 0 0 0 0 2762670080 1534 1534 down 2018-10-10 08:59:55.458385 209'153137 4325:489641 [3,26] 3 [3,26] 3 206'138751 2018-10-06 00:07:59.897668 206'130215 2018-10-03 00:35:09.654503 0

I check for the screen console. Node 4 and 7 are normal. No error. But on node 6, there are many error pop up:

Oct 10 16:48:22 node7 kernel: [28138.597565] Unable to locate Target Portal Group on iqn.2016-05.com.petasan:00002
Oct 10 16:48:22 node7 kernel: [28138.597572] iSCSI Login negotiation failed.

https://ibb.co/jvqym9

I have reinstalled the Node 5 OS, can I add back to the ceph with the old osd SSD to recover this problem?

https://ibb.co/hkM2eU

Thanks

Ben

 

Last edited on October 10, 2018, 8:58 am by Yipkaiwing · #5

admin
2,930 Posts
October 10, 2018, 9:34 am

I would not worry about iSCSI errors on node 6. The main issue it the 166 down pgs, we need to get those active again.

Initially, the recovery is very unstable. The node are up and down frequently. Then I change the RAM setting "bluestore_cache_size_ssd" to 1 GB Ram. The storage rebuilding is become normal and smooth.

The nodes going up and down means there is resource shortages on the nodes how much RAM % free do you have on the nodes now ?
Can you use 1073741824 instead of 1G (the later should work but i did see warning from the output you listed )
bluestore_cache_size_ssd = 1073741824
What do you mean by The storage rebuilding is become normal and smooth.? the rebuilding should gradually recover the degraded pgs to active/clean, which does not seem to have happened.

I have reinstalled the Node 5 OS, can I add back to the ceph with the old osd SSD to recover this problem?

So the Node 5 is not up ? Where are the old OSDs now ? If they are not up, place the OSDs back in the node and turn it on. This will help a lot to recover the downn pgs.

Hopefully adding the osds back will help as per above. But even if this solves things, there is something wrong with the cluster, it should recover even without this extra help.

can you also get the output of:

ceph osd tree --cluster ceph
ceph pg 1.3fe query --cluster ceph

 

#6

Yipkaiwing
18 Posts
October 10, 2018, 10:04 am

Dear Admin

The nodes going up and down means there is resource shortages on the nodes how much RAM % free do you have on the nodes now ?

Now all node are using less than 25% RAM. Each server has about 32GB RAM. So, I am able to change it to 2GB RAM, if 1GB is not enough. Before that, by default, under 3GB ram setting, the ram usage is around 90%.

Can you use 1073741824 instead of 1G (the later should work but i did see warning from the output you listed )
bluestore_cache_size_ssd = 1073741824

OK, I try later.

So the Node 5 is not up ? Where are the old OSDs now ? If they are not up, place the OSDs back in the node and turn it on. This will help a lot to recover the down pgs.

I try to reinstall the system OS and use the same IP. During the web interface setup, I join the node to  current cluster. The system can detect all OSD disks. But I did not add it because I am afraid that it will format these old osd disks. Then I quit the setup process. Now, I cannot connect the node 5 again by web interface. In the console page, I see that this node 5 has become the part of the cluster.

If I delete node 5, it will has this pop up:

Are you sure you wish to delete storage node node5 ? Any data stored in OSDs belonging to the node will be lost. If you still have working OSD disks please physically remove them and place them in other storage nodes before proceeding.

Any steps I was wrong. Please tell me the correct procedure. Shall I use setup the new node with another IP ?

root@node7:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 26.07910 root default
-3 6.51978 host node4
0 ssd 0.93140 osd.0 up 1.00000 1.00000
1 ssd 0.93140 osd.1 up 1.00000 1.00000
2 ssd 0.93140 osd.2 up 1.00000 1.00000
3 ssd 0.93140 osd.3 up 1.00000 1.00000
4 ssd 0.93140 osd.4 up 1.00000 1.00000
5 ssd 0.93140 osd.5 up 1.00000 1.00000
6 ssd 0.93140 osd.6 up 1.00000 1.00000
-5 6.51978 host node5
7 ssd 0.93140 osd.7 down 0 1.00000
8 ssd 0.93140 osd.8 down 0 1.00000
9 ssd 0.93140 osd.9 down 0 1.00000
10 ssd 0.93140 osd.10 down 0 1.00000
11 ssd 0.93140 osd.11 down 0 1.00000
12 ssd 0.93140 osd.12 down 0 1.00000
13 ssd 0.93140 osd.13 down 0 1.00000
-9 6.51978 host node6
21 ssd 0.93140 osd.21 up 1.00000 1.00000
22 ssd 0.93140 osd.22 up 1.00000 1.00000
23 ssd 0.93140 osd.23 up 1.00000 1.00000
24 ssd 0.93140 osd.24 up 1.00000 1.00000
25 ssd 0.93140 osd.25 up 1.00000 1.00000
26 ssd 0.93140 osd.26 up 1.00000 1.00000
27 ssd 0.93140 osd.27 up 1.00000 1.00000
-7 6.51978 host node7
14 ssd 0.93140 osd.14 up 1.00000 1.00000
15 ssd 0.93140 osd.15 up 1.00000 1.00000
16 ssd 0.93140 osd.16 up 1.00000 1.00000
17 ssd 0.93140 osd.17 up 1.00000 1.00000
18 ssd 0.93140 osd.18 up 1.00000 1.00000
19 ssd 0.93140 osd.19 up 1.00000 1.00000
20 ssd 0.93140 osd.20 up 1.00000 1.00000

 

root@node7:~# ceph pg 1.3fe query --cluster ceph
{
"state": "down",
"snap_trimq": "[]",
"snap_trimq_len": 0,
"epoch": 4326,
"up": [
6,
15
],
"acting": [
6,
15
],
"info": {
"pgid": "1.3fe",
"last_update": "209'285987",
"last_complete": "209'285987",
"log_tail": "206'284400",
"last_user_version": 285987,
"last_backfill": "MAX",
"last_backfill_bitwise": 0,
"purged_snaps": [
{
"start": "1",
"length": "3"
}
],
"history": {
"epoch_created": 2,
"epoch_pool_created": 2,
"last_epoch_started": 92,
"last_interval_started": 91,
"last_epoch_clean": 92,
"last_interval_clean": 91,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 4322,
"same_interval_since": 4322,
"same_primary_since": 4322,
"last_scrub": "206'285660",
"last_scrub_stamp": "2018-10-07 02:42:09.022545",
"last_deep_scrub": "206'251240",
"last_deep_scrub_stamp": "2018-10-04 20:36:18.485813",
"last_clean_scrub_stamp": "2018-10-07 02:42:09.022545"
},
"stats": {
"version": "209'285987",
"reported_seq": "525799",
"reported_epoch": "4326",
"state": "down",
"last_fresh": "2018-10-10 08:59:56.534175",
"last_change": "2018-10-10 08:59:52.396458",
"last_active": "2018-10-07 13:55:53.848842",
"last_peered": "2018-10-07 09:58:15.620040",
"last_clean": "2018-10-07 09:58:15.620040",
"last_became_active": "2018-09-04 09:36:05.916134",
"last_became_peered": "2018-09-04 09:36:05.916134",
"last_unstale": "2018-10-10 08:59:56.534175",
"last_undegraded": "2018-10-10 08:59:56.534175",
"last_fullsized": "2018-10-10 08:59:56.534175",
"mapping_epoch": 4322,
"log_start": "206'284400",
"ondisk_log_start": "206'284400",
"created": 2,
"last_epoch_clean": 92,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "206'285660",
"last_scrub_stamp": "2018-10-07 02:42:09.022545",
"last_deep_scrub": "206'251240",
"last_deep_scrub_stamp": "2018-10-04 20:36:18.485813",
"last_clean_scrub_stamp": "2018-10-07 02:42:09.022545",
"log_size": 1587,
"ondisk_log_size": 1587,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 2913992704,
"num_objects": 695,
"num_object_clones": 0,
"num_object_copies": 1390,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 695,
"num_whiteouts": 0,
"num_read": 237038,
"num_read_kb": 11396760,
"num_write": 570364,
"num_write_kb": 7966108,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0
},
"up": [
6,
15
],
"acting": [
6,
15
],
"blocked_by": [
12
],
"up_primary": 6,
"acting_primary": 6
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 92,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "15",
"pgid": "1.3fe",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"last_backfill_bitwise": 0,
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000"
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0
},
"up": [],
"acting": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "24",
"pgid": "1.3fe",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"last_backfill_bitwise": 0,
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000"
},
"stats": {
"version": "0'0",
"reported_seq": "0",
"reported_epoch": "0",
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"log_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0
},
"up": [],
"acting": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Peering/Down",
"enter_time": "2018-10-10 08:59:52.396439",
"comment": "not enough up instances of this PG to go active"
},
{
"name": "Started/Primary/Peering",
"enter_time": "2018-10-10 08:59:52.396417",
"past_intervals": [
{
"first": "91",
"last": "4321",
"all_participants": [
{
"osd": 6
},
{
"osd": 12
},
{
"osd": 15
},
{
"osd": 24
}
],
"intervals": [
{
"first": "223",
"last": "225",
"acting": "12"
},
{
"first": "4237",
"last": "4243",
"acting": "24"
},
{
"first": "4275",
"last": "4276",
"acting": "15"
},
{
"first": "4307",
"last": "4308",
"acting": "6"
}
]
}
],
"probing_osds": [
"6",
"15",
"24"
],
"blocked": "peering is blocked due to down osds",
"down_osds_we_would_probe": [
12
],
"peering_blocked_by": [
{
"osd": 12,
"current_lost_at": 0,
"comment": "starting or marking this osd lost may let us proceed"
}
]
},
{
"name": "Started",
"enter_time": "2018-10-10 08:59:52.396394"
}
],
"agent_state": {}
}
root@node7:~#

#7

admin
2,930 Posts
October 10, 2018, 12:18 pm

Install the PetaSAN system on the OS disk,  choose same hostname and ip. In the deployment wizard select join cluster, do not select to add the old osd disks, as stated this will format them first. Once deployment is done reboot the node, this will activate the old osds.

 

#8

Yipkaiwing
18 Posts
October 11, 2018, 3:17 am

Dear Admin

Thank you so much. After adding back the Node 5 with the old OSD, the ceph storage resumed healthy immediately. Afterwards, I reboot all other nodes, all ISCSI drives are functioning normally.

My VM servers can connect the Petsan again.

I really appreciate your kind and useful  support.

The Petasan project is great.

Regards

Ben

#9

admin
2,930 Posts
October 11, 2018, 6:43 am

Glad things are working now...and thanks for the nice comment 🙂

To understand why this problem occurred and how to prevent this in future:

Under normal cases, a single node failure should not cause the cluster to stop, but in this case you had 166 PGs down. Looking at PG 1.3fe example: it had 2 replicas on OSDs 6 and 12 on nodes 4 and 5. When node 5 crashed, a replacement OSD 15 on node 7 was to become the second replica for this PG, in normal case the PG would still be serving client io but be in degraded state (ie active/degraded) until OSD 15 creates a copy from 6 then it will be active/clean. Due to the resource load on the cluster, RAM maxed out and nodes were going up and down  + you had rebooted the nodes after changing the RAM settings. Now when OSD 6 rebooted, OSD 6 cannot assume the copy of the data it has is up to date, OSD 15 has no data, so it tries to peer with 12:

"peering_blocked_by": [
{
"osd": 12,
"current_lost_at": 0,
"comment": "starting or marking this osd lost may let us proceed"
}

The solution is to either start OSD 12 even on a different node, or telling Ceph to ignore this peering and assume OSD 6 is up to date.

It is important to make sure this does not happen again. Many people test their cluster hardware performance by stressing it with the max client workload they expect, this is not enough, it needs to be tested in conjunction with a recovery operation being simulated. So i suggest you do such tests by shutting down a node with client io being written and making sure your cluster is able to handle it, your resources should not exceed 80% utilization in such conditions.
Also for better safety you may increase your replica count to 3 instead of 2, so even if 2 nodes and their disks get damaged you still have a third copy, but tune your hardware first...Good luck.

Last edited on October 11, 2018, 6:48 am by admin · #10