Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Benchmark fails with "Error loading benchmark test report" on a new petasan 2.0.0 cluster

Pages: 1 2

tigerlinux
25 Posts
May 16, 2018, 8:30 pm

Greetings,

We installed a new Petasan 2.0.0 cluster (3 nodes, ciscos cm240-m3s, 64gb ram, 4 SSD 1TB OSD's, 24 cpu threads, 4 nics) and everything is working OK but the benchmark.

Anytime we try to run the cluster benchmark (1 minute, 5 minute, 15 minute, with the 4K option) it ends with "Error loading benchmark test report". We can observe the following errors on the log:

16/05/2018 16:27:02 INFO Benchmark Test Completed
AttributeError: 'NoneType' object has no attribute 'write_json'
result = result.write_json()
File "/opt/petasan/scripts/util/benchmark.py", line 133, in manager
Traceback (most recent call last):
16/05/2018 16:27:01 ERROR 'NoneType' object has no attribute 'write_json'
JSONDecodeError: Expecting value: line 2 column 1 (char 1)
return self.scan_once(s, idx=_w(s, idx).end())
File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 400, in raw_decode
obj, end = self.raw_decode(s)
File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 370, in decode
return cls(encoding=encoding, **kw).decode(s)
File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line 533, in loads
return _json.loads(s, **kwargs)
File "/usr/lib/python2.7/dist-packages/flask/json.py", line 149, in loads
self.__dict__ = json.loads(j)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/entity/benchmark.py", line 68, in load_json
sar_rs.load_json(out)
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/cluster/benchmark.py", line 165, in __write
self.__write()
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/cluster/benchmark.py", line 96, in manager
Traceback (most recent call last):
16/05/2018 16:27:00 ERROR Expecting value: line 2 column 1 (char 1)
16/05/2018 16:26:36 INFO Run sar state cmd on node mts-boo101cep03.host.mts.local : python /opt/petasan/scripts/jobs/benchmark/storage_load.py -d 15
16/05/2018 16:26:36 INFO Run sar state cmd on node mts-boo101cep02.host.mts.local : python /opt/petasan/scripts/jobs/benchmark/storage_load.py -d 15
16/05/2018 16:26:29 INFO Benchmark client cmd.
16/05/2018 16:26:28 INFO Wait time before collect storage state.
16/05/2018 16:26:28 INFO Start client stress job for rados
16/05/2018 16:26:28 INFO Run rados write cmd on node mts-boo101cep01.host.mts.local : python /opt/petasan/scripts/jobs/benchmark/client_stress.py -d 30 -t 16 -b 4096 -m w
16/05/2018 16:26:28 INFO Benchmark start rados write.
16/05/2018 16:26:27 INFO Benchmark manager cmd.
16/05/2018 16:26:26 INFO Benchmark Test Started
16/05/2018 16:24:12 INFO Benchmark client cmd.
16/05/2018 16:24:12 INFO Start client stress job for rados
ZeroDivisionError: integer division or modulo by zero
self.__stat.disk_avg = round(self.__stat.disk_avg / (len(self.__stat.disks)), 2)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/common/sar.py", line 109, in __process_sar_output_section
self.__process_sar_output_section()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/common/sar.py", line 51, in run
return Sar().run(duration)
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/cluster/benchmark.py", line 55, in sar_stats
result = Benchmark().sar_stats(args.d)
File "/opt/petasan/scripts/util/benchmark.py", line 88, in storage
Traceback (most recent call last):
16/05/2018 16:23:35 ERROR integer division or modulo by zero
16/05/2018 16:23:19 INFO Benchmark storage cmd.
16/05/2018 16:23:19 INFO Start storage load job for 'sar'
16/05/2018 16:20:31 INFO Benchmark Test Completed
AttributeError: 'NoneType' object has no attribute 'write_json'
result = result.write_json()
File "/opt/petasan/scripts/util/benchmark.py", line 133, in manager
Traceback (most recent call last):
16/05/2018 16:20:30 ERROR 'NoneType' object has no attribute 'write_json'
JSONDecodeError: Expecting value: line 2 column 1 (char 1)
return self.scan_once(s, idx=_w(s, idx).end())
File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 400, in raw_decode
obj, end = self.raw_decode(s)
File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 370, in decode
return cls(encoding=encoding, **kw).decode(s)
File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line 533, in loads
return _json.loads(s, **kwargs)
File "/usr/lib/python2.7/dist-packages/flask/json.py", line 149, in loads
self.__dict__ = json.loads(j)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/entity/benchmark.py", line 68, in load_json
sar_rs.load_json(out)
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/cluster/benchmark.py", line 165, in __write
self.__write()
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/cluster/benchmark.py", line 96, in manager
Traceback (most recent call last):
16/05/2018 16:20:29 ERROR Expecting value: line 2 column 1 (char 1)

We want to start into production soon. What could it be the problem here?

#1

admin
2,930 Posts
May 16, 2018, 9:24 pm

what is the output of

ceph osd tree --cluster CLUSTER_NAME

on all 3 nodes what is the output of:

sar 3 1 -u -P ALL -r -n DEV -d -p

#2

tigerlinux
25 Posts
May 17, 2018, 1:58 pm

Ok. First command:

ceph osd tree --cluster ceph
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 10.91510 root default
-5 3.63837 host mts-boo101cep01
4 ssd 0.90959 osd.4 up 1.00000 1.00000
5 ssd 0.90959 osd.5 up 1.00000 1.00000
6 ssd 0.90959 osd.6 up 1.00000 1.00000
7 ssd 0.90959 osd.7 up 1.00000 1.00000
-7 3.63837 host mts-boo101cep02
8 ssd 0.90959 osd.8 up 1.00000 1.00000
9 ssd 0.90959 osd.9 up 1.00000 1.00000
10 ssd 0.90959 osd.10 up 1.00000 1.00000
11 ssd 0.90959 osd.11 up 1.00000 1.00000
-3 3.63837 host mts-boo101cep03
0 ssd 0.90959 osd.0 up 1.00000 1.00000
1 ssd 0.90959 osd.1 up 1.00000 1.00000
2 ssd 0.90959 osd.2 up 1.00000 1.00000
3 ssd 0.90959 osd.3 up 1.00000 1.00000

For the other commands:

Node 01 (mts-boo101cep01)

Linux 4.4.92-09-petasan (mts-boo101cep01.host.mts.local) 05/17/18 _x86_64_ (32 CPU)

09:52:05 CPU %user %nice %system %iowait %steal %idle
09:52:08 all 0.08 0.00 0.08 0.00 0.00 99.83
09:52:08 0 0.33 0.00 0.66 0.00 0.00 99.00
09:52:08 1 0.33 0.00 0.00 0.00 0.00 99.67
09:52:08 2 0.00 0.00 0.00 0.33 0.00 99.67
09:52:08 3 0.33 0.00 0.33 0.00 0.00 99.34
09:52:08 4 0.33 0.00 0.33 0.00 0.00 99.34
09:52:08 5 0.00 0.00 0.33 0.00 0.00 99.67
09:52:08 6 0.33 0.00 0.00 0.00 0.00 99.67
09:52:08 7 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 8 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 9 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 10 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 11 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 12 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 13 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 14 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 15 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 16 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 17 0.33 0.00 0.33 0.00 0.00 99.34
09:52:08 18 0.00 0.00 0.33 0.00 0.00 99.67
09:52:08 19 0.33 0.00 0.00 0.00 0.00 99.67
09:52:08 20 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 21 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 22 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 23 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 24 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 25 0.00 0.00 0.33 0.00 0.00 99.67
09:52:08 26 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 27 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 28 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 29 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 30 0.00 0.00 0.00 0.00 0.00 100.00
09:52:08 31 0.00 0.00 0.00 0.00 0.00 100.00

09:52:05 kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
09:52:08 63604952 2236936 3.40 270684 254900 5859856 8.90 1776664 118020 108

09:52:05 DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
09:52:08 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:08 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:08 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:08 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:08 sda 4.67 0.00 69.33 14.86 0.00 0.86 0.86 0.40

09:52:05 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
09:52:08 lo 23.33 23.33 44.95 44.95 0.00 0.00 0.00 0.00
09:52:08 eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:08 eth2 83.33 117.67 68.00 41.29 0.00 0.00 0.00 0.01
09:52:08 eth1 34.67 44.00 26.87 27.37 0.00 0.00 0.00 0.00
09:52:08 eth0 1.00 0.00 0.09 0.00 0.00 0.00 0.67 0.00

Average: CPU %user %nice %system %iowait %steal %idle
Average: all 0.08 0.00 0.08 0.00 0.00 99.83
Average: 0 0.33 0.00 0.66 0.00 0.00 99.00
Average: 1 0.33 0.00 0.00 0.00 0.00 99.67
Average: 2 0.00 0.00 0.00 0.33 0.00 99.67
Average: 3 0.33 0.00 0.33 0.00 0.00 99.34
Average: 4 0.33 0.00 0.33 0.00 0.00 99.34
Average: 5 0.00 0.00 0.33 0.00 0.00 99.67
Average: 6 0.33 0.00 0.00 0.00 0.00 99.67
Average: 7 0.00 0.00 0.00 0.00 0.00 100.00
Average: 8 0.00 0.00 0.00 0.00 0.00 100.00
Average: 9 0.00 0.00 0.00 0.00 0.00 100.00
Average: 10 0.00 0.00 0.00 0.00 0.00 100.00
Average: 11 0.00 0.00 0.00 0.00 0.00 100.00
Average: 12 0.00 0.00 0.00 0.00 0.00 100.00
Average: 13 0.00 0.00 0.00 0.00 0.00 100.00
Average: 14 0.00 0.00 0.00 0.00 0.00 100.00
Average: 15 0.00 0.00 0.00 0.00 0.00 100.00
Average: 16 0.00 0.00 0.00 0.00 0.00 100.00
Average: 17 0.33 0.00 0.33 0.00 0.00 99.34
Average: 18 0.00 0.00 0.33 0.00 0.00 99.67
Average: 19 0.33 0.00 0.00 0.00 0.00 99.67
Average: 20 0.00 0.00 0.00 0.00 0.00 100.00
Average: 21 0.00 0.00 0.00 0.00 0.00 100.00
Average: 22 0.00 0.00 0.00 0.00 0.00 100.00
Average: 23 0.00 0.00 0.00 0.00 0.00 100.00
Average: 24 0.00 0.00 0.00 0.00 0.00 100.00
Average: 25 0.00 0.00 0.33 0.00 0.00 99.67
Average: 26 0.00 0.00 0.00 0.00 0.00 100.00
Average: 27 0.00 0.00 0.00 0.00 0.00 100.00
Average: 28 0.00 0.00 0.00 0.00 0.00 100.00
Average: 29 0.00 0.00 0.00 0.00 0.00 100.00
Average: 30 0.00 0.00 0.00 0.00 0.00 100.00
Average: 31 0.00 0.00 0.00 0.00 0.00 100.00

Average: kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
Average: 63604952 2236936 3.40 270684 254900 5859856 8.90 1776664 118020 108

Average: DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
Average: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: sda 4.67 0.00 69.33 14.86 0.00 0.86 0.86 0.40

Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
Average: lo 23.33 23.33 44.95 44.95 0.00 0.00 0.00 0.00
Average: eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth2 83.33 117.67 68.00 41.29 0.00 0.00 0.00 0.01
Average: eth1 34.67 44.00 26.87 27.37 0.00 0.00 0.00 0.00
Average: eth0 1.00 0.00 0.09 0.00 0.00 0.00 0.67 0.00

Node 02 (mts-boo101cep02)

Linux 4.4.92-09-petasan (mts-boo101cep02.host.mts.local) 05/17/18 _x86_64_ (32 CPU)

09:52:16 CPU %user %nice %system %iowait %steal %idle
09:52:19 all 0.04 0.00 0.01 0.01 0.00 99.94
09:52:19 0 0.33 0.00 0.00 0.33 0.00 99.34
09:52:19 1 0.00 0.00 0.33 0.00 0.00 99.67
09:52:19 2 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 3 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 4 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 5 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 6 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 7 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 8 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 9 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 10 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 11 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 12 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 13 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 14 0.33 0.00 0.00 0.00 0.00 99.67
09:52:19 15 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 16 0.33 0.00 0.00 0.00 0.00 99.67
09:52:19 17 0.00 0.00 0.33 0.00 0.00 99.67
09:52:19 18 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 19 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 20 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 21 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 22 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 23 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 24 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 25 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 26 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 27 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 28 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 29 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 30 0.00 0.00 0.00 0.00 0.00 100.00
09:52:19 31 0.00 0.00 0.00 0.00 0.00 100.00

09:52:16 kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
09:52:19 63405276 2350796 3.58 306752 320940 5743372 8.73 1812576 187536 344

09:52:16 DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
09:52:19 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:19 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:19 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:19 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:19 sda 17.33 0.00 208.00 12.00 0.01 0.62 0.23 0.40

09:52:16 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
09:52:19 lo 27.33 27.33 17.64 17.64 0.00 0.00 0.00 0.00
09:52:19 eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:19 eth2 91.00 91.67 44.89 77.56 0.00 0.00 0.00 0.01
09:52:19 eth1 50.67 64.00 38.85 39.57 0.00 0.00 0.00 0.00
09:52:19 eth0 1.00 0.00 0.09 0.00 0.00 0.00 0.67 0.00

Average: CPU %user %nice %system %iowait %steal %idle
Average: all 0.04 0.00 0.01 0.01 0.00 99.94
Average: 0 0.33 0.00 0.00 0.33 0.00 99.34
Average: 1 0.00 0.00 0.33 0.00 0.00 99.67
Average: 2 0.00 0.00 0.00 0.00 0.00 100.00
Average: 3 0.00 0.00 0.00 0.00 0.00 100.00
Average: 4 0.00 0.00 0.00 0.00 0.00 100.00
Average: 5 0.00 0.00 0.00 0.00 0.00 100.00
Average: 6 0.00 0.00 0.00 0.00 0.00 100.00
Average: 7 0.00 0.00 0.00 0.00 0.00 100.00
Average: 8 0.00 0.00 0.00 0.00 0.00 100.00
Average: 9 0.00 0.00 0.00 0.00 0.00 100.00
Average: 10 0.00 0.00 0.00 0.00 0.00 100.00
Average: 11 0.00 0.00 0.00 0.00 0.00 100.00
Average: 12 0.00 0.00 0.00 0.00 0.00 100.00
Average: 13 0.00 0.00 0.00 0.00 0.00 100.00
Average: 14 0.33 0.00 0.00 0.00 0.00 99.67
Average: 15 0.00 0.00 0.00 0.00 0.00 100.00
Average: 16 0.33 0.00 0.00 0.00 0.00 99.67
Average: 17 0.00 0.00 0.33 0.00 0.00 99.67
Average: 18 0.00 0.00 0.00 0.00 0.00 100.00
Average: 19 0.00 0.00 0.00 0.00 0.00 100.00
Average: 20 0.00 0.00 0.00 0.00 0.00 100.00
Average: 21 0.00 0.00 0.00 0.00 0.00 100.00
Average: 22 0.00 0.00 0.00 0.00 0.00 100.00
Average: 23 0.00 0.00 0.00 0.00 0.00 100.00
Average: 24 0.00 0.00 0.00 0.00 0.00 100.00
Average: 25 0.00 0.00 0.00 0.00 0.00 100.00
Average: 26 0.00 0.00 0.00 0.00 0.00 100.00
Average: 27 0.00 0.00 0.00 0.00 0.00 100.00
Average: 28 0.00 0.00 0.00 0.00 0.00 100.00
Average: 29 0.00 0.00 0.00 0.00 0.00 100.00
Average: 30 0.00 0.00 0.00 0.00 0.00 100.00
Average: 31 0.00 0.00 0.00 0.00 0.00 100.00

Average: kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
Average: 63405276 2350796 3.58 306752 320940 5743372 8.73 1812576 187536 344

Average: DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
Average: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: sda 17.33 0.00 208.00 12.00 0.01 0.62 0.23 0.40

Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
Average: lo 27.33 27.33 17.64 17.64 0.00 0.00 0.00 0.00
Average: eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth2 91.00 91.67 44.89 77.56 0.00 0.00 0.00 0.01
Average: eth1 50.67 64.00 38.85 39.57 0.00 0.00 0.00 0.00
Average: eth0 1.00 0.00 0.09 0.00 0.00 0.00 0.67 0.00

Node 03 (mts-boo101cep03)

Linux 4.4.92-09-petasan (mts-boo101cep03.host.mts.local) 05/17/18 _x86_64_ (40 CPU)

09:52:23 CPU %user %nice %system %iowait %steal %idle
09:52:26 all 0.03 0.00 0.03 0.01 0.00 99.94
09:52:26 0 0.33 0.00 0.33 0.00 0.00 99.33
09:52:26 1 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 2 0.00 0.00 0.33 0.33 0.00 99.34
09:52:26 3 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 4 0.33 0.00 0.00 0.00 0.00 99.67
09:52:26 5 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 6 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 7 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 8 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 9 0.00 0.00 0.00 0.33 0.00 99.67
09:52:26 10 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 11 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 12 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 13 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 14 0.00 0.00 0.33 0.00 0.00 99.67
09:52:26 15 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 16 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 17 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 18 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 19 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 20 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 21 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 22 0.00 0.00 0.00 0.33 0.00 99.67
09:52:26 23 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 24 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 25 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 26 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 27 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 28 0.00 0.00 0.33 0.00 0.00 99.67
09:52:26 29 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 30 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 31 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 32 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 33 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 34 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 35 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 36 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 37 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 38 0.00 0.00 0.00 0.00 0.00 100.00
09:52:26 39 0.00 0.00 0.00 0.00 0.00 100.00

09:52:23 kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
09:52:26 63111464 2643412 4.02 254500 349936 7457464 11.34 2070368 189468 176

09:52:23 DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
09:52:26 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:26 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:26 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:26 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:26 sda 53.00 0.00 584.00 11.02 0.23 4.38 0.10 0.53

09:52:23 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
09:52:26 lo 17.33 17.33 17.12 17.12 0.00 0.00 0.00 0.00
09:52:26 eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:52:26 eth2 123.00 77.33 64.99 118.12 0.00 0.00 0.00 0.01
09:52:26 eth1 77.33 43.00 48.90 46.25 0.00 0.00 0.00 0.00
09:52:26 eth0 1.33 0.00 0.12 0.00 0.00 0.00 1.00 0.00

Average: CPU %user %nice %system %iowait %steal %idle
Average: all 0.03 0.00 0.03 0.01 0.00 99.94
Average: 0 0.33 0.00 0.33 0.00 0.00 99.33
Average: 1 0.00 0.00 0.00 0.00 0.00 100.00
Average: 2 0.00 0.00 0.33 0.33 0.00 99.34
Average: 3 0.00 0.00 0.00 0.00 0.00 100.00
Average: 4 0.33 0.00 0.00 0.00 0.00 99.67
Average: 5 0.00 0.00 0.00 0.00 0.00 100.00
Average: 6 0.00 0.00 0.00 0.00 0.00 100.00
Average: 7 0.00 0.00 0.00 0.00 0.00 100.00
Average: 8 0.00 0.00 0.00 0.00 0.00 100.00
Average: 9 0.00 0.00 0.00 0.33 0.00 99.67
Average: 10 0.00 0.00 0.00 0.00 0.00 100.00
Average: 11 0.00 0.00 0.00 0.00 0.00 100.00
Average: 12 0.00 0.00 0.00 0.00 0.00 100.00
Average: 13 0.00 0.00 0.00 0.00 0.00 100.00
Average: 14 0.00 0.00 0.33 0.00 0.00 99.67
Average: 15 0.00 0.00 0.00 0.00 0.00 100.00
Average: 16 0.00 0.00 0.00 0.00 0.00 100.00
Average: 17 0.00 0.00 0.00 0.00 0.00 100.00
Average: 18 0.00 0.00 0.00 0.00 0.00 100.00
Average: 19 0.00 0.00 0.00 0.00 0.00 100.00
Average: 20 0.00 0.00 0.00 0.00 0.00 100.00
Average: 21 0.00 0.00 0.00 0.00 0.00 100.00
Average: 22 0.00 0.00 0.00 0.33 0.00 99.67
Average: 23 0.00 0.00 0.00 0.00 0.00 100.00
Average: 24 0.00 0.00 0.00 0.00 0.00 100.00
Average: 25 0.00 0.00 0.00 0.00 0.00 100.00
Average: 26 0.00 0.00 0.00 0.00 0.00 100.00
Average: 27 0.00 0.00 0.00 0.00 0.00 100.00
Average: 28 0.00 0.00 0.33 0.00 0.00 99.67
Average: 29 0.00 0.00 0.00 0.00 0.00 100.00
Average: 30 0.00 0.00 0.00 0.00 0.00 100.00
Average: 31 0.00 0.00 0.00 0.00 0.00 100.00
Average: 32 0.00 0.00 0.00 0.00 0.00 100.00
Average: 33 0.00 0.00 0.00 0.00 0.00 100.00
Average: 34 0.00 0.00 0.00 0.00 0.00 100.00
Average: 35 0.00 0.00 0.00 0.00 0.00 100.00
Average: 36 0.00 0.00 0.00 0.00 0.00 100.00
Average: 37 0.00 0.00 0.00 0.00 0.00 100.00
Average: 38 0.00 0.00 0.00 0.00 0.00 100.00
Average: 39 0.00 0.00 0.00 0.00 0.00 100.00

Average: kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
Average: 63111464 2643412 4.02 254500 349936 7457464 11.34 2070368 189468 176

Average: DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
Average: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: sda 53.00 0.00 584.00 11.02 0.23 4.38 0.10 0.53

Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
Average: lo 17.33 17.33 17.12 17.12 0.00 0.00 0.00 0.00
Average: eth3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth2 123.00 77.33 64.99 118.12 0.00 0.00 0.00 0.01
Average: eth1 77.33 43.00 48.90 46.25 0.00 0.00 0.00 0.00
Average: eth0 1.33 0.00 0.12 0.00 0.00 0.00 1.00 0.00

 

Last edited on May 17, 2018, 1:59 pm by tigerlinux · #3

tigerlinux
25 Posts
May 17, 2018, 2:03 pm

More information:

  • eth0 = management net (1Gbps nic, 1500 mtu)
  • eth1 = Ceph backend 2 (10Gbps, 9000 mtu)
  • eth2 = iscsi1 + ceph backend 1 (10Gbps, 9000 mtu)
  • eth3 = iscsi2 (10Gbps, 9000 mtu)

All servers where calculated (cpu and ram) to eventually support 16 1TB OSD SSD's but currently we are using just 4 x node (adding a new one x node next week, and all remaining up to 16 1TB SSD OSD's in the following months).

#4

tigerlinux
25 Posts
May 17, 2018, 2:20 pm

One more thing. I tried some minutes ago the other kind of benchmark (the 4M rados throughput) and I'm seeing the following errors on the logs:

ZeroDivisionError: integer division or modulo by zero
self.__stat.disk_avg = round(self.__stat.disk_avg / (len(self.__stat.disks)), 2)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/common/sar.py", line 109, in __process_sar_output_section
self.__process_sar_output_section()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/common/sar.py", line 51, in run
return Sar().run(duration)
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/cluster/benchmark.py", line 55, in sar_stats
result = Benchmark().sar_stats(args.d)
File "/opt/petasan/scripts/util/benchmark.py", line 88, in storage
Traceback (most recent call last):
17/05/2018 10:13:52 ERROR integer division or modulo by zero
17/05/2018 10:13:36 INFO Benchmark storage cmd.
17/05/2018 10:13:36 INFO Start storage load job for 'sar'
ZeroDivisionError: integer division or modulo by zero
self.__stat.disk_avg = round(self.__stat.disk_avg / (len(self.__stat.disks)), 2)
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/common/sar.py", line 109, in __process_sar_output_section
self.__process_sar_output_section()
File "/usr/lib/python2.7/dist-packages/PetaSAN/core/common/sar.py", line 51, in run
return Sar().run(duration)
File "/usr/lib/python2.7/dist-packages/PetaSAN/backend/cluster/benchmark.py", line 55, in sar_stats
result = Benchmark().sar_stats(args.d)
File "/opt/petasan/scripts/util/benchmark.py", line 88, in storage

#5

admin
2,930 Posts
May 17, 2018, 7:58 pm

It does seem like a divide by zero when calculating averages of disk utilization, for some reason the benchmark code it is not detecting any disks. Have you made any changes by chance, was it working before or it never worked before ? If you have not made changes, we will provide you with benchmark code than includes more log messages to get more insight on why the divide by zero occurs, probably by early next week.

#6

tigerlinux
25 Posts
May 17, 2018, 8:31 pm

Hi Admin,

Nope. The cluster is brand new and waiting for enter production. Sadly, we just found another problem that maybe could be related to the same source. This time with the path reassignment panel. It is not showing any of the paths.

We had just created two disks, one with 2 paths on both ISCSI nets. The disks are being detected OK by our vmware-based platform but the "path reassignment panel" is not showing any path on the nodes. Each node is shown with 0 paths.

To be sure we are not doing anything wrong (and we tested this a lot before with a reference virtualized lab) I activated the reference lab we have (3 virtual nodes 4 nics x server but with 2 OSDs x node) and the path reassignment panel is working there.

I stopped the disks, detached them, re-attached them, changed the paths (from 3 to 2), but the situation is the same. No paths on the nodes using the path reassignment panel.

Logs from one of the nodes:

17/05/2018 16:08:25 INFO Could not lock path 00002/2 with session d362eb5b-89f0-b923-d7ea-fa23f192a281.
17/05/2018 16:07:45 INFO Path 00001/2 acquired successfully
17/05/2018 16:07:42 INFO Image image-00001 mapped successfully.
17/05/2018 16:06:35 INFO PetaSAN removed key of stopped disk 00002 from consul.
17/05/2018 16:06:33 INFO Image image-00002 unmapped successfully.
17/05/2018 16:06:33 INFO LIO deleted Target iqn.2016-05.com.petasan:00002
17/05/2018 16:06:32 INFO LIO deleted backstore image image-00002
17/05/2018 16:06:32 ERROR Could not find ips for image-00002
17/05/2018 16:06:32 INFO Stopping disk 00002
17/05/2018 16:06:32 INFO PetaSAN removed key of stopped disk 00001 from consul.
17/05/2018 16:06:30 INFO Image image-00001 unmapped successfully.
17/05/2018 16:06:30 INFO LIO deleted Target iqn.2016-05.com.petasan:00001
17/05/2018 16:06:30 INFO LIO deleted backstore image image-00001
17/05/2018 16:06:30 ERROR Could not find ips for image-00001
17/05/2018 16:06:30 INFO Stopping disk 00001
17/05/2018 16:06:30 INFO PetaSAN cleaned local paths not locked by this node in consul.
17/05/2018 16:06:30 INFO Cleaned disk path 00002/1.
17/05/2018 16:06:30 INFO Cleaned disk path 00001/2.
Exception: Error running echo command :echo 'PetaSAN.NodeStats.mts-boo101cep02.host.mts.local.cpu_all.percent_util 0.04' `date +%s` | nc -q0 10.12.5.42 2003
raise Exception("Error running echo command :" + cmd)
File "/opt/petasan/scripts/node_stats.py", line 48, in _send_graphite
_send_graphite(path_key, val)

#7

tigerlinux
25 Posts
May 17, 2018, 8:49 pm

Admin,

Do you think that the issue with the path reassignment and the issue with the benchmark could be related? Next one of our nodes "lsblk" and "blkid" commands:

lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 111.8G 0 disk
|-sda1 8:1 0 964.8M 0 part /boot
|-sda2 8:2 0 14G 0 part /
|-sda3 8:3 0 9.3G 0 part /var/lib/ceph
`-sda4 8:4 0 87.6G 0 part /opt/petasan/config
sdb 8:16 0 931.5G 0 disk
|-sdb1 8:17 0 100M 0 part /var/lib/ceph/osd/ceph-5
`-sdb2 8:18 0 931.4G 0 part
sdc 8:32 0 931.5G 0 disk
|-sdc1 8:33 0 100M 0 part /var/lib/ceph/osd/ceph-6
`-sdc2 8:34 0 931.4G 0 part
sdd 8:48 0 931.5G 0 disk
|-sdd1 8:49 0 100M 0 part /var/lib/ceph/osd/ceph-4
`-sdd2 8:50 0 931.4G 0 part
sde 8:64 0 931.5G 0 disk
|-sde1 8:65 0 100M 0 part /var/lib/ceph/osd/ceph-7
`-sde2 8:66 0 931.4G 0 part
rbd0 254:0 0 300G 0 disk
rbd1 254:16 0 250G 0 disk

 

blkid
/dev/sdb1: UUID="f68ed0ed-c264-438c-a919-9c9fb0e7c123" TYPE="xfs" PARTLABEL="ceph data" PARTUUID="a948ad57-7cb9-41ba-b4a6-31143c8cfb0e"
/dev/sdb2: PARTLABEL="ceph block" PARTUUID="ed3fdfcd-f5d3-4b0f-a087-58f3a073f103"
/dev/sdc1: UUID="7f4d0dfb-82ad-4e1e-9bee-94d761873009" TYPE="xfs" PARTLABEL="ceph data" PARTUUID="ddca5fde-fcb2-4481-be93-93f837cb462c"
/dev/sdc2: PARTLABEL="ceph block" PARTUUID="04910810-4563-4b5f-8719-eeff3c42eda9"
/dev/sdd1: UUID="ff26f1a8-5069-45e8-ac89-8409467bdceb" TYPE="xfs" PARTLABEL="ceph data" PARTUUID="695bb5d7-5357-43a8-835a-9eb803b9b8fc"
/dev/sdd2: PARTLABEL="ceph block" PARTUUID="50529837-ebb9-448d-bdb3-8fcf87e49fb0"
/dev/sde1: UUID="a54a9f26-677e-445b-ae26-2a9345391a55" TYPE="xfs" PARTLABEL="ceph data" PARTUUID="25690758-0ae1-45b0-9cb4-c982bc6f9d58"
/dev/sde2: PARTLABEL="ceph block" PARTUUID="66cb0505-3bfc-4de2-891f-8da0c0c6ed35"
/dev/sda1: UUID="97de3da0-bea0-40f9-b691-afeff934e5ee" TYPE="ext4"
/dev/sda2: LABEL="PetaSANSystem" UUID="434401f2-99a8-4540-aac1-63e923ffa48d" TYPE="ext4"
/dev/sda3: UUID="fcaebf50-7337-45a0-a0ba-c639ff830287" TYPE="ext4"
/dev/sda4: UUID="2766abdf-e91f-416b-a5e9-9d5c6dc90dc4" TYPE="ext4"
/dev/rbd0: PTUUID="cf142599-15bc-4b8b-896f-700380260dc3" PTTYPE="gpt"

The controller is a LSI 9210-81 configure on IT mode:

Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)

The OSD disks are Samsungs SSD Evo 860 1TB

Is there any change that the hardware could be causing problems with both the benchmark and the path reassignment panel?

 

#8

admin
2,930 Posts
May 17, 2018, 9:23 pm

it is strange not to detect any paths, hard to say why this is happening. If i understand you have another cluster working fine and this is brand new install, if so i would recommend you re-install and double check the settings.

if you still have issues let us know and we will look deeper.

#9

tigerlinux
25 Posts
May 18, 2018, 2:24 pm

Sure thing Admin.

I'll also check the hardware profile on Petasan. The LAB cluster is on the lowest settings but for this one I opted for "high-end". Maybe I should check mid-range or custom.

#10
Pages: 1 2