Percentage usage and rules I should follow
R3LZX
50 Posts
May 16, 2020, 5:48 pmQuote from R3LZX on May 16, 2020, 5:48 pmI have a three server cluster that is all RBD and is on the latest version, do you have a calculation that would allow me to understand what best practice I should follow for usage, for example since its three nodes and I have 9 OSD disks @ 90TB if the cluster is replicating X3 wouldn't that mean I can only really use 30TB ? of course the cluster will allow me to use more than this, and here is a list of what I am using now.
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
3 hdd 9.15329 1.00000 9.2 TiB 5.0 TiB 4.9 TiB 20 KiB 9.4 GiB 4.2 TiB 54.30 0.92 346 up
4 hdd 9.15329 1.00000 9.2 TiB 4.7 TiB 4.6 TiB 16 KiB 9.1 GiB 4.5 TiB 51.38 0.87 327 up
5 hdd 9.15329 1.00000 9.2 TiB 5.0 TiB 5.0 TiB 72 KiB 9.6 GiB 4.1 TiB 55.10 0.94 351 up
0 hdd 9.15329 1.00000 9.2 TiB 5.1 TiB 5.0 TiB 132 KiB 9.9 GiB 4.1 TiB 55.40 0.94 353 up
1 hdd 9.15329 1.00000 9.2 TiB 4.9 TiB 4.9 TiB 84 KiB 10 GiB 4.2 TiB 53.67 0.91 342 up
2 hdd 9.15329 1.00000 9.2 TiB 4.7 TiB 4.7 TiB 20 KiB 9.2 GiB 4.4 TiB 51.71 0.88 329 up
7 hdd 9.15329 1.00000 9.2 TiB 6.5 TiB 6.4 TiB 24 KiB 12 GiB 2.7 TiB 70.53 1.20 451 up
8 hdd 9.15329 1.00000 9.2 TiB 7.1 TiB 7.0 TiB 136 KiB 13 GiB 2.1 TiB 77.04 1.31 492 up
6 ssd 1.80469 1.00000 1.8 TiB 1.2 TiB 1.1 TiB 16 KiB 2.8 GiB 613 GiB 66.85 1.14 81 up
TOTAL 75 TiB 44 TiB 44 TiB 522 KiB 85 GiB 31 TiB 58.84
another question, if an ISCSI drive is used, and then the data is moved somewhere else does it take a long time to reclaim this space? or should the iscsi drive be completely removed to free up the needed space?
if I go to five servers, would a best practice be to going with only dual duplication instead of three? I am metaphorically thinking in terms of raid 5, 6, 10 etc at server level and just trying to plan for this financially.
I have a three server cluster that is all RBD and is on the latest version, do you have a calculation that would allow me to understand what best practice I should follow for usage, for example since its three nodes and I have 9 OSD disks @ 90TB if the cluster is replicating X3 wouldn't that mean I can only really use 30TB ? of course the cluster will allow me to use more than this, and here is a list of what I am using now.
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
3 hdd 9.15329 1.00000 9.2 TiB 5.0 TiB 4.9 TiB 20 KiB 9.4 GiB 4.2 TiB 54.30 0.92 346 up
4 hdd 9.15329 1.00000 9.2 TiB 4.7 TiB 4.6 TiB 16 KiB 9.1 GiB 4.5 TiB 51.38 0.87 327 up
5 hdd 9.15329 1.00000 9.2 TiB 5.0 TiB 5.0 TiB 72 KiB 9.6 GiB 4.1 TiB 55.10 0.94 351 up
0 hdd 9.15329 1.00000 9.2 TiB 5.1 TiB 5.0 TiB 132 KiB 9.9 GiB 4.1 TiB 55.40 0.94 353 up
1 hdd 9.15329 1.00000 9.2 TiB 4.9 TiB 4.9 TiB 84 KiB 10 GiB 4.2 TiB 53.67 0.91 342 up
2 hdd 9.15329 1.00000 9.2 TiB 4.7 TiB 4.7 TiB 20 KiB 9.2 GiB 4.4 TiB 51.71 0.88 329 up
7 hdd 9.15329 1.00000 9.2 TiB 6.5 TiB 6.4 TiB 24 KiB 12 GiB 2.7 TiB 70.53 1.20 451 up
8 hdd 9.15329 1.00000 9.2 TiB 7.1 TiB 7.0 TiB 136 KiB 13 GiB 2.1 TiB 77.04 1.31 492 up
6 ssd 1.80469 1.00000 1.8 TiB 1.2 TiB 1.1 TiB 16 KiB 2.8 GiB 613 GiB 66.85 1.14 81 up
TOTAL 75 TiB 44 TiB 44 TiB 522 KiB 85 GiB 31 TiB 58.84
another question, if an ISCSI drive is used, and then the data is moved somewhere else does it take a long time to reclaim this space? or should the iscsi drive be completely removed to free up the needed space?
if I go to five servers, would a best practice be to going with only dual duplication instead of three? I am metaphorically thinking in terms of raid 5, 6, 10 etc at server level and just trying to plan for this financially.
admin
2,930 Posts
May 17, 2020, 3:35 pmQuote from admin on May 17, 2020, 3:35 pm-Yes in replication x 3 , if you have 90 TB of raw storage, the client will see 30 TB net
-Depends, if you delete ta file at the filesystem layer, the block storage will not know unless the filesystem sends a discard/trim commad. else the block layer will report the storage as used, but the filesystems knows the block are free and will over-write them. If you use VMWare/VMFS, it will issue discard when you delete a datastore.
-Not recommended to have 2 x replication. If you go with more nodes and wish to save space (at expense of speed) you can use EC rather than replication pools, we recommend EC 4x2 profile which requires 6 nodes, 1 TB client storage will require 1.5 TB raw.
-Yes in replication x 3 , if you have 90 TB of raw storage, the client will see 30 TB net
-Depends, if you delete ta file at the filesystem layer, the block storage will not know unless the filesystem sends a discard/trim commad. else the block layer will report the storage as used, but the filesystems knows the block are free and will over-write them. If you use VMWare/VMFS, it will issue discard when you delete a datastore.
-Not recommended to have 2 x replication. If you go with more nodes and wish to save space (at expense of speed) you can use EC rather than replication pools, we recommend EC 4x2 profile which requires 6 nodes, 1 TB client storage will require 1.5 TB raw.
R3LZX
50 Posts
May 17, 2020, 4:10 pmQuote from R3LZX on May 17, 2020, 4:10 pmthank you, the first was rather obvious but the second two questions really helped a lot thanks for these answers
thank you, the first was rather obvious but the second two questions really helped a lot thanks for these answers
Percentage usage and rules I should follow
R3LZX
50 Posts
Quote from R3LZX on May 16, 2020, 5:48 pmI have a three server cluster that is all RBD and is on the latest version, do you have a calculation that would allow me to understand what best practice I should follow for usage, for example since its three nodes and I have 9 OSD disks @ 90TB if the cluster is replicating X3 wouldn't that mean I can only really use 30TB ? of course the cluster will allow me to use more than this, and here is a list of what I am using now.
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
3 hdd 9.15329 1.00000 9.2 TiB 5.0 TiB 4.9 TiB 20 KiB 9.4 GiB 4.2 TiB 54.30 0.92 346 up
4 hdd 9.15329 1.00000 9.2 TiB 4.7 TiB 4.6 TiB 16 KiB 9.1 GiB 4.5 TiB 51.38 0.87 327 up
5 hdd 9.15329 1.00000 9.2 TiB 5.0 TiB 5.0 TiB 72 KiB 9.6 GiB 4.1 TiB 55.10 0.94 351 up
0 hdd 9.15329 1.00000 9.2 TiB 5.1 TiB 5.0 TiB 132 KiB 9.9 GiB 4.1 TiB 55.40 0.94 353 up
1 hdd 9.15329 1.00000 9.2 TiB 4.9 TiB 4.9 TiB 84 KiB 10 GiB 4.2 TiB 53.67 0.91 342 up
2 hdd 9.15329 1.00000 9.2 TiB 4.7 TiB 4.7 TiB 20 KiB 9.2 GiB 4.4 TiB 51.71 0.88 329 up
7 hdd 9.15329 1.00000 9.2 TiB 6.5 TiB 6.4 TiB 24 KiB 12 GiB 2.7 TiB 70.53 1.20 451 up
8 hdd 9.15329 1.00000 9.2 TiB 7.1 TiB 7.0 TiB 136 KiB 13 GiB 2.1 TiB 77.04 1.31 492 up
6 ssd 1.80469 1.00000 1.8 TiB 1.2 TiB 1.1 TiB 16 KiB 2.8 GiB 613 GiB 66.85 1.14 81 up
TOTAL 75 TiB 44 TiB 44 TiB 522 KiB 85 GiB 31 TiB 58.84
another question, if an ISCSI drive is used, and then the data is moved somewhere else does it take a long time to reclaim this space? or should the iscsi drive be completely removed to free up the needed space?
if I go to five servers, would a best practice be to going with only dual duplication instead of three? I am metaphorically thinking in terms of raid 5, 6, 10 etc at server level and just trying to plan for this financially.
I have a three server cluster that is all RBD and is on the latest version, do you have a calculation that would allow me to understand what best practice I should follow for usage, for example since its three nodes and I have 9 OSD disks @ 90TB if the cluster is replicating X3 wouldn't that mean I can only really use 30TB ? of course the cluster will allow me to use more than this, and here is a list of what I am using now.
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
3 hdd 9.15329 1.00000 9.2 TiB 5.0 TiB 4.9 TiB 20 KiB 9.4 GiB 4.2 TiB 54.30 0.92 346 up
4 hdd 9.15329 1.00000 9.2 TiB 4.7 TiB 4.6 TiB 16 KiB 9.1 GiB 4.5 TiB 51.38 0.87 327 up
5 hdd 9.15329 1.00000 9.2 TiB 5.0 TiB 5.0 TiB 72 KiB 9.6 GiB 4.1 TiB 55.10 0.94 351 up
0 hdd 9.15329 1.00000 9.2 TiB 5.1 TiB 5.0 TiB 132 KiB 9.9 GiB 4.1 TiB 55.40 0.94 353 up
1 hdd 9.15329 1.00000 9.2 TiB 4.9 TiB 4.9 TiB 84 KiB 10 GiB 4.2 TiB 53.67 0.91 342 up
2 hdd 9.15329 1.00000 9.2 TiB 4.7 TiB 4.7 TiB 20 KiB 9.2 GiB 4.4 TiB 51.71 0.88 329 up
7 hdd 9.15329 1.00000 9.2 TiB 6.5 TiB 6.4 TiB 24 KiB 12 GiB 2.7 TiB 70.53 1.20 451 up
8 hdd 9.15329 1.00000 9.2 TiB 7.1 TiB 7.0 TiB 136 KiB 13 GiB 2.1 TiB 77.04 1.31 492 up
6 ssd 1.80469 1.00000 1.8 TiB 1.2 TiB 1.1 TiB 16 KiB 2.8 GiB 613 GiB 66.85 1.14 81 up
TOTAL 75 TiB 44 TiB 44 TiB 522 KiB 85 GiB 31 TiB 58.84
another question, if an ISCSI drive is used, and then the data is moved somewhere else does it take a long time to reclaim this space? or should the iscsi drive be completely removed to free up the needed space?
if I go to five servers, would a best practice be to going with only dual duplication instead of three? I am metaphorically thinking in terms of raid 5, 6, 10 etc at server level and just trying to plan for this financially.
admin
2,930 Posts
Quote from admin on May 17, 2020, 3:35 pm-Yes in replication x 3 , if you have 90 TB of raw storage, the client will see 30 TB net
-Depends, if you delete ta file at the filesystem layer, the block storage will not know unless the filesystem sends a discard/trim commad. else the block layer will report the storage as used, but the filesystems knows the block are free and will over-write them. If you use VMWare/VMFS, it will issue discard when you delete a datastore.
-Not recommended to have 2 x replication. If you go with more nodes and wish to save space (at expense of speed) you can use EC rather than replication pools, we recommend EC 4x2 profile which requires 6 nodes, 1 TB client storage will require 1.5 TB raw.
-Yes in replication x 3 , if you have 90 TB of raw storage, the client will see 30 TB net
-Depends, if you delete ta file at the filesystem layer, the block storage will not know unless the filesystem sends a discard/trim commad. else the block layer will report the storage as used, but the filesystems knows the block are free and will over-write them. If you use VMWare/VMFS, it will issue discard when you delete a datastore.
-Not recommended to have 2 x replication. If you go with more nodes and wish to save space (at expense of speed) you can use EC rather than replication pools, we recommend EC 4x2 profile which requires 6 nodes, 1 TB client storage will require 1.5 TB raw.
R3LZX
50 Posts
Quote from R3LZX on May 17, 2020, 4:10 pmthank you, the first was rather obvious but the second two questions really helped a lot thanks for these answers
thank you, the first was rather obvious but the second two questions really helped a lot thanks for these answers