Petasan - primary storage for virtual environment
vantolik
5 Posts
January 20, 2020, 12:41 pmQuote from vantolik on January 20, 2020, 12:41 pmHi,
we plan to use Petasan as primary storage system for production environment(150 VMs and 40TB of storage with future growing). Due to HA we want to place the part of Petasan in second server room.
Is it suitable to start with 4 physical servers + 1 VM(only mgmt&monitor services, due to maintain quorum between two server room, disks under Proxmox HA outside of Petasan.)? From the point of performance would it be enough when I meet the criteria for servers under PetaSan Recommended Hardware guide?
Could you please to advice me what is the best use for this 4 servers? 4 replicas are probably too much, so 3 high performance servers(replicas and ISCSI Target service) + 1 "common" server(just mgmt&monitor to save money for disks). Or?
Thank you very much.
Vladislav
Hi,
we plan to use Petasan as primary storage system for production environment(150 VMs and 40TB of storage with future growing). Due to HA we want to place the part of Petasan in second server room.
Is it suitable to start with 4 physical servers + 1 VM(only mgmt&monitor services, due to maintain quorum between two server room, disks under Proxmox HA outside of Petasan.)? From the point of performance would it be enough when I meet the criteria for servers under PetaSan Recommended Hardware guide?
Could you please to advice me what is the best use for this 4 servers? 4 replicas are probably too much, so 3 high performance servers(replicas and ISCSI Target service) + 1 "common" server(just mgmt&monitor to save money for disks). Or?
Thank you very much.
Vladislav
admin
2,930 Posts
January 20, 2020, 2:27 pmQuote from admin on January 20, 2020, 2:27 pmSplitting the cluster into 2 is tricky to maintain quorum, there is no issue if you split in 3.
in case of quorum vm, make sure to store it outside PetaSAN and insure it can switch to the other room fast enough, i do not know how fast Proxmox can do ha switching, you can also look at other virtualization solutions that can do fast ha, you should test this solution, maybe all vms in a lab, to make sure it works well.
the hardware guide is what we recommend, running in vms is not something we support.
in your setup you would need a size 4 replica and min_size = 2, you would need a custom crush rule, i believe we have a template rule in the ui for this 2 room setup.
Splitting the cluster into 2 is tricky to maintain quorum, there is no issue if you split in 3.
in case of quorum vm, make sure to store it outside PetaSAN and insure it can switch to the other room fast enough, i do not know how fast Proxmox can do ha switching, you can also look at other virtualization solutions that can do fast ha, you should test this solution, maybe all vms in a lab, to make sure it works well.
the hardware guide is what we recommend, running in vms is not something we support.
in your setup you would need a size 4 replica and min_size = 2, you would need a custom crush rule, i believe we have a template rule in the ui for this 2 room setup.
vantolik
5 Posts
January 20, 2020, 8:38 pmQuote from vantolik on January 20, 2020, 8:38 pmHi,
could you please recommend me any setup how to handle this? Let's say you have a budget.
I would like to avoid non standard configuration of PetaSAN, like not supporting configuration or custom crush rule .
I think, that management would not like to build third server room :).
Or solution would be to place all 5 PetaSAN nodes into one server room and pray.
Thank you very much.
Vladislav
Hi,
could you please recommend me any setup how to handle this? Let's say you have a budget.
I would like to avoid non standard configuration of PetaSAN, like not supporting configuration or custom crush rule .
I think, that management would not like to build third server room :).
Or solution would be to place all 5 PetaSAN nodes into one server room and pray.
Thank you very much.
Vladislav
vantolik
5 Posts
January 21, 2020, 2:00 pmQuote from vantolik on January 21, 2020, 2:00 pmHi,
I don't want to overcomplicated the architecture. Let's say I would cover better protection from power outage on majority nodes to protect quorum in server room1.
If PetaSAN stops to work due to lose quorum and after time the quorum is restored, what is the procedure to have functional PetaSAN. Its enough just to start node which would restore quorum?
Is it ok for production environment with this load(150 VMs and 40TB of storage with future growing)to use 5 physical servers? What would be the best layout?
Could be it like this?
3 for local storage service+iscsi target service and one of them management&monitoring - all of them high performance servers, according to Recommended Hardware Guide
2 for management&monitoring - cheap servers, In hardware guide is just RAM recommendation, Could you please write me any basic HW recommendation, disk, cpu, I suppose 10G eth.
In case of use mix HDD and SSD is stil valid ratio 1:4 and SSD journal disk around 60GB. Or it is related to size of HDD? I would like to use HP Enterprise 2,4TB 10K SAS. So, as journal disk would be enough Enterprise 128GB Write intesive SSD?
Thank you.
Vladislav
Hi,
I don't want to overcomplicated the architecture. Let's say I would cover better protection from power outage on majority nodes to protect quorum in server room1.
If PetaSAN stops to work due to lose quorum and after time the quorum is restored, what is the procedure to have functional PetaSAN. Its enough just to start node which would restore quorum?
Is it ok for production environment with this load(150 VMs and 40TB of storage with future growing)to use 5 physical servers? What would be the best layout?
Could be it like this?
3 for local storage service+iscsi target service and one of them management&monitoring - all of them high performance servers, according to Recommended Hardware Guide
2 for management&monitoring - cheap servers, In hardware guide is just RAM recommendation, Could you please write me any basic HW recommendation, disk, cpu, I suppose 10G eth.
In case of use mix HDD and SSD is stil valid ratio 1:4 and SSD journal disk around 60GB. Or it is related to size of HDD? I would like to use HP Enterprise 2,4TB 10K SAS. So, as journal disk would be enough Enterprise 128GB Write intesive SSD?
Thank you.
Vladislav
admin
2,930 Posts
January 21, 2020, 3:30 pmQuote from admin on January 21, 2020, 3:30 pmTo get 2 rooms/centers to work well, it does require some complexity for quorum issue being split in 2. there is no issue with other numbers. The only way to support '2' is to have a quorum node HA available across both locations. This HA need to be outside PetaSAN, Could be done manually via a drbd setup (similar to using drbd to provide HA to databases) or via vm and having a hypervisor provide HA, either cases needs to be tested and is outside PetaSAN (we can do so via our consulting services if you wish).
as indicated, custom crush rules is required, you cannot use default crush rule.
Yes of course if you restart failed nodes, the cluster will be up again. this is the simplest solution, if you can afford some downtime.
Yes ssd:hdd ratio for journals is 1:4. so you need at least 256G ssds
3 storage nodes can give from about 10K iops for HDDs uo to 80K iops for high-end/all-flash random writes, so it should be ok for your vms unless you have some special workload. I all cases you should benchmark your cluster (via the ui) before going production to know your performance per node.
To get 2 rooms/centers to work well, it does require some complexity for quorum issue being split in 2. there is no issue with other numbers. The only way to support '2' is to have a quorum node HA available across both locations. This HA need to be outside PetaSAN, Could be done manually via a drbd setup (similar to using drbd to provide HA to databases) or via vm and having a hypervisor provide HA, either cases needs to be tested and is outside PetaSAN (we can do so via our consulting services if you wish).
as indicated, custom crush rules is required, you cannot use default crush rule.
Yes of course if you restart failed nodes, the cluster will be up again. this is the simplest solution, if you can afford some downtime.
Yes ssd:hdd ratio for journals is 1:4. so you need at least 256G ssds
3 storage nodes can give from about 10K iops for HDDs uo to 80K iops for high-end/all-flash random writes, so it should be ok for your vms unless you have some special workload. I all cases you should benchmark your cluster (via the ui) before going production to know your performance per node.
vantolik
5 Posts
January 21, 2020, 4:11 pmQuote from vantolik on January 21, 2020, 4:11 pmHi,
Thank you very much. I would like to summarize my setup.
5 nodes in 2 server room(take it as one, I just affraid of power outage, so we strengthen the UPS, so quorum will not be an issue anymore).
3 storage nodes - high performance
2 management&monitoring nodes - low performance, Could you please advice me some kind of HW? For example server with 2CPU/12Core, 64GB RAM, 4x256 SSD, 4x10GE Eth would be enough?
I would like to use Proxmox. What would you recommend to use - iSCSI or native Ceph RBD?
Thank you.
Vladislav
Hi,
Thank you very much. I would like to summarize my setup.
5 nodes in 2 server room(take it as one, I just affraid of power outage, so we strengthen the UPS, so quorum will not be an issue anymore).
3 storage nodes - high performance
2 management&monitoring nodes - low performance, Could you please advice me some kind of HW? For example server with 2CPU/12Core, 64GB RAM, 4x256 SSD, 4x10GE Eth would be enough?
I would like to use Proxmox. What would you recommend to use - iSCSI or native Ceph RBD?
Thank you.
Vladislav
admin
2,930 Posts
January 21, 2020, 8:32 pmQuote from admin on January 21, 2020, 8:32 pmYes the management/monitoring nodes hardware is enough.
Proxmox does use rbd directly, so maybe use this and avoid an extra io hop.
Yes the management/monitoring nodes hardware is enough.
Proxmox does use rbd directly, so maybe use this and avoid an extra io hop.
Petasan - primary storage for virtual environment
vantolik
5 Posts
Quote from vantolik on January 20, 2020, 12:41 pmHi,
we plan to use Petasan as primary storage system for production environment(150 VMs and 40TB of storage with future growing). Due to HA we want to place the part of Petasan in second server room.
Is it suitable to start with 4 physical servers + 1 VM(only mgmt&monitor services, due to maintain quorum between two server room, disks under Proxmox HA outside of Petasan.)? From the point of performance would it be enough when I meet the criteria for servers under PetaSan Recommended Hardware guide?
Could you please to advice me what is the best use for this 4 servers? 4 replicas are probably too much, so 3 high performance servers(replicas and ISCSI Target service) + 1 "common" server(just mgmt&monitor to save money for disks). Or?
Thank you very much.
Vladislav
Hi,
we plan to use Petasan as primary storage system for production environment(150 VMs and 40TB of storage with future growing). Due to HA we want to place the part of Petasan in second server room.
Is it suitable to start with 4 physical servers + 1 VM(only mgmt&monitor services, due to maintain quorum between two server room, disks under Proxmox HA outside of Petasan.)? From the point of performance would it be enough when I meet the criteria for servers under PetaSan Recommended Hardware guide?
Could you please to advice me what is the best use for this 4 servers? 4 replicas are probably too much, so 3 high performance servers(replicas and ISCSI Target service) + 1 "common" server(just mgmt&monitor to save money for disks). Or?
Thank you very much.
Vladislav
admin
2,930 Posts
Quote from admin on January 20, 2020, 2:27 pmSplitting the cluster into 2 is tricky to maintain quorum, there is no issue if you split in 3.
in case of quorum vm, make sure to store it outside PetaSAN and insure it can switch to the other room fast enough, i do not know how fast Proxmox can do ha switching, you can also look at other virtualization solutions that can do fast ha, you should test this solution, maybe all vms in a lab, to make sure it works well.
the hardware guide is what we recommend, running in vms is not something we support.
in your setup you would need a size 4 replica and min_size = 2, you would need a custom crush rule, i believe we have a template rule in the ui for this 2 room setup.
Splitting the cluster into 2 is tricky to maintain quorum, there is no issue if you split in 3.
in case of quorum vm, make sure to store it outside PetaSAN and insure it can switch to the other room fast enough, i do not know how fast Proxmox can do ha switching, you can also look at other virtualization solutions that can do fast ha, you should test this solution, maybe all vms in a lab, to make sure it works well.
the hardware guide is what we recommend, running in vms is not something we support.
in your setup you would need a size 4 replica and min_size = 2, you would need a custom crush rule, i believe we have a template rule in the ui for this 2 room setup.
vantolik
5 Posts
Quote from vantolik on January 20, 2020, 8:38 pmHi,
could you please recommend me any setup how to handle this? Let's say you have a budget.
I would like to avoid non standard configuration of PetaSAN, like not supporting configuration or custom crush rule .
I think, that management would not like to build third server room :).
Or solution would be to place all 5 PetaSAN nodes into one server room and pray.
Thank you very much.
Vladislav
Hi,
could you please recommend me any setup how to handle this? Let's say you have a budget.
I would like to avoid non standard configuration of PetaSAN, like not supporting configuration or custom crush rule .
I think, that management would not like to build third server room :).
Or solution would be to place all 5 PetaSAN nodes into one server room and pray.
Thank you very much.
Vladislav
vantolik
5 Posts
Quote from vantolik on January 21, 2020, 2:00 pmHi,
I don't want to overcomplicated the architecture. Let's say I would cover better protection from power outage on majority nodes to protect quorum in server room1.
If PetaSAN stops to work due to lose quorum and after time the quorum is restored, what is the procedure to have functional PetaSAN. Its enough just to start node which would restore quorum?
Is it ok for production environment with this load(150 VMs and 40TB of storage with future growing)to use 5 physical servers? What would be the best layout?
Could be it like this?
3 for local storage service+iscsi target service and one of them management&monitoring - all of them high performance servers, according to Recommended Hardware Guide
2 for management&monitoring - cheap servers, In hardware guide is just RAM recommendation, Could you please write me any basic HW recommendation, disk, cpu, I suppose 10G eth.
In case of use mix HDD and SSD is stil valid ratio 1:4 and SSD journal disk around 60GB. Or it is related to size of HDD? I would like to use HP Enterprise 2,4TB 10K SAS. So, as journal disk would be enough Enterprise 128GB Write intesive SSD?
Thank you.
Vladislav
Hi,
I don't want to overcomplicated the architecture. Let's say I would cover better protection from power outage on majority nodes to protect quorum in server room1.
If PetaSAN stops to work due to lose quorum and after time the quorum is restored, what is the procedure to have functional PetaSAN. Its enough just to start node which would restore quorum?
Is it ok for production environment with this load(150 VMs and 40TB of storage with future growing)to use 5 physical servers? What would be the best layout?
Could be it like this?
3 for local storage service+iscsi target service and one of them management&monitoring - all of them high performance servers, according to Recommended Hardware Guide
2 for management&monitoring - cheap servers, In hardware guide is just RAM recommendation, Could you please write me any basic HW recommendation, disk, cpu, I suppose 10G eth.
In case of use mix HDD and SSD is stil valid ratio 1:4 and SSD journal disk around 60GB. Or it is related to size of HDD? I would like to use HP Enterprise 2,4TB 10K SAS. So, as journal disk would be enough Enterprise 128GB Write intesive SSD?
Thank you.
Vladislav
admin
2,930 Posts
Quote from admin on January 21, 2020, 3:30 pmTo get 2 rooms/centers to work well, it does require some complexity for quorum issue being split in 2. there is no issue with other numbers. The only way to support '2' is to have a quorum node HA available across both locations. This HA need to be outside PetaSAN, Could be done manually via a drbd setup (similar to using drbd to provide HA to databases) or via vm and having a hypervisor provide HA, either cases needs to be tested and is outside PetaSAN (we can do so via our consulting services if you wish).
as indicated, custom crush rules is required, you cannot use default crush rule.
Yes of course if you restart failed nodes, the cluster will be up again. this is the simplest solution, if you can afford some downtime.
Yes ssd:hdd ratio for journals is 1:4. so you need at least 256G ssds
3 storage nodes can give from about 10K iops for HDDs uo to 80K iops for high-end/all-flash random writes, so it should be ok for your vms unless you have some special workload. I all cases you should benchmark your cluster (via the ui) before going production to know your performance per node.
To get 2 rooms/centers to work well, it does require some complexity for quorum issue being split in 2. there is no issue with other numbers. The only way to support '2' is to have a quorum node HA available across both locations. This HA need to be outside PetaSAN, Could be done manually via a drbd setup (similar to using drbd to provide HA to databases) or via vm and having a hypervisor provide HA, either cases needs to be tested and is outside PetaSAN (we can do so via our consulting services if you wish).
as indicated, custom crush rules is required, you cannot use default crush rule.
Yes of course if you restart failed nodes, the cluster will be up again. this is the simplest solution, if you can afford some downtime.
Yes ssd:hdd ratio for journals is 1:4. so you need at least 256G ssds
3 storage nodes can give from about 10K iops for HDDs uo to 80K iops for high-end/all-flash random writes, so it should be ok for your vms unless you have some special workload. I all cases you should benchmark your cluster (via the ui) before going production to know your performance per node.
vantolik
5 Posts
Quote from vantolik on January 21, 2020, 4:11 pmHi,
Thank you very much. I would like to summarize my setup.
5 nodes in 2 server room(take it as one, I just affraid of power outage, so we strengthen the UPS, so quorum will not be an issue anymore).
3 storage nodes - high performance
2 management&monitoring nodes - low performance, Could you please advice me some kind of HW? For example server with 2CPU/12Core, 64GB RAM, 4x256 SSD, 4x10GE Eth would be enough?
I would like to use Proxmox. What would you recommend to use - iSCSI or native Ceph RBD?
Thank you.
Vladislav
Hi,
Thank you very much. I would like to summarize my setup.
5 nodes in 2 server room(take it as one, I just affraid of power outage, so we strengthen the UPS, so quorum will not be an issue anymore).
3 storage nodes - high performance
2 management&monitoring nodes - low performance, Could you please advice me some kind of HW? For example server with 2CPU/12Core, 64GB RAM, 4x256 SSD, 4x10GE Eth would be enough?
I would like to use Proxmox. What would you recommend to use - iSCSI or native Ceph RBD?
Thank you.
Vladislav
admin
2,930 Posts
Quote from admin on January 21, 2020, 8:32 pmYes the management/monitoring nodes hardware is enough.
Proxmox does use rbd directly, so maybe use this and avoid an extra io hop.
Yes the management/monitoring nodes hardware is enough.
Proxmox does use rbd directly, so maybe use this and avoid an extra io hop.