New in Petasan - Disklayout - Network
BenSommer
2 Posts
March 7, 2023, 10:12 amQuote from BenSommer on March 7, 2023, 10:12 amHi everyone
I´m new in Petasan and have two questions for my setup, after reading the manuals.
1)
I´ve 3 server (256GB, Intel Silver CPU) with:
1x nvme - system disk
5x 18TB data
2x 1.9TB ssd cache / journal ?
I red the manual, but what is the best disk setup for this drives?
2)
3 network cards
1x 1GB
2x 10GB
I have a config network with 26.x for the management module from my servers, my switches ...
I like to put the petasan management network in this range
I just like to use nfs for my proxmox ve, the normal network is 30.x for all pcs. If I put the nfs from petasan in this range, can I use it directly? Or is it better to make a separate network between the server and petasan? My fileserver will have 2x 10GB nic too. I can also use VLAN, if it is better.
Greetings from Berlin
Ben
Hi everyone
I´m new in Petasan and have two questions for my setup, after reading the manuals.
1)
I´ve 3 server (256GB, Intel Silver CPU) with:
1x nvme - system disk
5x 18TB data
2x 1.9TB ssd cache / journal ?
I red the manual, but what is the best disk setup for this drives?
2)
3 network cards
1x 1GB
2x 10GB
I have a config network with 26.x for the management module from my servers, my switches ...
I like to put the petasan management network in this range
I just like to use nfs for my proxmox ve, the normal network is 30.x for all pcs. If I put the nfs from petasan in this range, can I use it directly? Or is it better to make a separate network between the server and petasan? My fileserver will have 2x 10GB nic too. I can also use VLAN, if it is better.
Greetings from Berlin
Ben
admin
2,930 Posts
March 12, 2023, 9:46 pmQuote from admin on March 12, 2023, 9:46 pm
- You should always use ssd journal with hdd osds. Cache usage depends on workload and may not be suitable for all cases, you need to test it.
- Use 1 GB for management, Bond the 2x 10 GB for backend network as well as NFS, make both use different IP subnets, better to have vlans as it separates the broadcast domains, but not critical. You should chose your NFS subnet to be same subnet as your NFS clients.
- You should always use ssd journal with hdd osds. Cache usage depends on workload and may not be suitable for all cases, you need to test it.
- Use 1 GB for management, Bond the 2x 10 GB for backend network as well as NFS, make both use different IP subnets, better to have vlans as it separates the broadcast domains, but not critical. You should chose your NFS subnet to be same subnet as your NFS clients.
BenSommer
2 Posts
March 14, 2023, 5:15 amQuote from BenSommer on March 14, 2023, 5:15 amThx for your answer.
I did it how you discribed it. There was just a problem with my Realtek NIC. A comma in the name gave me an error during the installation from the cluster, error 500 ... but I used the fix from the forum and it worked for me.
/opt/petasan/scripts/detect-interfaces.sh
change line
VENDOR=$(echo "$INFO" | grep ID_VENDOR_FROM_DATABASE | cut -f2 -d = )
to
VENDOR=$(echo "$INFO" | grep ID_VENDOR_FROM_DATABASE | cut -f2 -d = | tr ',' ' ')
Thx for your answer.
I did it how you discribed it. There was just a problem with my Realtek NIC. A comma in the name gave me an error during the installation from the cluster, error 500 ... but I used the fix from the forum and it worked for me.
/opt/petasan/scripts/detect-interfaces.sh
change line
VENDOR=$(echo "$INFO" | grep ID_VENDOR_FROM_DATABASE | cut -f2 -d = )
to
VENDOR=$(echo "$INFO" | grep ID_VENDOR_FROM_DATABASE | cut -f2 -d = | tr ',' ' ')
Last edited on March 14, 2023, 5:15 am by BenSommer · #3
admin
2,930 Posts
March 14, 2023, 8:12 amQuote from admin on March 14, 2023, 8:12 amthanks for the feedback, this fix will be in next release,
thanks for the feedback, this fix will be in next release,
X1M
5 Posts
April 23, 2023, 10:50 amQuote from X1M on April 23, 2023, 10:50 amI have been running CPEH since 2017 and used Linux of the last 20 years. I have had the following experience that could help you with PetaSAN and CEPH. You can disagree, however sticking with the following have never given me problems.
- With Linux in general always stick to Intel enterprise NIC card, ex. for 10GB Intel X520 or X710.
- Never use Seagate Hard disk, had a lot of problems with the Enterprise Wolf series and CEPH and had to change 48 Seagate out and replace them with HGST (Now Western Digital) after we figured out that it was the Seagate that gave problems. In general, I have had a lot of bad experience with Seagate disks.
I have been running CPEH since 2017 and used Linux of the last 20 years. I have had the following experience that could help you with PetaSAN and CEPH. You can disagree, however sticking with the following have never given me problems.
- With Linux in general always stick to Intel enterprise NIC card, ex. for 10GB Intel X520 or X710.
- Never use Seagate Hard disk, had a lot of problems with the Enterprise Wolf series and CEPH and had to change 48 Seagate out and replace them with HGST (Now Western Digital) after we figured out that it was the Seagate that gave problems. In general, I have had a lot of bad experience with Seagate disks.
New in Petasan - Disklayout - Network
BenSommer
2 Posts
Quote from BenSommer on March 7, 2023, 10:12 amHi everyone
I´m new in Petasan and have two questions for my setup, after reading the manuals.
1)
I´ve 3 server (256GB, Intel Silver CPU) with:
1x nvme - system disk
5x 18TB data
2x 1.9TB ssd cache / journal ?
I red the manual, but what is the best disk setup for this drives?
2)
3 network cards
1x 1GB
2x 10GB
I have a config network with 26.x for the management module from my servers, my switches ...
I like to put the petasan management network in this range
I just like to use nfs for my proxmox ve, the normal network is 30.x for all pcs. If I put the nfs from petasan in this range, can I use it directly? Or is it better to make a separate network between the server and petasan? My fileserver will have 2x 10GB nic too. I can also use VLAN, if it is better.
Greetings from Berlin
Ben
Hi everyone
I´m new in Petasan and have two questions for my setup, after reading the manuals.
1)
I´ve 3 server (256GB, Intel Silver CPU) with:
1x nvme - system disk
5x 18TB data
2x 1.9TB ssd cache / journal ?
I red the manual, but what is the best disk setup for this drives?
2)
3 network cards
1x 1GB
2x 10GB
I have a config network with 26.x for the management module from my servers, my switches ...
I like to put the petasan management network in this range
I just like to use nfs for my proxmox ve, the normal network is 30.x for all pcs. If I put the nfs from petasan in this range, can I use it directly? Or is it better to make a separate network between the server and petasan? My fileserver will have 2x 10GB nic too. I can also use VLAN, if it is better.
Greetings from Berlin
Ben
admin
2,930 Posts
Quote from admin on March 12, 2023, 9:46 pm
- You should always use ssd journal with hdd osds. Cache usage depends on workload and may not be suitable for all cases, you need to test it.
- Use 1 GB for management, Bond the 2x 10 GB for backend network as well as NFS, make both use different IP subnets, better to have vlans as it separates the broadcast domains, but not critical. You should chose your NFS subnet to be same subnet as your NFS clients.
- You should always use ssd journal with hdd osds. Cache usage depends on workload and may not be suitable for all cases, you need to test it.
- Use 1 GB for management, Bond the 2x 10 GB for backend network as well as NFS, make both use different IP subnets, better to have vlans as it separates the broadcast domains, but not critical. You should chose your NFS subnet to be same subnet as your NFS clients.
BenSommer
2 Posts
Quote from BenSommer on March 14, 2023, 5:15 amThx for your answer.
I did it how you discribed it. There was just a problem with my Realtek NIC. A comma in the name gave me an error during the installation from the cluster, error 500 ... but I used the fix from the forum and it worked for me.
/opt/petasan/scripts/detect-interfaces.sh
change line
VENDOR=$(echo "$INFO" | grep ID_VENDOR_FROM_DATABASE | cut -f2 -d = )
to
VENDOR=$(echo "$INFO" | grep ID_VENDOR_FROM_DATABASE | cut -f2 -d = | tr ',' ' ')
Thx for your answer.
I did it how you discribed it. There was just a problem with my Realtek NIC. A comma in the name gave me an error during the installation from the cluster, error 500 ... but I used the fix from the forum and it worked for me.
/opt/petasan/scripts/detect-interfaces.sh
change line
VENDOR=$(echo "$INFO" | grep ID_VENDOR_FROM_DATABASE | cut -f2 -d = )
to
VENDOR=$(echo "$INFO" | grep ID_VENDOR_FROM_DATABASE | cut -f2 -d = | tr ',' ' ')
admin
2,930 Posts
Quote from admin on March 14, 2023, 8:12 amthanks for the feedback, this fix will be in next release,
thanks for the feedback, this fix will be in next release,
X1M
5 Posts
Quote from X1M on April 23, 2023, 10:50 amI have been running CPEH since 2017 and used Linux of the last 20 years. I have had the following experience that could help you with PetaSAN and CEPH. You can disagree, however sticking with the following have never given me problems.
- With Linux in general always stick to Intel enterprise NIC card, ex. for 10GB Intel X520 or X710.
- Never use Seagate Hard disk, had a lot of problems with the Enterprise Wolf series and CEPH and had to change 48 Seagate out and replace them with HGST (Now Western Digital) after we figured out that it was the Seagate that gave problems. In general, I have had a lot of bad experience with Seagate disks.
I have been running CPEH since 2017 and used Linux of the last 20 years. I have had the following experience that could help you with PetaSAN and CEPH. You can disagree, however sticking with the following have never given me problems.
- With Linux in general always stick to Intel enterprise NIC card, ex. for 10GB Intel X520 or X710.
- Never use Seagate Hard disk, had a lot of problems with the Enterprise Wolf series and CEPH and had to change 48 Seagate out and replace them with HGST (Now Western Digital) after we figured out that it was the Seagate that gave problems. In general, I have had a lot of bad experience with Seagate disks.