10G vs 25G
pablov
5 Posts
January 9, 2023, 11:36 pmQuote from pablov on January 9, 2023, 11:36 pmHi,
I'm in the process of building my first Ceph cluster. It will be a 5 node one of HDD drives only (plus 1 NVME drive per node), with room to grow sooner than later.
I've budgeted a dual port 25G NIC for each node and a 32-port 100G switch. However, I'm having second thoughts about this. If I go smaller on network bandwidth I might be able to go bigger on node number.
The cluster will serve a small VFX studio with 32 seats but only a few of them need really high bandwidth. Latency should be good to fetch video files in a sequence at 60 frames/files per second (size of the files typically go from 2 to 10 MB).
Thanks for any help. Don't hesitate to ask any question and please forgive my ignorance.
Hi,
I'm in the process of building my first Ceph cluster. It will be a 5 node one of HDD drives only (plus 1 NVME drive per node), with room to grow sooner than later.
I've budgeted a dual port 25G NIC for each node and a 32-port 100G switch. However, I'm having second thoughts about this. If I go smaller on network bandwidth I might be able to go bigger on node number.
The cluster will serve a small VFX studio with 32 seats but only a few of them need really high bandwidth. Latency should be good to fetch video files in a sequence at 60 frames/files per second (size of the files typically go from 2 to 10 MB).
Thanks for any help. Don't hesitate to ask any question and please forgive my ignorance.
rhamon
30 Posts
January 10, 2023, 2:51 pmQuote from rhamon on January 10, 2023, 2:51 pmYou'll need lots of HDD (24+) in a single node to saturate that 25gbit and Ceph is scale out so more node means more total throughput, I would not spend the extra money on that 100Gbit switch and instead favor more nodes and especially more disks if the node have the bays and cpu to handle it. Even 10gbit is good enough if you can scale up the nodes and disks.
Don't cheap out on the number of disks, you'll regret it... and that nvme better be enterprise grade if you intend to use it as journal for all the HDD...
Maybe consider all-flash if you need performance more than capacity.
You'll need lots of HDD (24+) in a single node to saturate that 25gbit and Ceph is scale out so more node means more total throughput, I would not spend the extra money on that 100Gbit switch and instead favor more nodes and especially more disks if the node have the bays and cpu to handle it. Even 10gbit is good enough if you can scale up the nodes and disks.
Don't cheap out on the number of disks, you'll regret it... and that nvme better be enterprise grade if you intend to use it as journal for all the HDD...
Maybe consider all-flash if you need performance more than capacity.
pablov
5 Posts
January 10, 2023, 3:20 pmQuote from pablov on January 10, 2023, 3:20 pmThanks!
The 100G swtich is not just for this ceph cluster. It's meant to be the backbone of our local network and surprisingly, once you breakout each port, it becomes cheaper than other solutions. Besides, it's a future investment (for the next 5 years at least we won't need anything bigger).
I might get 10G NICs for the nodes. Thanks for the advice.
All-flash nodes will come later to work as cache or high speed tier. Can't go for them at the moment.
Thanks!
The 100G swtich is not just for this ceph cluster. It's meant to be the backbone of our local network and surprisingly, once you breakout each port, it becomes cheaper than other solutions. Besides, it's a future investment (for the next 5 years at least we won't need anything bigger).
I might get 10G NICs for the nodes. Thanks for the advice.
All-flash nodes will come later to work as cache or high speed tier. Can't go for them at the moment.
10G vs 25G
pablov
5 Posts
Quote from pablov on January 9, 2023, 11:36 pmHi,
I'm in the process of building my first Ceph cluster. It will be a 5 node one of HDD drives only (plus 1 NVME drive per node), with room to grow sooner than later.
I've budgeted a dual port 25G NIC for each node and a 32-port 100G switch. However, I'm having second thoughts about this. If I go smaller on network bandwidth I might be able to go bigger on node number.
The cluster will serve a small VFX studio with 32 seats but only a few of them need really high bandwidth. Latency should be good to fetch video files in a sequence at 60 frames/files per second (size of the files typically go from 2 to 10 MB).
Thanks for any help. Don't hesitate to ask any question and please forgive my ignorance.
Hi,
I'm in the process of building my first Ceph cluster. It will be a 5 node one of HDD drives only (plus 1 NVME drive per node), with room to grow sooner than later.
I've budgeted a dual port 25G NIC for each node and a 32-port 100G switch. However, I'm having second thoughts about this. If I go smaller on network bandwidth I might be able to go bigger on node number.
The cluster will serve a small VFX studio with 32 seats but only a few of them need really high bandwidth. Latency should be good to fetch video files in a sequence at 60 frames/files per second (size of the files typically go from 2 to 10 MB).
Thanks for any help. Don't hesitate to ask any question and please forgive my ignorance.
rhamon
30 Posts
Quote from rhamon on January 10, 2023, 2:51 pmYou'll need lots of HDD (24+) in a single node to saturate that 25gbit and Ceph is scale out so more node means more total throughput, I would not spend the extra money on that 100Gbit switch and instead favor more nodes and especially more disks if the node have the bays and cpu to handle it. Even 10gbit is good enough if you can scale up the nodes and disks.
Don't cheap out on the number of disks, you'll regret it... and that nvme better be enterprise grade if you intend to use it as journal for all the HDD...
Maybe consider all-flash if you need performance more than capacity.
You'll need lots of HDD (24+) in a single node to saturate that 25gbit and Ceph is scale out so more node means more total throughput, I would not spend the extra money on that 100Gbit switch and instead favor more nodes and especially more disks if the node have the bays and cpu to handle it. Even 10gbit is good enough if you can scale up the nodes and disks.
Don't cheap out on the number of disks, you'll regret it... and that nvme better be enterprise grade if you intend to use it as journal for all the HDD...
Maybe consider all-flash if you need performance more than capacity.
pablov
5 Posts
Quote from pablov on January 10, 2023, 3:20 pmThanks!
The 100G swtich is not just for this ceph cluster. It's meant to be the backbone of our local network and surprisingly, once you breakout each port, it becomes cheaper than other solutions. Besides, it's a future investment (for the next 5 years at least we won't need anything bigger).
I might get 10G NICs for the nodes. Thanks for the advice.
All-flash nodes will come later to work as cache or high speed tier. Can't go for them at the moment.
Thanks!
The 100G swtich is not just for this ceph cluster. It's meant to be the backbone of our local network and surprisingly, once you breakout each port, it becomes cheaper than other solutions. Besides, it's a future investment (for the next 5 years at least we won't need anything bigger).
I might get 10G NICs for the nodes. Thanks for the advice.
All-flash nodes will come later to work as cache or high speed tier. Can't go for them at the moment.