Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

S3 Configuration 504 Gateway Timeout


eazyadm
25 Posts
July 24, 2023, 12:06 pm

Hi,

I tried to configure S3 on the latest Petasan Release 3.2.0.

I got the Warning "Zone created successfully but failed to commit period." after creating clicking on "save" in configuration/S3-configuration/Zonegroups.

Now, I get the an 504 Gateway Timeout by Accessing "https://petasanmgmtip/configuration/s3_configuration/zones

How to debug the issue and get it working?

12 Nodes
- 3 management
- 3 iscsi and storage
- 3 s3 and storage
- 3 pure storage

root@la01-cnode-05:~# tail -n100 -f /opt/petasan/log/PetaSAN.log
22/07/2023 06:25:02 INFO GlusterFS mount attempt
23/07/2023 06:25:05 INFO GlusterFS mount attempt
24/07/2023 06:25:07 INFO GlusterFS mount attempt
24/07/2023 13:35:06 INFO Update roles.
24/07/2023 13:35:06 INFO Update node s3 role to true
24/07/2023 13:35:06 INFO Starting PetaSAN-s3 service
24/07/2023 13:35:06 INFO Update node info in file
24/07/2023 13:35:06 INFO sync_replication_node called on non-backup node
24/07/2023 13:35:06 INFO Starting S3Server Service.
24/07/2023 13:35:07 WARNING failed to list realms: (2) No such file or directory

24/07/2023 13:35:07 INFO RGW | get_rgw_environment no realms found.
24/07/2023 13:35:07 WARNING failed to list zonegroups: (2) No such file or directory

24/07/2023 13:35:07 INFO RGW | get_rgw_environment no zonegroups found.
24/07/2023 13:35:07 WARNING failed to list zones: (2) No such file or directory

24/07/2023 13:35:07 INFO S3 init_action : cluster has no local zone
24/07/2023 13:35:08 INFO Stopping S3 Service
24/07/2023 13:35:08 INFO S3Server : clean all local resources
24/07/2023 13:35:39 INFO Starting S3Server Service.
24/07/2023 13:35:39 WARNING failed to list realms: (2) No such file or directory

24/07/2023 13:35:39 INFO RGW | get_rgw_environment no realms found.
24/07/2023 13:35:39 WARNING failed to list zonegroups: (2) No such file or directory

24/07/2023 13:35:39 INFO RGW | get_rgw_environment no zonegroups found.
24/07/2023 13:35:39 WARNING failed to list zones: (2) No such file or directory

24/07/2023 13:35:39 INFO S3 init_action : cluster has no local zone
24/07/2023 13:35:40 INFO Stopping S3 Service
24/07/2023 13:35:40 INFO S3Server : clean all local resources
24/07/2023 13:36:11 INFO Starting S3Server Service.
24/07/2023 13:36:12 WARNING failed to list realms: (2) No such file or directory

24/07/2023 13:36:12 INFO RGW | get_rgw_environment no realms found.
24/07/2023 13:36:13 WARNING failed to list zonegroups: (2) No such file or directory

24/07/2023 13:36:13 INFO RGW | get_rgw_environment no zonegroups found.
24/07/2023 13:36:13 WARNING failed to list zones: (2) No such file or directory

24/07/2023 13:36:13 INFO S3 init_action : cluster has no local zone
24/07/2023 13:36:14 INFO Stopping S3 Service
24/07/2023 13:36:14 INFO S3Server : clean all local resources
24/07/2023 13:36:45 INFO Starting S3Server Service.
24/07/2023 13:36:48 INFO S3 init_action : cluster has no local zone
24/07/2023 13:36:49 INFO Stopping S3 Service
24/07/2023 13:36:49 INFO S3Server : clean all local resources
24/07/2023 13:37:20 INFO Starting S3Server Service.
24/07/2023 13:37:27 INFO S3 init_action : cluster has no local zone
24/07/2023 13:37:27 INFO Stopping S3 Service
24/07/2023 13:37:27 INFO S3Server : clean all local resources
24/07/2023 13:37:58 INFO Starting S3Server Service.
24/07/2023 13:38:09 INFO S3 init_action : cluster has no local zone
24/07/2023 13:38:10 INFO Stopping S3 Service
24/07/2023 13:38:10 INFO S3Server : clean all local resources
24/07/2023 13:38:41 INFO Starting S3Server Service.
24/07/2023 13:40:00 INFO create_rgw() started
24/07/2023 13:40:01 INFO create_rgw() ended successfully

24/07/2023 13:40:01 INFO LockBase : Dropping old sessions
24/07/2023 13:40:11 INFO LockBase : Successfully dropped old sessions
24/07/2023 13:40:11 INFO Clean all old local resources.
24/07/2023 13:40:13 INFO S3Server : sync Consul settings
24/07/2023 13:40:13 INFO S3Server : sync Consul settings -> done
24/07/2023 13:40:14 INFO LockBase : Try to acquire the resource = S3-192-168-219-4.
24/07/2023 13:40:14 INFO LockBase : Succeeded on acquiring the resource = S3-192-168-219-4
24/07/2023 13:40:15 INFO LockBase : Try to acquire the resource = S3-192-168-219-5.
24/07/2023 13:40:18 INFO LockBase : Succeeded on acquiring the resource = S3-192-168-219-5
24/07/2023 13:40:21 INFO S3Server : sync Consul settings
24/07/2023 13:40:21 INFO S3Server : sync Consul settings -> done
24/07/2023 13:41:05 INFO LockBase : Attempt 1 for requesting : "PetaSAN/S3Server" - an update is in progress
24/07/2023 13:41:16 INFO S3Server : sync Consul settings
24/07/2023 13:41:16 INFO S3Server : Consul settings changed,will clean all resources and restart S3Server
24/07/2023 13:41:16 INFO S3Server : clean local resource : S3-192-168-219-4
24/07/2023 13:41:16 INFO S3Server : clean local resource : 1) Delete ip address of resource : S3-192-168-219-4
24/07/2023 13:41:16 INFO S3Server : clean local resource : S3-192-168-219-5
24/07/2023 13:41:16 INFO S3Server : clean local resource : 1) Delete ip address of resource : S3-192-168-219-5
24/07/2023 13:41:16 INFO LockBase : unlock_all_consul_resources : Unlock key of resource : S3-192-168-219-4
24/07/2023 13:41:16 INFO LockBase : unlock_all_consul_resources : Unlock key of resource : S3-192-168-219-5
24/07/2023 13:41:16 INFO Stopping S3 Service
24/07/2023 13:41:17 INFO S3Server : clean all local resources
24/07/2023 13:41:48 INFO Starting S3Server Service.
24/07/2023 13:43:22 INFO create_rgw() started
24/07/2023 13:43:22 INFO Ceph RGW already installed
24/07/2023 13:43:22 INFO create_rgw() ended successfully

24/07/2023 13:43:22 INFO LockBase : Dropping old sessions
24/07/2023 13:43:32 INFO LockBase : Successfully dropped old sessions
24/07/2023 13:43:32 INFO Clean all old local resources.
24/07/2023 13:43:34 INFO S3Server : sync Consul settings
24/07/2023 13:43:34 INFO S3Server : sync Consul settings -> done
24/07/2023 13:43:35 INFO LockBase : Try to acquire the resource = S3-192-168-219-4.
24/07/2023 13:43:35 INFO LockBase : Succeeded on acquiring the resource = S3-192-168-219-4
24/07/2023 13:43:36 INFO LockBase : Try to acquire the resource = S3-192-168-219-3.
24/07/2023 13:43:39 INFO LockBase : Succeeded on acquiring the resource = S3-192-168-219-3
24/07/2023 13:43:40 INFO LockBase : Try to acquire the resource = S3-192-168-219-1.
24/07/2023 13:43:46 INFO LockBase : Succeeded on acquiring the resource = S3-192-168-219-1
24/07/2023 13:43:49 INFO S3Server : sync Consul settings
24/07/2023 13:43:49 INFO S3Server : sync Consul settings -> done

Thanks

bye
eazy

Last edited on July 24, 2023, 12:43 pm by eazyadm · #1

admin
2,930 Posts
July 24, 2023, 1:01 pm

is the cluster health ok ?

are you using relatively good hardware ? is the system under a lot of load ?

#2

eazyadm
25 Posts
July 24, 2023, 1:34 pm

Hi,

health Warning: 17 pgs not deep-scrubbed in time

This Warning appears some times.

Load each node 1.5 with 8 Cores, so should be fine.
128GB RAM only 25% used each node
NVMe Journal, ssd Cache, spinning hdd for storage
4x 10GBE Uplink each node

#3

admin
2,930 Posts
July 24, 2023, 3:06 pm

is the cluster loaded with i/o ? are the hdd disk % util high ?

can you delete the zone/zone group and redo ?

if it times out, how long does it take to timeout ? 1 min ?

#4

eazyadm
25 Posts
July 25, 2023, 6:06 am

Hi,

the Disc utilization on all nodes is about 20%.

It seems that the problem is gone after ~12 hours.

I will give feedback if there will be any more problems.

Thanks!

 

#5