ForumBug ReportingPossible bug with multipart uploa …

You need to log in to create posts and topics. Login · Register

Possible bug with multipart upload and S3

Pages: 1 2

f.cuseo
66 Posts

June 6, 2024, 4:13 pm

Quote from f.cuseo on June 6, 2024, 4:13 pm

Hello.

I have a 10 server cluster, upgraded to petasan 3.3.0 (from 3.2.0).
After upgrading, I have noticed that a nextcloud installation (with s3 primary storage) can't upload big files (not so big); so, after a lot of tests, I have tried with two different ubuntu servers, rclone, and if I try to upload without multipart upload, all works perfectly.
With multipart upload, rclone fails.
I am not 100% sure of the problem, but it starts after 3.2.0 to 3.3.0 upgrade.

panservice@s3tools:~$ rclone -vvvvvvvv copy pippo.tar s3storage:/nexttest
2024/06/06 16:01:01 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "-vvvvvvvv" "copy" "pippo.tar" "s3storage:/nexttest"]
2024/06/06 16:01:01 DEBUG : Creating backend with remote "pippo.tar"
2024/06/06 16:01:01 DEBUG : Using config file from "/home/panservice/.config/rclone/rclone.conf"
2024/06/06 16:01:01 DEBUG : fs cache: adding new entry for parent of "pippo.tar", "/home/panservice"
2024/06/06 16:01:01 DEBUG : Creating backend with remote "s3storage:/nexttest"
2024/06/06 16:01:01 DEBUG : fs cache: renaming cache item "s3storage:/nexttest" to be canonical "s3storage:nexttest"
2024/06/06 16:01:04 DEBUG : pippo.tar: Need to transfer - File not found at Destination
2024/06/06 16:01:04 INFO : S3 bucket nexttest: Bucket "nexttest" created with ACL "private"
2024/06/06 16:01:07 DEBUG : pippo.tar: multipart upload starting chunk 1 size 5M offset 0/309.307M
2024/06/06 16:01:07 DEBUG : pippo.tar: multipart upload starting chunk 2 size 5M offset 5M/309.307M
2024/06/06 16:01:07 DEBUG : pippo.tar: multipart upload starting chunk 3 size 5M offset 10M/309.307M
2024/06/06 16:01:07 DEBUG : pippo.tar: multipart upload starting chunk 4 size 5M offset 15M/309.307M
2024/06/06 16:01:37 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://s3storage.panservice.it:8000/nexttest/pippo.tar?partNumber=1&uploadId=2~nPO_7p470cF71bKfFpV2_7_XR9W-UKO": net/http: HTTP/1.x transport connec tion broken: unexpected EOF)
2024/06/06 16:01:37 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2024/06/06 16:01:40 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://s3storage.panservice.it:8000/nexttest/pippo.tar?partNumber=2&uploadId=2~nPO_7p470cF71bKfFpV2_7_XR9W-UKO": net/http: HTTP/1.x transport connec tion broken: unexpected EOF)
2024/06/06 16:01:40 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2024/06/06 16:01:40 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://s3storage.panservice.it:8000/nexttest/pippo.tar?partNumber=4&uploadId=2~nPO_7p470cF71bKfFpV2_7_XR9W-UKO": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/06 16:01:40 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2024/06/06 16:01:40 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://s3storage.panservice.it:8000/nexttest/pippo.tar?partNumber=3&uploadId=2~nPO_7p470cF71bKfFpV2_7_XR9W-UKO": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/06 16:01:40 DEBUG : pacer: Rate limited, increasing sleep to 80ms

Hello.

I have a 10 server cluster, upgraded to petasan 3.3.0 (from 3.2.0).
After upgrading, I have noticed that a nextcloud installation (with s3 primary storage) can't upload big files (not so big); so, after a lot of tests, I have tried with two different ubuntu servers, rclone, and if I try to upload without multipart upload, all works perfectly.
With multipart upload, rclone fails.
I am not 100% sure of the problem, but it starts after 3.2.0 to 3.3.0 upgrade.

panservice@s3tools:~$ rclone -vvvvvvvv copy pippo.tar s3storage:/nexttest
2024/06/06 16:01:01 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "-vvvvvvvv" "copy" "pippo.tar" "s3storage:/nexttest"]
2024/06/06 16:01:01 DEBUG : Creating backend with remote "pippo.tar"
2024/06/06 16:01:01 DEBUG : Using config file from "/home/panservice/.config/rclone/rclone.conf"
2024/06/06 16:01:01 DEBUG : fs cache: adding new entry for parent of "pippo.tar", "/home/panservice"
2024/06/06 16:01:01 DEBUG : Creating backend with remote "s3storage:/nexttest"
2024/06/06 16:01:01 DEBUG : fs cache: renaming cache item "s3storage:/nexttest" to be canonical "s3storage:nexttest"
2024/06/06 16:01:04 DEBUG : pippo.tar: Need to transfer - File not found at Destination
2024/06/06 16:01:04 INFO : S3 bucket nexttest: Bucket "nexttest" created with ACL "private"
2024/06/06 16:01:07 DEBUG : pippo.tar: multipart upload starting chunk 1 size 5M offset 0/309.307M
2024/06/06 16:01:07 DEBUG : pippo.tar: multipart upload starting chunk 2 size 5M offset 5M/309.307M
2024/06/06 16:01:07 DEBUG : pippo.tar: multipart upload starting chunk 3 size 5M offset 10M/309.307M
2024/06/06 16:01:07 DEBUG : pippo.tar: multipart upload starting chunk 4 size 5M offset 15M/309.307M
2024/06/06 16:01:37 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://s3storage.panservice.it:8000/nexttest/pippo.tar?partNumber=1&uploadId=2~nPO_7p470cF71bKfFpV2_7_XR9W-UKO": net/http: HTTP/1.x transport connec tion broken: unexpected EOF)
2024/06/06 16:01:37 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2024/06/06 16:01:40 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://s3storage.panservice.it:8000/nexttest/pippo.tar?partNumber=2&uploadId=2~nPO_7p470cF71bKfFpV2_7_XR9W-UKO": net/http: HTTP/1.x transport connec tion broken: unexpected EOF)
2024/06/06 16:01:40 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2024/06/06 16:01:40 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://s3storage.panservice.it:8000/nexttest/pippo.tar?partNumber=4&uploadId=2~nPO_7p470cF71bKfFpV2_7_XR9W-UKO": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/06 16:01:40 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2024/06/06 16:01:40 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://s3storage.panservice.it:8000/nexttest/pippo.tar?partNumber=3&uploadId=2~nPO_7p470cF71bKfFpV2_7_XR9W-UKO": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/06 16:01:40 DEBUG : pacer: Rate limited, increasing sleep to 80ms

#1

f.cuseo
66 Posts

June 7, 2024, 8:40 am

Quote from f.cuseo on June 7, 2024, 8:40 am

I have just installed a test petasan cluster with 3.2.1 ; no any problem at all.
Now i will try to upgrade to 3.3 to see if the problem will appear.

 

I have just installed a test petasan cluster with 3.2.1 ; no any problem at all.
Now i will try to upgrade to 3.3 to see if the problem will appear.

 

#2

f.cuseo
66 Posts

June 7, 2024, 9:23 am

Quote from f.cuseo on June 7, 2024, 9:23 am

So: installed a 3.2.1, all is fine:

rclone -vvvvvvvvvvvv copy pippo.tar s3test:/nexttest
2024/06/07 08:31:00 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "-vvvvvvvvvvvv" "copy" "pippo.tar" "s3test:/nexttest"]
2024/06/07 08:31:00 DEBUG : Creating backend with remote "pippo.tar"
2024/06/07 08:31:00 DEBUG : Using config file from "/home/panservice/.config/rclone/rclone.conf"
2024/06/07 08:31:00 DEBUG : fs cache: adding new entry for parent of "pippo.tar", "/home/panservice"
2024/06/07 08:31:00 DEBUG : Creating backend with remote "s3test:/nexttest"
2024/06/07 08:31:00 DEBUG : fs cache: renaming cache item "s3test:/nexttest" to be canonical "s3test:nexttest"
2024/06/07 08:31:00 DEBUG : pippo.tar: Need to transfer - File not found at Destination
2024/06/07 08:31:00 INFO : S3 bucket nexttest: Bucket "nexttest" created with ACL "private"
2024/06/07 08:31:02 DEBUG : pippo.tar: multipart upload starting chunk 1 size 5M offset 0/309.307M
2024/06/07 08:31:02 DEBUG : pippo.tar: multipart upload starting chunk 2 size 5M offset 5M/309.307M
2024/06/07 08:31:02 DEBUG : pippo.tar: multipart upload starting chunk 3 size 5M offset 10M/309.307M
2024/06/07 08:31:02 DEBUG : pippo.tar: multipart upload starting chunk 4 size 5M offset 15M/309.307M
2024/06/07 08:31:03 DEBUG : pippo.tar: multipart upload starting chunk 5 size 5M offset 20M/309.307M
2024/06/07 08:31:03 DEBUG : pippo.tar: multipart upload starting chunk 6 size 5M offset 25M/309.307M
2024/06/07 08:31:03 DEBUG : pippo.tar: multipart upload starting chunk 7 size 5M offset 30M/309.307M
2024/06/07 08:31:03 DEBUG : pippo.tar: multipart upload starting chunk 8 size 5M offset 35M/309.307M
2024/06/07 08:31:03 DEBUG : pippo.tar: multipart upload starting chunk 9 size 5M offset 40M/309.307M
2024/06/07 08:31:04 DEBUG : pippo.tar: multipart upload starting chunk 10 size 5M offset 45M/309.307M
2024/06/07 08:31:04 DEBUG : pippo.tar: multipart upload starting chunk 11 size 5M offset 50M/309.307M
2024/06/07 08:31:04 DEBUG : pippo.tar: multipart upload starting chunk 12 size 5M offset 55M/309.307M
2024/06/07 08:31:04 DEBUG : pippo.tar: multipart upload starting chunk 13 size 5M offset 60M/309.307M
2024/06/07 08:31:04 DEBUG : pippo.tar: multipart upload starting chunk 14 size 5M offset 65M/309.307M
2024/06/07 08:31:05 DEBUG : pippo.tar: multipart upload starting chunk 15 size 5M offset 70M/309.307M
2024/06/07 08:31:05 DEBUG : pippo.tar: multipart upload starting chunk 16 size 5M offset 75M/309.307M
2024/06/07 08:31:05 DEBUG : pippo.tar: multipart upload starting chunk 17 size 5M offset 80M/309.307M
2024/06/07 08:31:05 DEBUG : pippo.tar: multipart upload starting chunk 18 size 5M offset 85M/309.307M
2024/06/07 08:31:05 DEBUG : pippo.tar: multipart upload starting chunk 19 size 5M offset 90M/309.307M
2024/06/07 08:31:06 DEBUG : pippo.tar: multipart upload starting chunk 20 size 5M offset 95M/309.307M
2024/06/07 08:31:06 DEBUG : pippo.tar: multipart upload starting chunk 21 size 5M offset 100M/309.307M
2024/06/07 08:31:06 DEBUG : pippo.tar: multipart upload starting chunk 22 size 5M offset 105M/309.307M
2024/06/07 08:31:07 DEBUG : pippo.tar: multipart upload starting chunk 23 size 5M offset 110M/309.307M
2024/06/07 08:31:07 DEBUG : pippo.tar: multipart upload starting chunk 24 size 5M offset 115M/309.307M
2024/06/07 08:31:07 DEBUG : pippo.tar: multipart upload starting chunk 25 size 5M offset 120M/309.307M
2024/06/07 08:31:07 DEBUG : pippo.tar: multipart upload starting chunk 26 size 5M offset 125M/309.307M
2024/06/07 08:31:08 DEBUG : pippo.tar: multipart upload starting chunk 27 size 5M offset 130M/309.307M
2024/06/07 08:31:08 DEBUG : pippo.tar: multipart upload starting chunk 28 size 5M offset 135M/309.307M
2024/06/07 08:31:08 DEBUG : pippo.tar: multipart upload starting chunk 29 size 5M offset 140M/309.307M
2024/06/07 08:31:09 DEBUG : pippo.tar: multipart upload starting chunk 30 size 5M offset 145M/309.307M
2024/06/07 08:31:09 DEBUG : pippo.tar: multipart upload starting chunk 31 size 5M offset 150M/309.307M
2024/06/07 08:31:09 DEBUG : pippo.tar: multipart upload starting chunk 32 size 5M offset 155M/309.307M
2024/06/07 08:31:09 DEBUG : pippo.tar: multipart upload starting chunk 33 size 5M offset 160M/309.307M
2024/06/07 08:31:09 DEBUG : pippo.tar: multipart upload starting chunk 34 size 5M offset 165M/309.307M
2024/06/07 08:31:10 DEBUG : pippo.tar: multipart upload starting chunk 35 size 5M offset 170M/309.307M
2024/06/07 08:31:10 DEBUG : pippo.tar: multipart upload starting chunk 36 size 5M offset 175M/309.307M
2024/06/07 08:31:10 DEBUG : pippo.tar: multipart upload starting chunk 37 size 5M offset 180M/309.307M
2024/06/07 08:31:10 DEBUG : pippo.tar: multipart upload starting chunk 38 size 5M offset 185M/309.307M
2024/06/07 08:31:10 DEBUG : pippo.tar: multipart upload starting chunk 39 size 5M offset 190M/309.307M
2024/06/07 08:31:10 DEBUG : pippo.tar: multipart upload starting chunk 40 size 5M offset 195M/309.307M
2024/06/07 08:31:11 DEBUG : pippo.tar: multipart upload starting chunk 41 size 5M offset 200M/309.307M
2024/06/07 08:31:11 DEBUG : pippo.tar: multipart upload starting chunk 42 size 5M offset 205M/309.307M
2024/06/07 08:31:11 DEBUG : pippo.tar: multipart upload starting chunk 43 size 5M offset 210M/309.307M
2024/06/07 08:31:11 DEBUG : pippo.tar: multipart upload starting chunk 44 size 5M offset 215M/309.307M
2024/06/07 08:31:11 DEBUG : pippo.tar: multipart upload starting chunk 45 size 5M offset 220M/309.307M
2024/06/07 08:31:12 DEBUG : pippo.tar: multipart upload starting chunk 46 size 5M offset 225M/309.307M
2024/06/07 08:31:12 DEBUG : pippo.tar: multipart upload starting chunk 47 size 5M offset 230M/309.307M
2024/06/07 08:31:12 DEBUG : pippo.tar: multipart upload starting chunk 48 size 5M offset 235M/309.307M
2024/06/07 08:31:12 DEBUG : pippo.tar: multipart upload starting chunk 49 size 5M offset 240M/309.307M
2024/06/07 08:31:12 DEBUG : pippo.tar: multipart upload starting chunk 50 size 5M offset 245M/309.307M
2024/06/07 08:31:12 DEBUG : pippo.tar: multipart upload starting chunk 51 size 5M offset 250M/309.307M
2024/06/07 08:31:13 DEBUG : pippo.tar: multipart upload starting chunk 52 size 5M offset 255M/309.307M
2024/06/07 08:31:13 DEBUG : pippo.tar: multipart upload starting chunk 53 size 5M offset 260M/309.307M
2024/06/07 08:31:13 DEBUG : pippo.tar: multipart upload starting chunk 54 size 5M offset 265M/309.307M
2024/06/07 08:31:13 DEBUG : pippo.tar: multipart upload starting chunk 55 size 5M offset 270M/309.307M
2024/06/07 08:31:13 DEBUG : pippo.tar: multipart upload starting chunk 56 size 5M offset 275M/309.307M
2024/06/07 08:31:13 DEBUG : pippo.tar: multipart upload starting chunk 57 size 5M offset 280M/309.307M
2024/06/07 08:31:14 DEBUG : pippo.tar: multipart upload starting chunk 58 size 5M offset 285M/309.307M
2024/06/07 08:31:14 DEBUG : pippo.tar: multipart upload starting chunk 59 size 5M offset 290M/309.307M
2024/06/07 08:31:14 DEBUG : pippo.tar: multipart upload starting chunk 60 size 5M offset 295M/309.307M
2024/06/07 08:31:14 DEBUG : pippo.tar: multipart upload starting chunk 61 size 5M offset 300M/309.307M
2024/06/07 08:31:14 DEBUG : pippo.tar: multipart upload starting chunk 62 size 4.307M offset 305M/309.307M
2024/06/07 08:31:15 DEBUG : pippo.tar: MD5 = 5e3aca0406e0744e65903a166769903c OK
2024/06/07 08:31:15 INFO : pippo.tar: Copied (new)
2024/06/07 08:31:15 INFO :
Transferred: 309.307M / 309.307 MBytes, 100%, 20.440 MBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 15.1s

2024/06/07 08:31:15 DEBUG : 6 go routines active

------------------------------------------

Upgraded to 3.3.0 with /opt/petasan/scripts/online-updates/update.sh:

rclone -vvvvvvvvvvvv copy pippo3.tar s3test:/nexttest
2024/06/07 09:20:53 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "-vvvvvvvvvvvv" "copy" "pippo3.tar" "s3test:/nexttest"]
2024/06/07 09:20:53 DEBUG : Creating backend with remote "pippo3.tar"
2024/06/07 09:20:53 DEBUG : Using config file from "/home/panservice/.config/rclone/rclone.conf"
2024/06/07 09:20:53 DEBUG : fs cache: adding new entry for parent of "pippo3.tar", "/home/panservice"
2024/06/07 09:20:53 DEBUG : Creating backend with remote "s3test:/nexttest"
2024/06/07 09:20:53 DEBUG : fs cache: renaming cache item "s3test:/nexttest" to be canonical "s3test:nexttest"
2024/06/07 09:20:53 DEBUG : pippo3.tar: Need to transfer - File not found at Destination
2024/06/07 09:20:53 INFO : S3 bucket nexttest: Bucket "nexttest" created with ACL "private"
2024/06/07 09:20:55 DEBUG : pippo3.tar: multipart upload starting chunk 1 size 5M offset 0/309.307M
2024/06/07 09:20:55 DEBUG : pippo3.tar: multipart upload starting chunk 2 size 5M offset 5M/309.307M
2024/06/07 09:20:55 DEBUG : pippo3.tar: multipart upload starting chunk 3 size 5M offset 10M/309.307M
2024/06/07 09:20:55 DEBUG : pippo3.tar: multipart upload starting chunk 4 size 5M offset 15M/309.307M
2024/06/07 09:20:56 DEBUG : pippo3.tar: multipart upload starting chunk 5 size 5M offset 20M/309.307M
2024/06/07 09:20:56 DEBUG : pippo3.tar: multipart upload starting chunk 6 size 5M offset 25M/309.307M
2024/06/07 09:20:56 DEBUG : pippo3.tar: multipart upload starting chunk 7 size 5M offset 30M/309.307M
2024/06/07 09:20:56 DEBUG : pippo3.tar: multipart upload starting chunk 8 size 5M offset 35M/309.307M
2024/06/07 09:21:26 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=5&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:26 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2024/06/07 09:21:27 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=7&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:27 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2024/06/07 09:21:27 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=6&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:27 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2024/06/07 09:21:27 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=8&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:27 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2024/06/07 09:21:27 DEBUG : pacer: Reducing sleep to 60ms
2024/06/07 09:21:27 DEBUG : pippo3.tar: multipart upload starting chunk 9 size 5M offset 40M/309.307M
2024/06/07 09:21:27 DEBUG : pacer: Reducing sleep to 45ms
2024/06/07 09:21:27 DEBUG : pippo3.tar: multipart upload starting chunk 10 size 5M offset 45M/309.307M
2024/06/07 09:21:53 INFO :
Transferred: 50M / 309.307 MBytes, 16%, 854.688 kBytes/s, ETA 5m10s
Transferred: 0 / 1, 0%
Elapsed time: 1m0.0s
Transferring:
* pippo3.tar: 16% /309.307M, 291.325k/s, 15m11s

2024/06/07 09:21:57 DEBUG : pacer: low level retry 2/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=7&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:57 DEBUG : pacer: Rate limited, increasing sleep to 90ms
2024/06/07 09:21:57 DEBUG : pacer: low level retry 2/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=8&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:57 DEBUG : pacer: Rate limited, increasing sleep to 180ms
2024/06/07 09:21:57 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=9&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:57 DEBUG : pacer: Rate limited, increasing sleep to 360ms
2024/06/07 09:21:57 DEBUG : pacer: Reducing sleep to 270ms
2024/06/07 09:21:57 DEBUG : pippo3.tar: multipart upload starting chunk 11 size 5M offset 50M/309.307M
2024/06/07 09:21:58 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=10&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:58 DEBUG : pacer: Rate limited, increasing sleep to 540ms
2024/06/07 09:21:58 DEBUG : pacer: Reducing sleep to 405ms
2024/06/07 09:21:58 DEBUG : pippo3.tar: multipart upload starting chunk 12 size 5M offset 55M/309.307M
^C2024/06/07 09:21:59 INFO : Signal received: interrupt
2024/06/07 09:21:59 DEBUG : pippo3.tar: Cancelling multipart upload
2024/06/07 09:22:00 DEBUG : pacer: Reducing sleep to 303.75ms
2024/06/07 09:22:00 INFO : Exiting...

 

So: installed a 3.2.1, all is fine:

rclone -vvvvvvvvvvvv copy pippo.tar s3test:/nexttest
2024/06/07 08:31:00 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "-vvvvvvvvvvvv" "copy" "pippo.tar" "s3test:/nexttest"]
2024/06/07 08:31:00 DEBUG : Creating backend with remote "pippo.tar"
2024/06/07 08:31:00 DEBUG : Using config file from "/home/panservice/.config/rclone/rclone.conf"
2024/06/07 08:31:00 DEBUG : fs cache: adding new entry for parent of "pippo.tar", "/home/panservice"
2024/06/07 08:31:00 DEBUG : Creating backend with remote "s3test:/nexttest"
2024/06/07 08:31:00 DEBUG : fs cache: renaming cache item "s3test:/nexttest" to be canonical "s3test:nexttest"
2024/06/07 08:31:00 DEBUG : pippo.tar: Need to transfer - File not found at Destination
2024/06/07 08:31:00 INFO : S3 bucket nexttest: Bucket "nexttest" created with ACL "private"
2024/06/07 08:31:02 DEBUG : pippo.tar: multipart upload starting chunk 1 size 5M offset 0/309.307M
2024/06/07 08:31:02 DEBUG : pippo.tar: multipart upload starting chunk 2 size 5M offset 5M/309.307M
2024/06/07 08:31:02 DEBUG : pippo.tar: multipart upload starting chunk 3 size 5M offset 10M/309.307M
2024/06/07 08:31:02 DEBUG : pippo.tar: multipart upload starting chunk 4 size 5M offset 15M/309.307M
2024/06/07 08:31:03 DEBUG : pippo.tar: multipart upload starting chunk 5 size 5M offset 20M/309.307M
2024/06/07 08:31:03 DEBUG : pippo.tar: multipart upload starting chunk 6 size 5M offset 25M/309.307M
2024/06/07 08:31:03 DEBUG : pippo.tar: multipart upload starting chunk 7 size 5M offset 30M/309.307M
2024/06/07 08:31:03 DEBUG : pippo.tar: multipart upload starting chunk 8 size 5M offset 35M/309.307M
2024/06/07 08:31:03 DEBUG : pippo.tar: multipart upload starting chunk 9 size 5M offset 40M/309.307M
2024/06/07 08:31:04 DEBUG : pippo.tar: multipart upload starting chunk 10 size 5M offset 45M/309.307M
2024/06/07 08:31:04 DEBUG : pippo.tar: multipart upload starting chunk 11 size 5M offset 50M/309.307M
2024/06/07 08:31:04 DEBUG : pippo.tar: multipart upload starting chunk 12 size 5M offset 55M/309.307M
2024/06/07 08:31:04 DEBUG : pippo.tar: multipart upload starting chunk 13 size 5M offset 60M/309.307M
2024/06/07 08:31:04 DEBUG : pippo.tar: multipart upload starting chunk 14 size 5M offset 65M/309.307M
2024/06/07 08:31:05 DEBUG : pippo.tar: multipart upload starting chunk 15 size 5M offset 70M/309.307M
2024/06/07 08:31:05 DEBUG : pippo.tar: multipart upload starting chunk 16 size 5M offset 75M/309.307M
2024/06/07 08:31:05 DEBUG : pippo.tar: multipart upload starting chunk 17 size 5M offset 80M/309.307M
2024/06/07 08:31:05 DEBUG : pippo.tar: multipart upload starting chunk 18 size 5M offset 85M/309.307M
2024/06/07 08:31:05 DEBUG : pippo.tar: multipart upload starting chunk 19 size 5M offset 90M/309.307M
2024/06/07 08:31:06 DEBUG : pippo.tar: multipart upload starting chunk 20 size 5M offset 95M/309.307M
2024/06/07 08:31:06 DEBUG : pippo.tar: multipart upload starting chunk 21 size 5M offset 100M/309.307M
2024/06/07 08:31:06 DEBUG : pippo.tar: multipart upload starting chunk 22 size 5M offset 105M/309.307M
2024/06/07 08:31:07 DEBUG : pippo.tar: multipart upload starting chunk 23 size 5M offset 110M/309.307M
2024/06/07 08:31:07 DEBUG : pippo.tar: multipart upload starting chunk 24 size 5M offset 115M/309.307M
2024/06/07 08:31:07 DEBUG : pippo.tar: multipart upload starting chunk 25 size 5M offset 120M/309.307M
2024/06/07 08:31:07 DEBUG : pippo.tar: multipart upload starting chunk 26 size 5M offset 125M/309.307M
2024/06/07 08:31:08 DEBUG : pippo.tar: multipart upload starting chunk 27 size 5M offset 130M/309.307M
2024/06/07 08:31:08 DEBUG : pippo.tar: multipart upload starting chunk 28 size 5M offset 135M/309.307M
2024/06/07 08:31:08 DEBUG : pippo.tar: multipart upload starting chunk 29 size 5M offset 140M/309.307M
2024/06/07 08:31:09 DEBUG : pippo.tar: multipart upload starting chunk 30 size 5M offset 145M/309.307M
2024/06/07 08:31:09 DEBUG : pippo.tar: multipart upload starting chunk 31 size 5M offset 150M/309.307M
2024/06/07 08:31:09 DEBUG : pippo.tar: multipart upload starting chunk 32 size 5M offset 155M/309.307M
2024/06/07 08:31:09 DEBUG : pippo.tar: multipart upload starting chunk 33 size 5M offset 160M/309.307M
2024/06/07 08:31:09 DEBUG : pippo.tar: multipart upload starting chunk 34 size 5M offset 165M/309.307M
2024/06/07 08:31:10 DEBUG : pippo.tar: multipart upload starting chunk 35 size 5M offset 170M/309.307M
2024/06/07 08:31:10 DEBUG : pippo.tar: multipart upload starting chunk 36 size 5M offset 175M/309.307M
2024/06/07 08:31:10 DEBUG : pippo.tar: multipart upload starting chunk 37 size 5M offset 180M/309.307M
2024/06/07 08:31:10 DEBUG : pippo.tar: multipart upload starting chunk 38 size 5M offset 185M/309.307M
2024/06/07 08:31:10 DEBUG : pippo.tar: multipart upload starting chunk 39 size 5M offset 190M/309.307M
2024/06/07 08:31:10 DEBUG : pippo.tar: multipart upload starting chunk 40 size 5M offset 195M/309.307M
2024/06/07 08:31:11 DEBUG : pippo.tar: multipart upload starting chunk 41 size 5M offset 200M/309.307M
2024/06/07 08:31:11 DEBUG : pippo.tar: multipart upload starting chunk 42 size 5M offset 205M/309.307M
2024/06/07 08:31:11 DEBUG : pippo.tar: multipart upload starting chunk 43 size 5M offset 210M/309.307M
2024/06/07 08:31:11 DEBUG : pippo.tar: multipart upload starting chunk 44 size 5M offset 215M/309.307M
2024/06/07 08:31:11 DEBUG : pippo.tar: multipart upload starting chunk 45 size 5M offset 220M/309.307M
2024/06/07 08:31:12 DEBUG : pippo.tar: multipart upload starting chunk 46 size 5M offset 225M/309.307M
2024/06/07 08:31:12 DEBUG : pippo.tar: multipart upload starting chunk 47 size 5M offset 230M/309.307M
2024/06/07 08:31:12 DEBUG : pippo.tar: multipart upload starting chunk 48 size 5M offset 235M/309.307M
2024/06/07 08:31:12 DEBUG : pippo.tar: multipart upload starting chunk 49 size 5M offset 240M/309.307M
2024/06/07 08:31:12 DEBUG : pippo.tar: multipart upload starting chunk 50 size 5M offset 245M/309.307M
2024/06/07 08:31:12 DEBUG : pippo.tar: multipart upload starting chunk 51 size 5M offset 250M/309.307M
2024/06/07 08:31:13 DEBUG : pippo.tar: multipart upload starting chunk 52 size 5M offset 255M/309.307M
2024/06/07 08:31:13 DEBUG : pippo.tar: multipart upload starting chunk 53 size 5M offset 260M/309.307M
2024/06/07 08:31:13 DEBUG : pippo.tar: multipart upload starting chunk 54 size 5M offset 265M/309.307M
2024/06/07 08:31:13 DEBUG : pippo.tar: multipart upload starting chunk 55 size 5M offset 270M/309.307M
2024/06/07 08:31:13 DEBUG : pippo.tar: multipart upload starting chunk 56 size 5M offset 275M/309.307M
2024/06/07 08:31:13 DEBUG : pippo.tar: multipart upload starting chunk 57 size 5M offset 280M/309.307M
2024/06/07 08:31:14 DEBUG : pippo.tar: multipart upload starting chunk 58 size 5M offset 285M/309.307M
2024/06/07 08:31:14 DEBUG : pippo.tar: multipart upload starting chunk 59 size 5M offset 290M/309.307M
2024/06/07 08:31:14 DEBUG : pippo.tar: multipart upload starting chunk 60 size 5M offset 295M/309.307M
2024/06/07 08:31:14 DEBUG : pippo.tar: multipart upload starting chunk 61 size 5M offset 300M/309.307M
2024/06/07 08:31:14 DEBUG : pippo.tar: multipart upload starting chunk 62 size 4.307M offset 305M/309.307M
2024/06/07 08:31:15 DEBUG : pippo.tar: MD5 = 5e3aca0406e0744e65903a166769903c OK
2024/06/07 08:31:15 INFO : pippo.tar: Copied (new)
2024/06/07 08:31:15 INFO :
Transferred: 309.307M / 309.307 MBytes, 100%, 20.440 MBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 15.1s

2024/06/07 08:31:15 DEBUG : 6 go routines active

------------------------------------------

Upgraded to 3.3.0 with /opt/petasan/scripts/online-updates/update.sh:

rclone -vvvvvvvvvvvv copy pippo3.tar s3test:/nexttest
2024/06/07 09:20:53 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "-vvvvvvvvvvvv" "copy" "pippo3.tar" "s3test:/nexttest"]
2024/06/07 09:20:53 DEBUG : Creating backend with remote "pippo3.tar"
2024/06/07 09:20:53 DEBUG : Using config file from "/home/panservice/.config/rclone/rclone.conf"
2024/06/07 09:20:53 DEBUG : fs cache: adding new entry for parent of "pippo3.tar", "/home/panservice"
2024/06/07 09:20:53 DEBUG : Creating backend with remote "s3test:/nexttest"
2024/06/07 09:20:53 DEBUG : fs cache: renaming cache item "s3test:/nexttest" to be canonical "s3test:nexttest"
2024/06/07 09:20:53 DEBUG : pippo3.tar: Need to transfer - File not found at Destination
2024/06/07 09:20:53 INFO : S3 bucket nexttest: Bucket "nexttest" created with ACL "private"
2024/06/07 09:20:55 DEBUG : pippo3.tar: multipart upload starting chunk 1 size 5M offset 0/309.307M
2024/06/07 09:20:55 DEBUG : pippo3.tar: multipart upload starting chunk 2 size 5M offset 5M/309.307M
2024/06/07 09:20:55 DEBUG : pippo3.tar: multipart upload starting chunk 3 size 5M offset 10M/309.307M
2024/06/07 09:20:55 DEBUG : pippo3.tar: multipart upload starting chunk 4 size 5M offset 15M/309.307M
2024/06/07 09:20:56 DEBUG : pippo3.tar: multipart upload starting chunk 5 size 5M offset 20M/309.307M
2024/06/07 09:20:56 DEBUG : pippo3.tar: multipart upload starting chunk 6 size 5M offset 25M/309.307M
2024/06/07 09:20:56 DEBUG : pippo3.tar: multipart upload starting chunk 7 size 5M offset 30M/309.307M
2024/06/07 09:20:56 DEBUG : pippo3.tar: multipart upload starting chunk 8 size 5M offset 35M/309.307M
2024/06/07 09:21:26 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=5&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:26 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2024/06/07 09:21:27 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=7&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:27 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2024/06/07 09:21:27 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=6&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:27 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2024/06/07 09:21:27 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=8&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:27 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2024/06/07 09:21:27 DEBUG : pacer: Reducing sleep to 60ms
2024/06/07 09:21:27 DEBUG : pippo3.tar: multipart upload starting chunk 9 size 5M offset 40M/309.307M
2024/06/07 09:21:27 DEBUG : pacer: Reducing sleep to 45ms
2024/06/07 09:21:27 DEBUG : pippo3.tar: multipart upload starting chunk 10 size 5M offset 45M/309.307M
2024/06/07 09:21:53 INFO :
Transferred: 50M / 309.307 MBytes, 16%, 854.688 kBytes/s, ETA 5m10s
Transferred: 0 / 1, 0%
Elapsed time: 1m0.0s
Transferring:
* pippo3.tar: 16% /309.307M, 291.325k/s, 15m11s

2024/06/07 09:21:57 DEBUG : pacer: low level retry 2/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=7&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:57 DEBUG : pacer: Rate limited, increasing sleep to 90ms
2024/06/07 09:21:57 DEBUG : pacer: low level retry 2/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=8&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:57 DEBUG : pacer: Rate limited, increasing sleep to 180ms
2024/06/07 09:21:57 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=9&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:57 DEBUG : pacer: Rate limited, increasing sleep to 360ms
2024/06/07 09:21:57 DEBUG : pacer: Reducing sleep to 270ms
2024/06/07 09:21:57 DEBUG : pippo3.tar: multipart upload starting chunk 11 size 5M offset 50M/309.307M
2024/06/07 09:21:58 DEBUG : pacer: low level retry 1/10 (error RequestError: send request failed
caused by: Put "https://tests3.panservice.it:8000/nexttest/pippo3.tar?partNumber=10&uploadId=2~lonwIXW840kcrjLUfomBWqb_pb5U4ZB": net/http: HTTP/1.x transport connection broken: unexpected EOF)
2024/06/07 09:21:58 DEBUG : pacer: Rate limited, increasing sleep to 540ms
2024/06/07 09:21:58 DEBUG : pacer: Reducing sleep to 405ms
2024/06/07 09:21:58 DEBUG : pippo3.tar: multipart upload starting chunk 12 size 5M offset 55M/309.307M
^C2024/06/07 09:21:59 INFO : Signal received: interrupt
2024/06/07 09:21:59 DEBUG : pippo3.tar: Cancelling multipart upload
2024/06/07 09:22:00 DEBUG : pacer: Reducing sleep to 303.75ms
2024/06/07 09:22:00 INFO : Exiting...

 

#3

f.cuseo
66 Posts

June 7, 2024, 12:03 pm

Quote from f.cuseo on June 7, 2024, 12:03 pm

Using a more recent rclone version:

./rclone -vvvvvvvvvvvv copy /home/panservice/pippo3.tar s3test:/nexttest
2024/06/07 11:00:06 DEBUG : rclone: Version "v1.64.2" starting with parameters ["./rclone" "-vvvvvvvvvvvv" "copy" "/home/panservice/pippo3.tar" "s3test:/nexttest"]
2024/06/07 11:00:06 DEBUG : Creating backend with remote "/home/panservice/pippo3.tar"
2024/06/07 11:00:06 DEBUG : Using config file from "/home/panservice/.config/rclone/rclone.conf"
2024/06/07 11:00:06 DEBUG : fs cache: adding new entry for parent of "/home/panservice/pippo3.tar", "/home/panservice"
2024/06/07 11:00:06 DEBUG : Creating backend with remote "s3test:/nexttest"
2024/06/07 11:00:06 DEBUG : Resolving service "s3" region "us-east-1"
2024/06/07 11:00:06 DEBUG : fs cache: renaming cache item "s3test:/nexttest" to be canonical "s3test:nexttest"
2024/06/07 11:00:06 DEBUG : pippo3.tar: Need to transfer - File not found at Destination
2024/06/07 11:00:06 INFO : S3 bucket nexttest: Bucket "nexttest" created with ACL "private"
2024/06/07 11:00:08 DEBUG : pippo3.tar: open chunk writer: started multipart upload: 2~3T3t8SiAuo-3uCdLmuaLNo6SG89jVKW
2024/06/07 11:00:08 DEBUG : pippo3.tar: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2024/06/07 11:00:08 DEBUG : pippo3.tar: Starting multi-thread copy with 62 chunks of size 5Mi with 4 parallel streams
2024/06/07 11:00:08 DEBUG : pippo3.tar: multi-thread copy: chunk 4/62 (15728640-20971520) size 5Mi starting
2024/06/07 11:00:08 DEBUG : pippo3.tar: multi-thread copy: chunk 2/62 (5242880-10485760) size 5Mi starting
2024/06/07 11:00:08 DEBUG : pippo3.tar: multi-thread copy: chunk 1/62 (0-5242880) size 5Mi starting
2024/06/07 11:00:08 DEBUG : pippo3.tar: multi-thread copy: chunk 3/62 (10485760-15728640) size 5Mi starting
2024/06/07 11:00:09 DEBUG : pippo3.tar: multipart upload wrote chunk 4 with 5242880 bytes and etag "63bf58d732bc48b7e9fac7996a1b0763"
2024/06/07 11:00:09 DEBUG : pippo3.tar: multi-thread copy: chunk 4/62 (15728640-20971520) size 5Mi finished
2024/06/07 11:00:09 DEBUG : pippo3.tar: multi-thread copy: chunk 5/62 (20971520-26214400) size 5Mi starting
2024/06/07 11:00:09 DEBUG : pippo3.tar: multipart upload wrote chunk 2 with 5242880 bytes and etag "94298cb24c5f8a50416fe305ec184147"
2024/06/07 11:00:09 DEBUG : pippo3.tar: multi-thread copy: chunk 2/62 (5242880-10485760) size 5Mi finished
2024/06/07 11:00:09 DEBUG : pippo3.tar: multi-thread copy: chunk 6/62 (26214400-31457280) size 5Mi starting
2024/06/07 11:00:39 DEBUG : pippo3.tar: multipart upload wrote chunk 1 with 5242880 bytes and etag "ba5fcc6eb966cba35817f4f264877cea"
2024/06/07 11:00:39 DEBUG : pippo3.tar: multi-thread copy: chunk 1/62 (0-5242880) size 5Mi finished
2024/06/07 11:00:39 DEBUG : pippo3.tar: multi-thread copy: chunk 7/62 (31457280-36700160) size 5Mi starting
2024/06/07 11:00:40 DEBUG : pippo3.tar: multipart upload wrote chunk 5 with 5242880 bytes and etag "cbb3009b1af1bde2d64b2f6a7055f5d9"
2024/06/07 11:00:40 DEBUG : pippo3.tar: multi-thread copy: chunk 5/62 (20971520-26214400) size 5Mi finished
2024/06/07 11:00:40 DEBUG : pippo3.tar: multi-thread copy: chunk 8/62 (36700160-41943040) size 5Mi starting
2024/06/07 11:01:06 INFO :
Transferred: 60 MiB / 309.307 MiB, 19%, 490.050 KiB/s, ETA 8m40s
Transferred: 0 / 1, 0%
Elapsed time: 1m0.0s
Transferring:
* pippo3.tar: 19% /309.307Mi, 478.682Ki/s, 8m53s
...........................
...........................

pippo3.tar:295% /309.307Mi, 99.701Ki/s, -

2024/06/07 11:17:21 DEBUG : pippo3.tar: multipart upload wrote chunk 55 with 5242880 bytes and etag "6952fd52be4940c9730c2ab988009110"
2024/06/07 11:17:21 DEBUG : pippo3.tar: multi-thread copy: chunk 55/62 (283115520-288358400) size 5Mi finished
2024/06/07 11:17:21 DEBUG : pippo3.tar: multipart upload "2~3T3t8SiAuo-3uCdLmuaLNo6SG89jVKW" finished
2024/06/07 11:17:21 DEBUG : pippo3.tar: Finished multi-thread copy with 62 parts of size 5Mi
2024/06/07 11:17:21 DEBUG : pippo3.tar: md5 = 5e3aca0406e0744e65903a166769903c OK
2024/06/07 11:17:21 INFO : pippo3.tar: Multi-thread Copied (new)
2024/06/07 11:17:21 INFO :
Transferred: 917.920 MiB / 917.920 MiB, 100%, 40.128 KiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 17m14.6s

So, 17 minutes to upload a 300Mbyte file, but with 917Mbyte transferred.
So, something is wrong in petasan 3.3.0 (or ceph 17.2.7)

 

Using a more recent rclone version:

./rclone -vvvvvvvvvvvv copy /home/panservice/pippo3.tar s3test:/nexttest
2024/06/07 11:00:06 DEBUG : rclone: Version "v1.64.2" starting with parameters ["./rclone" "-vvvvvvvvvvvv" "copy" "/home/panservice/pippo3.tar" "s3test:/nexttest"]
2024/06/07 11:00:06 DEBUG : Creating backend with remote "/home/panservice/pippo3.tar"
2024/06/07 11:00:06 DEBUG : Using config file from "/home/panservice/.config/rclone/rclone.conf"
2024/06/07 11:00:06 DEBUG : fs cache: adding new entry for parent of "/home/panservice/pippo3.tar", "/home/panservice"
2024/06/07 11:00:06 DEBUG : Creating backend with remote "s3test:/nexttest"
2024/06/07 11:00:06 DEBUG : Resolving service "s3" region "us-east-1"
2024/06/07 11:00:06 DEBUG : fs cache: renaming cache item "s3test:/nexttest" to be canonical "s3test:nexttest"
2024/06/07 11:00:06 DEBUG : pippo3.tar: Need to transfer - File not found at Destination
2024/06/07 11:00:06 INFO : S3 bucket nexttest: Bucket "nexttest" created with ACL "private"
2024/06/07 11:00:08 DEBUG : pippo3.tar: open chunk writer: started multipart upload: 2~3T3t8SiAuo-3uCdLmuaLNo6SG89jVKW
2024/06/07 11:00:08 DEBUG : pippo3.tar: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2024/06/07 11:00:08 DEBUG : pippo3.tar: Starting multi-thread copy with 62 chunks of size 5Mi with 4 parallel streams
2024/06/07 11:00:08 DEBUG : pippo3.tar: multi-thread copy: chunk 4/62 (15728640-20971520) size 5Mi starting
2024/06/07 11:00:08 DEBUG : pippo3.tar: multi-thread copy: chunk 2/62 (5242880-10485760) size 5Mi starting
2024/06/07 11:00:08 DEBUG : pippo3.tar: multi-thread copy: chunk 1/62 (0-5242880) size 5Mi starting
2024/06/07 11:00:08 DEBUG : pippo3.tar: multi-thread copy: chunk 3/62 (10485760-15728640) size 5Mi starting
2024/06/07 11:00:09 DEBUG : pippo3.tar: multipart upload wrote chunk 4 with 5242880 bytes and etag "63bf58d732bc48b7e9fac7996a1b0763"
2024/06/07 11:00:09 DEBUG : pippo3.tar: multi-thread copy: chunk 4/62 (15728640-20971520) size 5Mi finished
2024/06/07 11:00:09 DEBUG : pippo3.tar: multi-thread copy: chunk 5/62 (20971520-26214400) size 5Mi starting
2024/06/07 11:00:09 DEBUG : pippo3.tar: multipart upload wrote chunk 2 with 5242880 bytes and etag "94298cb24c5f8a50416fe305ec184147"
2024/06/07 11:00:09 DEBUG : pippo3.tar: multi-thread copy: chunk 2/62 (5242880-10485760) size 5Mi finished
2024/06/07 11:00:09 DEBUG : pippo3.tar: multi-thread copy: chunk 6/62 (26214400-31457280) size 5Mi starting
2024/06/07 11:00:39 DEBUG : pippo3.tar: multipart upload wrote chunk 1 with 5242880 bytes and etag "ba5fcc6eb966cba35817f4f264877cea"
2024/06/07 11:00:39 DEBUG : pippo3.tar: multi-thread copy: chunk 1/62 (0-5242880) size 5Mi finished
2024/06/07 11:00:39 DEBUG : pippo3.tar: multi-thread copy: chunk 7/62 (31457280-36700160) size 5Mi starting
2024/06/07 11:00:40 DEBUG : pippo3.tar: multipart upload wrote chunk 5 with 5242880 bytes and etag "cbb3009b1af1bde2d64b2f6a7055f5d9"
2024/06/07 11:00:40 DEBUG : pippo3.tar: multi-thread copy: chunk 5/62 (20971520-26214400) size 5Mi finished
2024/06/07 11:00:40 DEBUG : pippo3.tar: multi-thread copy: chunk 8/62 (36700160-41943040) size 5Mi starting
2024/06/07 11:01:06 INFO :
Transferred: 60 MiB / 309.307 MiB, 19%, 490.050 KiB/s, ETA 8m40s
Transferred: 0 / 1, 0%
Elapsed time: 1m0.0s
Transferring:
* pippo3.tar: 19% /309.307Mi, 478.682Ki/s, 8m53s
...........................
...........................

pippo3.tar:295% /309.307Mi, 99.701Ki/s, -

2024/06/07 11:17:21 DEBUG : pippo3.tar: multipart upload wrote chunk 55 with 5242880 bytes and etag "6952fd52be4940c9730c2ab988009110"
2024/06/07 11:17:21 DEBUG : pippo3.tar: multi-thread copy: chunk 55/62 (283115520-288358400) size 5Mi finished
2024/06/07 11:17:21 DEBUG : pippo3.tar: multipart upload "2~3T3t8SiAuo-3uCdLmuaLNo6SG89jVKW" finished
2024/06/07 11:17:21 DEBUG : pippo3.tar: Finished multi-thread copy with 62 parts of size 5Mi
2024/06/07 11:17:21 DEBUG : pippo3.tar: md5 = 5e3aca0406e0744e65903a166769903c OK
2024/06/07 11:17:21 INFO : pippo3.tar: Multi-thread Copied (new)
2024/06/07 11:17:21 INFO :
Transferred: 917.920 MiB / 917.920 MiB, 100%, 40.128 KiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 17m14.6s

So, 17 minutes to upload a 300Mbyte file, but with 917Mbyte transferred.
So, something is wrong in petasan 3.3.0 (or ceph 17.2.7)

 

Last edited on June 7, 2024, 12:09 pm by f.cuseo · #4

admin
2,930 Posts

June 7, 2024, 12:28 pm

Quote from admin on June 7, 2024, 12:28 pm

does it happen on all multi-part uploads or some ? can you test with small sizes ?

can you dis-able multi-part and use single part uploads ?

does it happen on all multi-part uploads or some ? can you test with small sizes ?

can you dis-able multi-part and use single part uploads ?

#5

f.cuseo
66 Posts

June 7, 2024, 12:37 pm

Quote from f.cuseo on June 7, 2024, 12:37 pm

No any problem with single part uploads (even big files), and with small multiparts the problem is present.

 

No any problem with single part uploads (even big files), and with small multiparts the problem is present.

 

#6

admin
2,930 Posts

June 8, 2024, 12:13 pm

Quote from admin on June 8, 2024, 12:13 pm

Could be related to

https://tracker.ceph.com/issues/59730

If you can configure use single part uploads from your client. You can also try adding extra nodes for s3 service running 3.21. If you engage us in support we can look into more detail to fixing the root issue.

Could be related to

https://tracker.ceph.com/issues/59730

If you can configure use single part uploads from your client. You can also try adding extra nodes for s3 service running 3.21. If you engage us in support we can look into more detail to fixing the root issue.

#7

f.cuseo
66 Posts

June 10, 2024, 7:00 am

Quote from f.cuseo on June 10, 2024, 7:00 am

Seems that this could help, but i don't know how to compile to have a compatible executable for petasan 3.3.0

-------------------

 

 

diff --git a/src/rgw/rgw_op.cc b/src/rgw/rgw_op.cc
index 3c4c59fc70c..f3f4ce859cb 100644
--- a/src/rgw/rgw_op.cc
+++ b/src/rgw/rgw_op.cc
@@ -6809,10 +6809,17 @@ void RGWCompleteMultipart::execute(optional_yield y)
     ldpp_dout(this, 0) << "WARNING: failed to remove object " << meta_obj << dendl;
   }

+  r = s->object->get_obj_attrs(s->yield, this);
+  if (r < 0) {
+    ldpp_dout(this, 0) << __func__ << "() ERROR: get_obj_attrs() returned ret=" << r << dendl;
+  }
+
+  etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
+
   // send request to notification manager
-  int ret = res->publish_commit(this, ofs, upload->get_mtime(), etag, target_obj->get_instance());
-  if (ret < 0) {
-    ldpp_dout(this, 1) << "ERROR: publishing notification failed, with error: " << ret << dendl;
+  r = res->publish_commit(this, ofs, upload->get_mtime(), etag, target_obj->get_instance());
+  if (r < 0) {
+    ldpp_dout(this, 1) << "ERROR: publishing notification failed, with error: " << r << dendl;
     // too late to rollback operation, hence op_ret is not set here
   }
 } // RGWCompleteMultipart::execute
@@ -6866,8 +6873,6 @@ void RGWCompleteMultipart::complete()
     }
   }

-  etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
-
   send_response();
 }

Seems that this could help, but i don't know how to compile to have a compatible executable for petasan 3.3.0

-------------------

 

 

diff --git a/src/rgw/rgw_op.cc b/src/rgw/rgw_op.cc
index 3c4c59fc70c..f3f4ce859cb 100644
--- a/src/rgw/rgw_op.cc
+++ b/src/rgw/rgw_op.cc
@@ -6809,10 +6809,17 @@ void RGWCompleteMultipart::execute(optional_yield y)
     ldpp_dout(this, 0) << "WARNING: failed to remove object " << meta_obj << dendl;
   }

+  r = s->object->get_obj_attrs(s->yield, this);
+  if (r < 0) {
+    ldpp_dout(this, 0) << __func__ << "() ERROR: get_obj_attrs() returned ret=" << r << dendl;
+  }
+
+  etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
+
   // send request to notification manager
-  int ret = res->publish_commit(this, ofs, upload->get_mtime(), etag, target_obj->get_instance());
-  if (ret < 0) {
-    ldpp_dout(this, 1) << "ERROR: publishing notification failed, with error: " << ret << dendl;
+  r = res->publish_commit(this, ofs, upload->get_mtime(), etag, target_obj->get_instance());
+  if (r < 0) {
+    ldpp_dout(this, 1) << "ERROR: publishing notification failed, with error: " << r << dendl;
     // too late to rollback operation, hence op_ret is not set here
   }
 } // RGWCompleteMultipart::execute
@@ -6866,8 +6873,6 @@ void RGWCompleteMultipart::complete()
     }
   }

-  etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();
-
   send_response();
 }

#8

f.cuseo
66 Posts

June 20, 2024, 4:52 pm

Quote from f.cuseo on June 20, 2024, 4:52 pm

I have temporary fixed the problem installing other 2 small nodes with 3.2.1 and only s3 service.

But, of course now:

• i can't use the big nodes (5 nodes for both storage/s3)
• i have less speed (because i can't have 4 x 10Gbit ports on theese small nodes) and memory/cpu
• i am using more space+energy
• i have less redundancy

So I can't wait to have s3 multipart upload fixed.
Have a good time and thanks for your work.

 

 

I have temporary fixed the problem installing other 2 small nodes with 3.2.1 and only s3 service.

But, of course now:

• i can't use the big nodes (5 nodes for both storage/s3)
• i have less speed (because i can't have 4 x 10Gbit ports on theese small nodes) and memory/cpu
• i am using more space+energy
• i have less redundancy

So I can't wait to have s3 multipart upload fixed.
Have a good time and thanks for your work.

 

 

#9

f.cuseo
66 Posts

August 29, 2024, 12:57 pm

Quote from f.cuseo on August 29, 2024, 12:57 pm

Hello.
Any plan to release the new version with a fix for S3 Multipart Upload ?
Thank you very much, Fabrizio

 

Hello.
Any plan to release the new version with a fix for S3 Multipart Upload ?
Thank you very much, Fabrizio

 

#10

Post Reply: Possible bug with multipart upload and S3

Cancel

Pages: 1 2