wyllys66 / s3multi Goto Github PK

Does s3multi run on a single machine and have run multiple-threads to upload multiple parts concurrently?

Can I use multiple machines to launch s3multi and upload a very big object?
Example:
Single object size: 100TB
Multipart portioning : each part is 5GB, 20,000 parts
use 100 machines to upload data: 200 parts / machines
each machine launch 20 concurrent sessions of s3multi to upload data

Thanks.

HB

Hi,

After I install "s3multi" with "swift" in OpenStack, it works for starting the multipart upload, and can upload all data segments to the "_segments" container. However, when I call "completeMultipartUpload", it failed with a strange error "NoSuchBucket" from the respond XML message, and from /var/log/message, there is a message like:

Jan 15 10:19:13 localhost proxy-server ERROR WSGI: code 400, message Bad request syntax ('1157832f52f26c4d8f533d617fba200302afa936a26cab1526de176445368b2639DELETE /CloudBacko/CloudTest2%252FLargerThan5M%252Fapache-tomcat-6.0.36-windows-x86.zip.7674a0.13c83f3367e.cgz?uploadId=7457a81c11a6d2c3b080698754e3e140 HTTP/1.1') (txn: tx103c97d3662f4776acafe-0052d5f021) (client_ip: 192.168.8.174)

(I have just chosen a file with size > 5M, and separate it into 2 segments in my test)

Any idea?

Thanks a lot!

MP

Hi,

I was wondering why aws s3 multipart is implemented as a separate middleware,
is there any reason for that ? (I'm relatively new to the project so may be I missed some discussions ?)

What would be the problem of merging this work into swift3 middleware and thus gain more visibility ? (or at least may be have it as a stackforge project ?)

I will be happy to help for that by the way.
Thanks in advance.

Nassim
(IRC: nacim)