Chunk uploading and load balancing - how is it supposed to work about oneupuploaderbundle HOT 6 CLOSED

Hey,

You introduced gaufrette as a storage so I thought I would be able to store chunks on S3, but the only storage supported is local storage

The reason behind this is as you have also noted, streaming. It allows us to append files directly to the already existing ones. Otherwise you would end up with the chunks on your storage, with no way to combine them other than downloading them all and uploading the complete file.

So how is it different from previous approach since we use local stream storage for chunks?

The difference lies in the ability to use NFS mounts. Don't get me wrong, it was possible before too, but now with the Gaufrette layer. This might seem inconvenient but if ever one of the services will start providing streaming (in a more filesystem like way), and Gaufrette follows, it will allow everything to work correctly in the uploader too, without any changes other than configuration. Also it maintains the file abstraction through the whole bundle, ensuring that changes in your storage or servers don't imply any changes in your code, maybe not even in your configuration.

So with 3MB file with 1 MB chunks they will be sent: 1st to Server#1, 2nd to Server#2 and 3rd to Server#3 - how can Server#3 assemble the file when all chunks are distributed to other servers?

The way it is supposed to be set-up on 2 or more servers (it works for me):

1. You select one server and export a directory using NFS (preferably) or whatever else solution is capable
2. On every other server you mount that directory
3. On all of the servers you set up that directory as the filesystem for chunk storage

So what happens if our "master" goes down? This is more of an architectual thing, but you could mirror the exported directory on an other server and in your infrastructure management switch the IP from the unresponsive server to the mirrored one. This part really is out of the scope of the bundle. But high availability for nfs shares is possible, and that's all this bundle needs to continue working.

What happens after uploading a chunked file is, every server will write the files directly to the server selected at point 1. The first chunk will create the file, and every part afterwards will be immediately appended to it. Keep in mind that this won't work if the files are uploaded simultaneously as the order they are appended won't be the same, resulting in a corrupted file. After the last chunk is appended, the file will be moved to the destination specified in it's mapping configuration. It doesn't matter which server gets the last request (or the first, or any of them) as all of them will have equal access to the shared folder.

This still implies a delay of uploading the file on the last request, which could cause a time-out in php. This can be helped by setting up a mapping locally (for best performance the same gaufrette filesystem as the chunk storage) and setting it as the final storage for the files. This implies that the files from that directory can be served by your web servers. The file information (I assume) will be persisted in the database, including a way to reach it, which in our case should send the user to your actual servers. Then comes the S3 part.

It is a simple matter of a background task, which will either look up files in your database which are still on your server, or look up files directly on the server and get the database entries, then copy the files in the background to your S3 bucket, upgrade the database entries, and remove the files locally. This way your files were accessible uninterrupted from the upload right until they appeared on S3, and your application has a predictable behaviour for all sizes of files.

And how to send the file later to S3 when moving a file from stream to stream loads the whole file into memory (with bigger upload I can run out of memory for PHP).

The uploader will stream the file reading smaller parts of it, specified in sync_buffer_size to ensure you don't run out of memory. To be completely honest I had never tried S3, only hosting files on local storages, but this is the whole point of Gaufrette.

from oneupuploaderbundle.

Thanks for your detailed answer, this makes sense now. But I see now that we will probably switch to S3 Multipart Upload, it will be more fail-safe.

from oneupuploaderbundle.

Thanks for your detailed answer, this makes sense now. But I see now that we will probably switch to S3 Multipart Upload, it will be more fail-safe.

I don't think it can be any more fail safe than this. But, it's your call after all ^^

from oneupuploaderbundle.

@eXtreme have you implemented the chunks upload to S3? I got similar issue as you.

from oneupuploaderbundle.

@sela it was a long time ago :) As far as I remember I used some jquery uploading library to upload the file directly to S3 using S3 Multipart Upload (with progress bar and so on) and some custom backend code to handle moving that file to a destination bucket; without acutally using this bundle. I can't help more, I don't have the access to the code.

from oneupuploaderbundle.

Thanks for the help @eXtreme I assume you use POST signed url then. That's my plan B if the first one doesn't go well.

from oneupuploaderbundle.