Support recursive download about s3-plugin HOT 4 CLOSED

That makes sense. It used to be one had to list the S3 bucket and then issue batches of download requests which was a pain. Now it looks like TransferManager.downloadDirectory makes that much easier and more efficient.

I'd add keyPrefix and destDir properties to S3Download to indicate a recursive download is desired. Ideally there should also be a timeout property to abort download transfers that take too long. Finally we should add a progress listener to report progress.

I don't need this functionality myself currently, but it makes sense to have so I may give it a go. In the meantime I'd welcome a pull request that used the transfer manager as above.

from s3-plugin.

Progress is underway in my little fork. I have basic downloading implemented, but still print statements in place of real use of the ProgressListener, no timeout, and I didn't figure out how to alternately require key/file or keyPrefix/destDir rather than all four.

from s3-plugin.

Also, I'm new to groovy and doing this a bit hastily, so the code quality could probably be cleaned up.

from s3-plugin.

@NickMcAvoy Thanks for getting the ball rolling on this. I implemented recursive downloads as described above except for the timeout property which is directly supported on single file downloads but looks like it requires some additional state management for multi-file downloads.

I've tested it with a handful of files. Let me know how it works for you.

from s3-plugin.