Would love to upload entire directories w/out first having to zip them up about sagu HOT 6 CLOSED

Looks like FastGlacier just walks the directory tree, uploading individual files. Good enough, except for their stupid throttling limit on the free client of two files in parallel.

from sagu.

It's not impossible to do that, but consider that Glacier is really designed for storage in case of catastrophic loss. You're going to be retrieving every one of those files individually and waiting 4 hours to get them. Also, files are not stored on Glacier servers by directory, so if you do need to restore a folder that includes a bunch of files you're going to have a bad time.

from sagu.

This is at least the second time on this forum when a fairly obvious usability issue has been ignored as unimportant.

Glacier obviously thinks in terms of a file webservice. Almost all other clients support upload by directory and have some kind of mapping scheme to make that work. Doing that makes the usability of the client much better. At least in some cases, as Johnl_4 mentions it is simply not possible to merge directories into a single file.

Generally, we say software is "simple" when it does what it advertises to do and otherwise gets out of the way. Simple for the user typically means "complicated" for the programmer. Right now you have a product that is "simple to program, complicated to use"

I understand that this might be a side project and all. I have no expectation that you would labor for me for nothing. You have absolutely no obligation to improve or modify your project based on user feedback. But you are coming on these forums and telling your users (who are telling you how to fix your product) how they are wrong, or how what they want is impossible.

If you don't want to do a feature or you don't have time... say that. If you do not know how to implement something, say that. If you are working on it.. say that. But you might consider not pretending that the features that are being asked for are unimportant...

That is fairly frustrating.

-FT

from sagu.

Hi ftrotter. There are some pretty severe limitations enforced by the Glacier service which makes it unsuitable to be a widely used consumer service directly. Glacier doesn't store your files in any sort of recognizable format and requires that you request each one individually. In order for this to approach any sort of consumer-level use you're going to need to find someone willing to maintain a database infrastructure independently of the glacier service that will maintain all of this metadata. I'm not willing to do that, I'm also not willing to take the liability of modifying files for people prior to upload.

I've put together a program that will let you upload and download files to a service that is designed to be used by data centers. I've also provided the code free of charge and under a permissive license so that anyone who thinks they can do it better is free to do so. Fork button is at the top right.

from sagu.

I'll add here for anyone considering a fork that the decision to not allow recursive directory uploads has been a deliberate one on my part due to the current limitations of SAGU.
Following are technical considerations for anyone looking to add this functionality to their fork.

-Currently the program only allows for deleting and downloading a single archive at a time.
-The AWS Glacier service does not allow a vault to be deleted unless all archives have first been deleted from that vault.
-It is therefore very important to understand that if someone, for example, decides to upload their entire hard drive to a glacier vault, they are going to need to individually delete every one of those files before they can delete the vault.

Prior to implementation of a method to upload directory trees I suggest that SAGU implement:

-A "mass delete" function to delete all files from a vault. The easiest way to do this would probably be to accept a downloaded inventory file and then submit the delete command for each file in the vault. Only then can the vault itself be deleted.
-A better download function. Something that will take the current log and/or a downloaded vault inventory and provide a better view of the data.
-A problem with glacier from the outset has been the fact the AWS does not store file names. Implementations of SAGU have included the user's local file path and name in the archive description field, but the names may not always be complete as AWS enforces strict character limitations which require regex cleaning of special characters. Filenames either have to be re-typed manually or retrieved from the local logs.

Until these problems are resolved, expect that any forked implementation of SAGU that allows users to massively upload files in the manner described will cause much grievance when users attempt to retrieve those files or delete their archives to prevent AWS fees.

I understand that other programs already implemented the ability to upload directories in the manner requested, but I'm not sure how those programs propose retrieval and deletion of user files should the user's local logs be compromised. Note that this software is considered experimental and includes no warranty of merchantability. I am not going to deliberately introduce a feature that, in my opinion, has the potential to cause more harm than good. Allowing directory uploads would be a simple implementation, but I have no desire to support end users who upload massive amounts of files without understanding the consequences.

from sagu.

Your second post is a much better description of the development issues at play, which makes me, as a user feel that I am not getting blown off.

Your point is, essentially, that the methods of "backing off from" glacier backup need to be present before this kind of mass upload is prudent, and that makes sense. I would add that a feature to "download slowly and stay below %5 per month" would also be a welcome safety feature. Your feature list makes sense. If you will create specific issues for each of the prerequisite features you list, I will see if I can get some time from a Java-fluent friend with the goal of submitting a pull request.

Your argument does beg the question, however: If the only safe way to use your project is to make a gigantic zip/tar file and then use that, why not build support for that functionality into the client itself? As it stands, users can still achieve this problematic state of lots of files in the archive, its just difficult for them to reach. There nothing in your software that moves users toward "the safe path of really big zip files" rather than the apparently dangerous "lots of little files path".

As for liability, everyone reading this and using this software should understand that the licenses in question are clear: brianmcmichael has no legal liability for failure in this software. Buyer beware and testing is your friend. So when you say "I am not going to take the liability"... there was none for you to take, legally speaking. What you really mean is something like "Mucking about with important files is risky, and I would rather have the user take that risk". Of course, the problem with this is that many of your potential users have no idea how to handle that risk. It would be difficult to implement this feature well (as you mention) but that is the only way to actually eliminate the risk involved, rather than just passing the problem on to the users...

Open Source is kind of rough in these kinds of situations. I and others criticize harshly when we are frustrated. But there is also an implicit respect here. We are bringing our compliants to you, and nothing else beyond that makes you the "leader" of the project. The issues that we are bringing up are fairly obvious components of "how this should work". Your reaction to them how you decide if you would prefer to be the "go to" project in this space. I do not have the Java skills to fork, you, but I need a cross-platform reliable "simple to use" Glacier uploader. I and others in the same boat will support whichever project makes it obvious that it is stepping in that direction.

-ft

from sagu.