failed when using jgi-query to download files wanted about jgi-query HOT 5 CLOSED

Hi Pengfei,

The error you are getting is not because you are running jgi-query incorrectly. Rather, it is due to the way JGI stores certain files in their database.

Sometimes (and I am not entirely clear on the details here), files stored in tape archives on JGI are not properly triggered for retrieval by cURL, and jgi-query simply gets an error XML file instead.

There is a (annoying) way to fix this; you can use the web interface to force JGI to activate the archive manually:

1. Open the file hierarchy via a web browser
2. Find the file you wish to download
3. Click on the file name via the browser; this should produce a message from JGI: "Checking tape transfer status"
4. Wait (possibly for quite a while)
5. Once the file download dialog box pops up in your browser, cancel it and retry your command via jgi-query and it should work.

You can tell which files are tape archives in the web interface from their legend icons:

There is probably a way to force the query to spin up the archive, but I haven't worked out what it is yet. For reference see this commit: 378f6bd

from jgi-query.

Hi Glarue,
Thanks.
Actually I want to download all .faa and .gbk file in this project. I just choose one file to test. For just one or few files it's fine to do in your proposed way, but for all is too much. I think it is also does not matter if I use jgi-query or the API directly in a loop, right, the same issue will occur?

Otherwise I may contact with our IT to make the globus working.

Thanks again.

Cheers,
Pengfei

from jgi-query.

@liupfskygre unfortunately yes, if you are downloading a large number of files (the majority of which are on tape archives) then using jgi-query may prove difficult. It might be worth manually checking a random subset of the files you want via the web interface to see if they are mostly in tape archives or not (as I believe it is based on file age, not category).

And yes, I believe you are correct that using the API in a loop won't fix things, since all jgi-query is doing is performing queries automatically using the API backend.

If you find a solution outside of globus, please let me know! Closing for now.

from jgi-query.

@liupfskygre my recent commit—moving queries from http to https—may fix the issue you were having and allow you to download tape file archives.

from jgi-query.

@liupfskygre just another small update: there is now an option in the file selection dialog to use a regular expression pattern to match an arbitrarily large number of files, should that prove helpful.

from jgi-query.