Comments (8)
gabrielefronze
commented on September 21, 2024
Can you please share your globals.ccp file content?
Also, I have tested everything with the next Rucio branch, so I am not sure enough about compatibility with previous versions.
I feel like all the issues you are observing are simply due to the first call failing.
Also, the answer code "200" tells you the request successfully found some files.
I think here the problem is with the server configuration...
from fuse-posix.
astroclark
commented on September 21, 2024
Version:
$ curl https://ligo-rucio.nautilus.optiputer.net/ping
{"version": "1.20.6"}
Config:
std::unordered_map<std::string, rucio_server> rucio_server_map = {
{
"rucio-server",
rucio_server("https://ligo-rucio.nautilus.optiputer.net",
"root",
"ligolab",
"XXX")
},
{
"rucio-server-clone",
rucio_server("https://ligo-rucio.nautilus.optiputer.net",
"root",
"ligolab",
"XXX")
}
};
Which corresponds to:
$ rucio-admin account list-identities root
Identity: ligolab, type: USERPASS
where client configs are happy with:
[client]
auth_host = https://ligo-rucio.nautilus.optiputer.net
auth_type = userpass
username = ligolab
password = XXX
from fuse-posix.
gabrielefronze
commented on September 21, 2024
First things first: the rucio-server-clone is there just as a concept of multiple servers support.
You can remove it without issues.
On my side I am using version 1.20.6.
I'll provide a version with debug output enabled to figure what are the raw REST calls and answers.
from fuse-posix.
gabrielefronze
commented on September 21, 2024
I think I have just found the reason why it fails. Have a look at the following curl verbose output:
--------------------------------------------------------------------------------------------------------
Retrieving scopes from server rucio-server-ligo
--------------------------------------------------------------------------------------------------------
* Found bundle for host ligo-rucio.nautilus.optiputer.net: 0x1136650
* Re-using existing connection! (#1) with host ligo-rucio.nautilus.optiputer.net
* Connected to ligo-rucio.nautilus.optiputer.net (XXX.XXX.XXX.XXX) port 443 (#1)
> GET //scopes/ HTTP/1.1
Host: ligo-rucio.nautilus.optiputer.net
Accept: */*
X-Rucio-Auth-Token: root-ligolab-unknown-###################
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Headers: None
< Access-Control-Allow-Methods: *
< Access-Control-Allow-Origin: None
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Cache-Control: post-check=0, pre-check=0
< Content-Type: text/plain; charset=UTF-8
< Date: Fri, 01 Nov 2019 20:58:33 GMT
< Pragma: no-cache
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
< X-Rucio-Host: ligo-rucio.nautilus.optiputer.net
< Content-Length: 19
<
* Connection #1 to host ligo-rucio.nautilus.optiputer.net left intact
Detected scopes:
ER8
O3
O2
* Found bundle for host ligo-rucio.nautilus.optiputer.net: 0x1136650
* Re-using existing connection! (#1) with host ligo-rucio.nautilus.optiputer.net
* Connected to ligo-rucio.nautilus.optiputer.net (XXX.XXX.XXX.XXX) port 443 (#1)
> GET //scopes/ HTTP/1.1
Host: ligo-rucio.nautilus.optiputer.net
Accept: */*
X-Rucio-Auth-Token: root-ligolab-unknown-###################
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
< Connection: close
<
* Closing connection 1
from fuse-posix.
gabrielefronze
commented on September 21, 2024
The issue is caused by reusing the already allocated CURL connections.
Working on a fix!
from fuse-posix.
gabrielefronze
commented on September 21, 2024
I managed to find the issue, and why it wasn't present on my local test.
It has to deal with the number of served files.
I expected the CURL response to be one line per DiD while, instead, multiple DiDs are in the same line. In addition each line can happen to have a DiD split with the following line, causing many issues.
It is clear from the following CURL response dump:
GET https://ligo-rucio.nautilus.optiputer.net/dids/ER8/H-H1_HOFT_C00/dids
line #1
/t{"adler32": "b0e8061a", "name": "H-H1_HOFT_C00-1126301696-4096.gwf", "bytes": 457826859, "scope": "ER8", "type": "FILE", "md5": "65c6
924477ef37bca15c5830ee56db8b"}
{"adler32": "91e498c8", "name": "H-H1_HOFT_C00-1126305792-4096.gwf", "bytes": 457659635, "scope": "ER8", "type": "FILE", "md5": "9843df
60bd67b2adb12363214182c354"}
{"adler32": "03810657", "name": "H-H1_HOFT_C00-1126309888-4096.gwf", "bytes": 458508855, "scope": "ER8", "type": "FILE", "md5": "22169a
da2b7b900922e8389de41e688e"}
{"adler32": "0494c933", "name": "H-H1_HOFT_C00-1126313984-4096.gwf", "bytes": 458985547, "scope": "ER8", "type": "FILE", "md5": "b7a210
65a751d5e1b3f6125d4a520b21"}
line #2
/t{"adler32": "7db31601", "name": "H-H1_HOFT_C00-1126318080-4096.gwf", "bytes": 457235619, "scope": "ER8", "type": "FILE", "md5": "f640
d5ed520ea3796009414186280100"}
{"adler32": "79dc80e1", "name": "H-H1_HOFT_C00-1126322176-4096.gwf", "bytes": 458288639, "scope": "ER8", "type": "FILE", "md5": "3768f0
63850d98b2591b956a7bfa20ca"}
{"adler32": "bcda9fc0", "name": "H-H1_HOFT_C00-1126326272-4096.gwf", "bytes": 457754791, "scope": "ER8", "type": "FILE", "md5": "e9346a
b9ef33318f04fa34a094400af5"}
{"adler32": "1ca00494", "name": "H-H1_HOFT_C00-1126330368-4096.gwf", "bytes": 458548431, "scope": "ER8", "type": "FILE", "md5": "1ee2e4
4a18d0a56da72016f1d8f90cea"}
{"adler32": "586f5268", "name": "H-H1_HOFT_C00-1126334464-4096.gwf", "bytes": 458566507, "scope": "ER8", "type": "FILE", "md5": "a9d325
ec42b8143f0c971d0fef15864b"}
{"adler32": "c4bc1eb1", "name": "H-H1_HOFT_C00-1126338560-4096.gwf", "bytes": 459318223, "scope": "ER8", "type": "FILE", "md5": "7e36da
244f6fd929079d3ea02a3804eb"}
{"adler32": "f18934e3", "name": "H-H1_HOFT_C00-1126342656-4096.gwf", "bytes": 458147391, "scope": "ER8", "type": "FILE", "md5": "02c640
2922c1cb304c0c9e34e2c004db"}
{"adler32": "f907738a", "name": "H-H1_HOFT_C00-1126346752-4096.gwf", "bytes": 458416223, "scope": "ER8", "type": "FILE", "md5": "8b50ad
882ea5813f2b8b17dccfba2d26"}
{"adler32": "0a6ec5ac", "name": "H-H1_HOFT_C00-1126350848-4096.gwf", "bytes": 45
line #3
/t8425803, "scope": "ER8", "type": "FILE", "md5": "f1d70565a33fc075feb62f4a5c2e3b9f"}
{"adler32": "b76bd851", "name": "H-H1_HOFT_C00-1126354944-4096.gwf", "bytes": 456326199, "scope": "ER8", "type": "FILE", "md5": "088509
c189b893dedf397eadf79cb980"}
{"adler32": "31823759", "name": "H-H1_HOFT_C00-1126359040-4096.gwf", "bytes": 456500631, "scope": "ER8", "type": "FILE", "md5": "85ba81
9887418314d72a481b814dbb74"}
{"adler32": "d0c2d29b", "name": "H-H1_HOFT_C00-1126363136-4096.gwf", "bytes": 456342619, "scope": "ER8", "type": "FILE", "md5": "839c64
e7f7dcc95b49f606ae39b4270a"}
{"adler32": "e285b76b", "name": "H-H1_HOFT_C00-1126367232-4096.gwf", "bytes": 430761851, "scope": "ER8", "type": "FILE", "md5": "056bdf
987ae42ff464e4c1edced747d5"}
{"adler32": "380f7cd9", "name": "H-H1_HOFT_C00-1126371328-4096.gwf", "bytes": 423337567, "scope": "ER8", "type": "FILE", "md5": "eb4b43
4013368fd1ded0482cec59f4cb"}
{"adler32": "60923312", "name": "H-H1_HOFT_C00-11263754
Working on a fix (finally knowing what the issue is :) )
from fuse-posix.
gabrielefronze
commented on September 21, 2024
The issue has been fixed:
[root@rucio-client ~]# ls /ruciofs/rucio-server-ligo/
ER8 O2 O3
[root@rucio-client ~]# ls /ruciofs/rucio-server-ligo/ER8/
H-H1_HOFT_C00
[root@rucio-client ~]# ls /ruciofs/rucio-server-ligo/ER8/
H-H1_HOFT_C00
[root@rucio-client ~]# ls /ruciofs/rucio-server-ligo/ER8/H-H1_HOFT_C00/
H-H1_HOFT_C00-1126301696-4096.gwf H-H1_HOFT_C00-1126354944-4096.gwf H-H1_HOFT_C00-1126408192-4096.gwf H-H1_HOFT_C00-1126461440-4096.gwf H-H1_HOFT_C00-1126514688-4096.gwf H-H1_HOFT_C00-1126567936-4096.gwf
H-H1_HOFT_C00-1126305792-4096.gwf H-H1_HOFT_C00-1126359040-4096.gwf H-H1_HOFT_C00-1126412288-4096.gwf H-H1_HOFT_C00-1126465536-4096.gwf H-H1_HOFT_C00-1126518784-4096.gwf H-H1_HOFT_C00-1126572032-4096.gwf
H-H1_HOFT_C00-1126309888-4096.gwf H-H1_HOFT_C00-1126363136-4096.gwf H-H1_HOFT_C00-1126416384-4096.gwf H-H1_HOFT_C00-1126469632-4096.gwf H-H1_HOFT_C00-1126522880-4096.gwf H-H1_HOFT_C00-1126576128-4096.gwf
H-H1_HOFT_C00-1126313984-4096.gwf H-H1_HOFT_C00-1126367232-4096.gwf H-H1_HOFT_C00-1126420480-4096.gwf H-H1_HOFT_C00-1126473728-4096.gwf H-H1_HOFT_C00-1126526976-4096.gwf H-H1_HOFT_C00-1126580224-4096.gwf
H-H1_HOFT_C00-1126318080-4096.gwf H-H1_HOFT_C00-1126371328-4096.gwf H-H1_HOFT_C00-1126424576-4096.gwf H-H1_HOFT_C00-1126477824-4096.gwf H-H1_HOFT_C00-1126531072-4096.gwf H-H1_HOFT_C00-1126584320-4096.gwf
H-H1_HOFT_C00-1126322176-4096.gwf H-H1_HOFT_C00-1126375424-4096.gwf H-H1_HOFT_C00-1126428672-4096.gwf H-H1_HOFT_C00-1126481920-4096.gwf H-H1_HOFT_C00-1126535168-4096.gwf H-H1_HOFT_C00-1126588416-4096.gwf
H-H1_HOFT_C00-1126326272-4096.gwf H-H1_HOFT_C00-1126379520-4096.gwf H-H1_HOFT_C00-1126432768-4096.gwf H-H1_HOFT_C00-1126486016-4096.gwf H-H1_HOFT_C00-1126539264-4096.gwf H-H1_HOFT_C00-1126592512-4096.gwf
H-H1_HOFT_C00-1126330368-4096.gwf H-H1_HOFT_C00-1126383616-4096.gwf H-H1_HOFT_C00-1126436864-4096.gwf H-H1_HOFT_C00-1126490112-4096.gwf H-H1_HOFT_C00-1126543360-4096.gwf H-H1_HOFT_C00-1126596608-4096.gwf
H-H1_HOFT_C00-1126334464-4096.gwf H-H1_HOFT_C00-1126387712-4096.gwf H-H1_HOFT_C00-1126440960-4096.gwf H-H1_HOFT_C00-1126494208-4096.gwf H-H1_HOFT_C00-1126547456-4096.gwf H-H1_HOFT_C00-1126600704-4096.gwf
H-H1_HOFT_C00-1126338560-4096.gwf H-H1_HOFT_C00-1126391808-4096.gwf H-H1_HOFT_C00-1126445056-4096.gwf H-H1_HOFT_C00-1126498304-4096.gwf H-H1_HOFT_C00-1126551552-4096.gwf H-H1_HOFT_C00-1126604800-4096.gwf
H-H1_HOFT_C00-1126342656-4096.gwf H-H1_HOFT_C00-1126395904-4096.gwf H-H1_HOFT_C00-1126449152-4096.gwf H-H1_HOFT_C00-1126502400-4096.gwf H-H1_HOFT_C00-1126555648-4096.gwf H-H1_HOFT_C00-1126608896-4096.gwf
H-H1_HOFT_C00-1126346752-4096.gwf H-H1_HOFT_C00-1126400000-4096.gwf H-H1_HOFT_C00-1126453248-4096.gwf H-H1_HOFT_C00-1126506496-4096.gwf H-H1_HOFT_C00-1126559744-4096.gwf H-H1_HOFT_C00-1126612992-4096.gwf
H-H1_HOFT_C00-1126350848-4096.gwf H-H1_HOFT_C00-1126404096-4096.gwf H-H1_HOFT_C00-1126457344-4096.gwf H-H1_HOFT_C00-1126510592-4096.gwf H-H1_HOFT_C00-1126563840-4096.gwf H-H1_HOFT_C00-1126617088-4096.gwf
The retrieving is quite slow, but we are working on that :)
from fuse-posix.
gabrielefronze
commented on September 21, 2024
Closing.
from fuse-posix.
Related Issues (20)
- Hide hidden files from FUSE handling HOT 1
- Convert and extend existing tests in unit tests HOT 2
- Make token expiration time eval consistent with UTC
- Avoid segfault if file not found/yet downloaded
- Implement file download to persistent cache
- Server config files should use same syntax as rucio ones
- Introduce multi server support when calling rucio download
- Notice wrong authentication parameters with pingable server
- Implement x509-based auth
- Check write permissions at startup
- Verify existence of Certification Authority key when parsing configs
- Remove possibility to cd to non existent directories
- cd to non existent directory still possible at server mountpoint and scopes levels
- Mark incoming files as downloading
- Add some more comments
- Retry download not working
- Mount point specification flag -f not parsed at runtime.
- Add script to setup FUSE mount after build. HOT 1
- Improve and Fix Documentation
- Adding a settings.json.template
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fuse-posix.