rucio / fuse-posix Goto Github PK
View Code? Open in Web Editor NEWPOSIX interface to Rucio namespace
POSIX interface to Rucio namespace
The documentation mentions mounting the server to the desired location overriding the default /ruciofs
directory by adding the -f
option while running the script. However, this does not work at the time of execution.
Environment: Ubuntu 18.04.4 LTS (bionic)
vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ mkdir ~/ruciofs
vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ ./cmake-build-debug/bin/rucio-fuse-main -f ~/ruciofs
vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ mkdir ~/ruciofs
vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ ./cmake-build-debug/bin/rucio-fuse-main -f ~/ruciofs
Settings file at: ./settings.json
Parsing settings file:
Server 0 -> rucio-dev-server:
url = https://<rucio-server-url>/
account = <your_account>
username = <username>
password = <password>
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ mkdir ~/ruciofs
vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ ./cmake-build-debug/bin/rucio-fuse-main -f ~/ruciofs
Settings file at: ./settings.json
Parsing settings file: Server 0 -> rucio-dev-server:
url = https://<rucio-server-url>/
account = <your_account>
username = <username>
password = <password>
fuse: bad mount point `/ruciofs': No such file or directory
Some tests are already implemented, but they are not part of a test framework.
Reimplementing and extending them with ctest or google test is crucial for CI and CD.
Add units tests written with ctest or Google Test for Rucio-FUSE
Not a lot more to add...
When a file starts being downloaded it's inode should be marked as incoming, or an error message should be triggered to inform the opener the file is getting ready.
Alternatively the download should be blocking.
This simply makes sense.
All the required fields are contained in the standard format, no need to use a json alternative.
Rucio server always uses UTC and the fuse module must do the same.
At the moment too much use of local timezones in the logs and internal operations is causing issues when moving from one timezone to another or connecting through multiple timezones.
Moving to UTC will make everything universal, the logs as well.
The FUSE module does not provide yet the ability to download files via rucio CLI.
The implementation in use-rucio-cli
branch does that, but the downloaded files are not stored locally across reboots of the system.
The downloads should happen towards a separate cache location. The FUSE representation would rely on this separate persistent cache location to provide the actual file contents via a mechanism similar to a symlink directly handled by FUSE.
Apparently there's an error within the code wrapping rucio download
, which was previously commented to avoid issues.
If a download fails it should be retried by the pipeline:
fuse-posix/include/download-pipeline.h
Lines 50 to 52 in 2fbe45e
The idea is nice, but the issue is with the if condition. In fact testing:
output->freturn_code != MAX_ATTEMPTS
should be:
output->freturn_code != TOO_MANY_ATTEMPTS
or:
output->fattempt > MAX_ATTEMPTS
based on the definition here:
fuse-posix/include/rucio-download.h
Lines 123 to 140 in 2fbe45e
In branch use-rucio-cli
, if the call through bash to rucio download
fails, the fuse module breaks.
It might be enough to touch the file and just return nothing if the file is not found.
Even if the call trough bash won't be there in the production environment, this check will remain the same and will make the algorithm widely more reliable.
This issue ends up causing multiple segfaults and multiple failures.
It also prevents Mac compatibility.
Both on macOS and Ubuntu (and many other OSs), hidden files are the current implementation for recycle bin and many other features.
We must avoid Rucio FUSE mount from trying to handle them since they result in unsuccessful server calls.
Rucio FUSE should avoid handling of .hidden
files
Example:
Enviroment: macOS
$ ./cmake-build-debug/bin/rucio-fuse-main
Settings file at: ./settings.json
Parsing settings file:
curl_easy_perform() failed: Couldn't resolve host name
Server https://rucio-server not reachable, skipping connection.
curl_easy_perform() failed: Couldn't resolve host name
Server https://rucio-server not reachable, skipping connection.
Server DCIM not found. Aborting!
Server .Spotlight-V100 not found. Aborting!
Server .Spotlight-V100 not found. Aborting!
Server .Spotlight-V100 not found. Aborting!
Server .Spotlight-V100 not found. Aborting!
Server .Spotlight-V100 not found. Aborting!
Server .metadata_never_index not found. Aborting!
Server .hidden not found. Aborting!
Server .localized not found. Aborting!
Server .DS_Store not found. Aborting!
Server ._.DS_Store not found. Aborting!
Environment: Ubuntu
$ ./cmake-build-debug/bin/rucio-fuse-main -f ~/ruciofs
Settings file at: ./settings.json
Parsing settings file:
curl_easy_perform() failed: Couldn't resolve host name
Server https://rucio-server not reachable, skipping connection.
curl_easy_perform() failed: Couldn't resolve host name
Server https://rucio-server not reachable, skipping connection.
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .xdg-volume-info not found. Aborting!
Server autorun.inf not found. Aborting!
Server .hidden not found. Aborting!
It should be enough to return if the last item of the path starts by .
.
After running the rucio-fuse-main
executable, I try to list the /ruciofs/rucio-server
directory:
# ls /ruciofs/rucio-server
00BadReques
If i list the contents beyond rucio-server/
everything crashes:
# ls /ruciofs/rucio-server/00BadReques
ls: reading directory /ruciofs/rucio-server/00BadReques: Software caused connection abort
Sometimes, if i try the rucio-server-clone
directory, i can actually see the scopes from my server's namespace, though:
# ls /ruciofs/rucio-server-clone
ls: cannot access /ruciofs/rucio-server-clone/ER8: Software caused connection abort
ls: cannot access /ruciofs/rucio-server-clone/O3: Transport endpoint is not connected
ls: cannot access /ruciofs/rucio-server-clone/O2: Transport endpoint is not connected
ls: reading directory /ruciofs/rucio-server-clone: Transport endpoint is not connected
ER8 O2 O3
The log on my rucio server shows:
10.244.11.1 [31/Oct/2019:21:47:53 +0000] - 1 1770247 "root-ligolab-unknown-3214775a91484c389a9634e4cf77f681" XbtWiY7nZXm5q7NqKYYBpQAAAAI - "GET /scopes/ HTTP/1.1" 200 19
10.244.11.1 [31/Oct/2019:21:47:56 +0000] - 0 118925 "root-ligolab-unknown-3214775a91484c389a9634e4cf77f681" XbtWjI7nZXm5q7NqKYYBpgAAAAI - "GET /dids/00BadReques/ HTTP/1.1" 200 -
All of these actions eventually result in the whole thing crashing:
docker run -it --device /dev/fuse --cap-add SYS_ADMIN jclarkastro/fuse-posix
----------------------------------------
Activating rucio fuse-posix interface...
Creating CURL instance
/docker-entrypoint.sh: line 4: 6 Segmentation fault (core dumped) /fuse-posix/cmake-build-debug/bin/rucio-fuse-main
cd
to non existent directory is still possible at server and scopes level.
[root@ruciofs]/ruciofs# ls
test-userpass test-x509
[root@ruciofs]/ruciofs# cd this-server-doesnt-exist
[root@ruciofs]/ruciofs/this-server-doesnt-exist# ls
[root@ruciofs]/ruciofs/this-server-doesnt-exist#
[root@ruciofs]/ruciofs/test-userpass# ls
NoProjectDefined archive data13_hip mock test
[root@ruciofs]/ruciofs/test-userpass# cd this-scope-doesnt-exist
[root@ruciofs]/ruciofs/test-userpass/this-scope-doesnt-exist#
[root@ruciofs]/ruciofs/test-userpass# cd test
[root@ruciofs]/ruciofs/test-userpass/test# cd this-file-doesnt-exist
cd: Input/output error: this-file-doesnt-exist
[root@ruciofs]/ruciofs/test-userpass/test/container# cd this-file-doesnt-exist
cd: Input/output error: this-file-doesnt-exist
The code is barely commented and some more detailing is definitely needed.
At startup all the servers are pinged, but the authentication is not tested.
Later on, at runtime, this might cause bizarre issues when trying to contact the server, leading to error messages rendered as folders and files.
The server credentials must be tested at startup, disabling servers impossible to ping AND authenticate to.
This feature allows for multiple server connection even when calling rucio download
through a bash session via the --config
parameter.
Necessary to promote the download through bash to a production state.
The documentation for this software needs improvements for better collaboration and improving code readability. The required section should be created with a CONTRIBUTING and SET UP guide, along with proper formatting of existing README.
The build script handles the compilation of the program but does not set up FUSE in the way it can be used for mounting the Rucio server(s) to a directory using the FUSE mount. A post-build script is thus required to automate the process of setting up FUSE and mounting the server using the same.
vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ sudo ./postbuild.sh
Setting up FUSE for ruciofs.........
Done.
vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ ./cmake-build-debug/bin/rucio-fuse-main
Settings file at: ./settings.json
Parsing settings file:
Server 0 -> rucio-dev-server:
url = https://localhost/
account = root
Manually set up FUSE. Add the user to the fuse
group and set the right permissions.
The FUSE mount does not directly work after executing ./build.sh
and needs some additional steps to mount the server to a directory. This issue is opened to improve the documentation and add the post-build actions to successfully mount the server.
To avoid issues at runtime it might be worth checking write permissions in /ruciofs
and /rucio-cache
paths at startup.
Not a whole lot more to add..
The settings.json
file provides the fixed key-value pairs which help the program to fetch the details of the rucio-server
. However, this may be changed accordingly depending on the dev environment or user requirements. So a settings.json.template
is required which can be modified according to the user/developer's need.
A settings.json.template
based on the current settings.json
file.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.