Rucio server always uses UTC and the fuse module must do the same.
At the moment too much use of local timezones in the logs and internal operations is causing issues when moving from one timezone to another or connecting through multiple timezones.

Moving to UTC will make everything universal, the logs as well.

This simply makes sense.
All the required fields are contained in the standard format, no need to use a json alternative.

When a file starts being downloaded it's inode should be marked as incoming, or an error message should be triggered to inform the opener the file is getting ready.
Alternatively the download should be blocking.

In branch use-rucio-cli, if the call through bash to rucio download fails, the fuse module breaks.
It might be enough to touch the file and just return nothing if the file is not found.
Even if the call trough bash won't be there in the production environment, this check will remain the same and will make the algorithm widely more reliable.

Motivation

The documentation for this software needs improvements for better collaboration and improving code readability. The required section should be created with a CONTRIBUTING and SET UP guide, along with proper formatting of existing README.

This feature allows for multiple server connection even when calling rucio download through a bash session via the --config parameter.
Necessary to promote the download through bash to a production state.

Motivation

The build script handles the compilation of the program but does not set up FUSE in the way it can be used for mounting the Rucio server(s) to a directory using the FUSE mount. A post-build script is thus required to automate the process of setting up FUSE and mounting the server using the same.

Expected Result

vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ sudo ./postbuild.sh
Setting up FUSE for ruciofs.........
Done.

vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ ./cmake-build-debug/bin/rucio-fuse-main

Settings file at: ./settings.json
Parsing settings file:

	Server 0 -> rucio-dev-server:
		url = https://localhost/
		account = root

Actual Result

Manually set up FUSE. Add the user to the fuse group and set the right permissions.

Motivation

Some tests are already implemented, but they are not part of a test framework.
Reimplementing and extending them with ctest or google test is crucial for CI and CD.

Modification

Add units tests written with ctest or Google Test for Rucio-FUSE

Motivation

Both on macOS and Ubuntu (and many other OSs), hidden files are the current implementation for recycle bin and many other features.
We must avoid Rucio FUSE mount from trying to handle them since they result in unsuccessful server calls.

Expected result

Rucio FUSE should avoid handling of .hidden files

Actual Result

Example:
Enviroment: macOS

$ ./cmake-build-debug/bin/rucio-fuse-main

Settings file at: ./settings.json
Parsing settings file:
curl_easy_perform() failed: Couldn't resolve host name

Server https://rucio-server not reachable, skipping connection.
curl_easy_perform() failed: Couldn't resolve host name

Server https://rucio-server not reachable, skipping connection.
Server DCIM not found. Aborting!
Server .Spotlight-V100 not found. Aborting!
Server .Spotlight-V100 not found. Aborting!
Server .Spotlight-V100 not found. Aborting!
Server .Spotlight-V100 not found. Aborting!
Server .Spotlight-V100 not found. Aborting!
Server .metadata_never_index not found. Aborting!
Server .hidden not found. Aborting!
Server .localized not found. Aborting!
Server .DS_Store not found. Aborting!
Server ._.DS_Store not found. Aborting!

Environment: Ubuntu

$ ./cmake-build-debug/bin/rucio-fuse-main -f ~/ruciofs

Settings file at: ./settings.json
Parsing settings file:
curl_easy_perform() failed: Couldn't resolve host name
Server https://rucio-server not reachable, skipping connection.
curl_easy_perform() failed: Couldn't resolve host name
Server https://rucio-server not reachable, skipping connection.
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .xdg-volume-info not found. Aborting!
Server autorun.inf not found. Aborting!
Server .hidden not found. Aborting!

Modification

It should be enough to return if the last item of the path starts by ..

Motivation

The settings.json file provides the fixed key-value pairs which help the program to fetch the details of the rucio-server. However, this may be changed accordingly depending on the dev environment or user requirements. So a settings.json.template is required which can be modified according to the user/developer's need.

Expected Result

A settings.json.template based on the current settings.json file.

After running the rucio-fuse-main executable, I try to list the /ruciofs/rucio-server directory:

# ls /ruciofs/rucio-server
00BadReques

If i list the contents beyond rucio-server/ everything crashes:

# ls /ruciofs/rucio-server/00BadReques
ls: reading directory /ruciofs/rucio-server/00BadReques: Software caused connection abort

Sometimes, if i try the rucio-server-clone directory, i can actually see the scopes from my server's namespace, though:

# ls /ruciofs/rucio-server-clone
ls: cannot access /ruciofs/rucio-server-clone/ER8: Software caused connection abort
ls: cannot access /ruciofs/rucio-server-clone/O3: Transport endpoint is not connected
ls: cannot access /ruciofs/rucio-server-clone/O2: Transport endpoint is not connected
ls: reading directory /ruciofs/rucio-server-clone: Transport endpoint is not connected
ER8  O2  O3

The log on my rucio server shows:

10.244.11.1	[31/Oct/2019:21:47:53 +0000]	-	1	1770247	"root-ligolab-unknown-3214775a91484c389a9634e4cf77f681"	XbtWiY7nZXm5q7NqKYYBpQAAAAI	-	"GET /scopes/ HTTP/1.1"	200	19
10.244.11.1	[31/Oct/2019:21:47:56 +0000]	-	0	118925	"root-ligolab-unknown-3214775a91484c389a9634e4cf77f681"	XbtWjI7nZXm5q7NqKYYBpgAAAAI	-	"GET /dids/00BadReques/ HTTP/1.1"	200	-

All of these actions eventually result in the whole thing crashing:

docker run -it  --device /dev/fuse --cap-add SYS_ADMIN jclarkastro/fuse-posix
----------------------------------------
Activating rucio fuse-posix interface...
Creating CURL instance
/docker-entrypoint.sh: line 4:     6 Segmentation fault      (core dumped) /fuse-posix/cmake-build-debug/bin/rucio-fuse-main

Motivation

The FUSE mount does not directly work after executing ./build.sh and needs some additional steps to mount the server to a directory. This issue is opened to improve the documentation and add the post-build actions to successfully mount the server.

The FUSE module does not provide yet the ability to download files via rucio CLI.
The implementation in use-rucio-cli branch does that, but the downloaded files are not stored locally across reboots of the system.
The downloads should happen towards a separate cache location. The FUSE representation would rely on this separate persistent cache location to provide the actual file contents via a mechanism similar to a symlink directly handled by FUSE.

To avoid issues at runtime it might be worth checking write permissions in /ruciofs and /rucio-cache paths at startup.

Motivation

The documentation mentions mounting the server to the desired location overriding the default /ruciofs directory by adding the -f option while running the script. However, this does not work at the time of execution.

Environment: Ubuntu 18.04.4 LTS (bionic)

Steps to reproduce

vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ mkdir ~/ruciofs
vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ ./cmake-build-debug/bin/rucio-fuse-main -f ~/ruciofs

Expected Result

vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ mkdir ~/ruciofs
vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ ./cmake-build-debug/bin/rucio-fuse-main -f ~/ruciofs

Settings file at: ./settings.json
Parsing settings file:

	Server 0 -> rucio-dev-server:
		url = https://<rucio-server-url>/
		account = <your_account>
		username = <username>
		password = <password>
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!
Server .Trash-1000 not found. Aborting!

Actual Result

vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ mkdir ~/ruciofs
vivek@ubuntu:~/Desktop/GSoC_2020/fuse-posix$ ./cmake-build-debug/bin/rucio-fuse-main -f ~/ruciofs

 Settings file at: ./settings.json
Parsing settings file:	Server 0 -> rucio-dev-server:
		url = https://<rucio-server-url>/
		account = <your_account>
		username = <username>
		password = <password>
fuse: bad mount point `/ruciofs': No such file or directory

Not a lot more to add...

At startup all the servers are pinged, but the authentication is not tested.
Later on, at runtime, this might cause bizarre issues when trying to contact the server, leading to error messages rendered as folders and files.
The server credentials must be tested at startup, disabling servers impossible to ping AND authenticate to.

Not a whole lot more to add..

cd to non existent directory is still possible at server and scopes level.

[root@ruciofs]/ruciofs# ls
test-userpass  test-x509
[root@ruciofs]/ruciofs# cd this-server-doesnt-exist
[root@ruciofs]/ruciofs/this-server-doesnt-exist# ls
[root@ruciofs]/ruciofs/this-server-doesnt-exist#

[root@ruciofs]/ruciofs/test-userpass# ls
NoProjectDefined  archive  data13_hip  mock  test
[root@ruciofs]/ruciofs/test-userpass# cd this-scope-doesnt-exist
[root@ruciofs]/ruciofs/test-userpass/this-scope-doesnt-exist#

[root@ruciofs]/ruciofs/test-userpass# cd test
[root@ruciofs]/ruciofs/test-userpass/test# cd this-file-doesnt-exist
cd: Input/output error: this-file-doesnt-exist

[root@ruciofs]/ruciofs/test-userpass/test/container# cd this-file-doesnt-exist
cd: Input/output error: this-file-doesnt-exist

The code is barely commented and some more detailing is definitely needed.

Apparently there's an error within the code wrapping rucio download, which was previously commented to avoid issues.
If a download fails it should be retried by the pipeline:

The idea is nice, but the issue is with the if condition. In fact testing:

output->freturn_code != MAX_ATTEMPTS

should be:

output->freturn_code != TOO_MANY_ATTEMPTS

or:

output->fattempt > MAX_ATTEMPTS

based on the definition here:

This issue ends up causing multiple segfaults and multiple failures.
It also prevents Mac compatibility.