cscfi / sda-filesystem Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 1.0 2.5 MB

FUSE implementation for working with files in SD Desktop https://research.csc.fi/sensitive-data

License: MIT License

Go 84.47% NSIS 3.14% HTML 0.11% Vue 9.01% TypeScript 0.58% CSS 0.14% JavaScript 2.55%

sda-filesystem's Introduction

SDA-Filesystem / Data Gateway

This project has been rebranded as Data Gateway

Data Gateway makes use of the:

It builds a FUSE (Filesystem in Userspace) layer and uses Airlock to export files to SD Connect. Software currently supports Linux, macOS and Windows for:

Graphical User Interface
Command Line Interface

Binaries are built on each release for all supported Operating Systems.

Requirements

Go version 1.21

Set these environment variables before running the application:

FS_SD_CONNECT_API - API for SD-Connect
FS_SD_SUBMIT_API – a comma-separated list of APIs for SD Apply/SD Submit
SDS_ACCESS_TOKEN - a JWT for authenticating to the SD APIs
FS_CERTS - path to a file that contains certificates required by SD Connect, SD Apply/SD Submit, and SDS AAI

Optional envronment variables:

CSC_USERNAME - username for SDA-Filesystem
CSC_PASSWORD - password for SDA-Filesystem and Airlock CLI

For test environment follow instructions at https://gitlab.ci.csc.fi/sds-dev/local-proxy

Graphical User Interface

Dependencies

cgofuse and its dependencies on different operating systems.

Install Wails and its dependencies.

Install pnpm

Build and Run

Before running/building the repository for the first time, generate the frontend assests by running:

pnpm install --prefix frontend
pnpm --prefix frontend run build

To run in development mode:

cd cmd/gui
wails dev

To build for production:

cd cmd/gui

# For Linux and macOS
wails build -upx -trimpath -clean -s

# For Windows
wails build -upx -trimpath -clean -s -webview2=embed

Deploy

See Linux setup.

Command Line Interface

Two command line binaries are released, one for SDA-Filesystem and one for Airlock.

SDA-Fileystem

The CLI binary will require a username and password for accessing the SD-Connect Proxy API. Username is given as input. Password is either given as input or in an environmental variable.

Build and Run

go build -o ./go-fuse ./cmd/fuse/main.go

Test install.

./go-fuse -help                        
Usage of ./go-fuse:
  -alsologtostderr
    	log to standard error as well as files
  -http_timeout int
    	Number of seconds to wait before timing out an HTTP request (default 20)
  -log_backtrace_at value
    	when logging hits line file:N, emit a stack trace
  -log_dir string
    	If non-empty, write log files in this directory
  -loglevel string
    	Logging level. Possible values: {debug,info,warning,error} (default "info")
  -logtostderr
    	log to standard error instead of files
  -mount string
    	Path to Data Gateway mount point
  -project string
    	SD Connect project if it differs from that in the VM
  -sdapply
      Connect only to SD Apply
  -stderrthreshold value
    	logs at or above this threshold go to stderr
  -v value
    	log level for V logs
  -vmodule value
    	comma-separated list of pattern=N settings for file-filtered logging

Example run: ./go-fuse -mount=$HOME/ExampleMount will create the FUSE layer in the directory $HOME/ExampleMount for both 'SD Connect' and 'SD Apply'.

User input

User can update the filesystem by inputting the command update. This requires that no files inside the filesystem are being used. Update also clears cache. As a result of this operation, new files may be added and some old ones removed.

The filesystem can be also updated programatically with the SIGUSR2 signal.

To update filesystem on bash in SD Desktop:

# Update CLI version
kill -s SIGUSR2 $(pgrep go-fuse)

# Update GUI version
kill -s SIGUSR2 $(pgrep sda-fuse)

If the user wants to update particular SD Connect files inside the filesystem, the user can input command clear <path>. <path> is the path to the file/folder that the user wishes to update. <path> must at least contain a bucket, i.e. SD-Connect/project/bucket or SD-Connect/project/bucket/file would be acceptable paths, but not, e.g., SD-Connect/project. If the user gives a path to a folder, all files inside this folder are updated but no files are added or removed. This operation clears the cache for all the neccessary files so that the new content is read from the database and sizes of these files are updated in the filesystem.

Airlock

The CLI binary will require a username, a bucket and a filename. Password is either given as input or in an environmental variable.

Build and Run

go build -o ./airlock ./cmd/airlock/main.go

Test install.

./airlock -help
Usage of ./airlock:
  -alsologtostderr
    	log to standard error as well as files
  -debug
    	Enable debug prints
  -journal-number string
    	Journal Number/Name specific for Findata uploads
  -log_backtrace_at value
    	when logging hits line file:N, emit a stack trace
  -log_dir string
    	If non-empty, write log files in this directory
  -logtostderr
    	log to standard error instead of files
  -original-file string
    	Filename of original unecrypted file when uploading pre-encrypted file from Findata vm
  -project string
    	SD Connect project if it differs from that in the VM
  -quiet
    	Print only errors
  -segment-size int
    	Maximum size of segments in Mb used to upload data. Valid range is 10-4000. (default 4000)
  -stderrthreshold value
    	logs at or above this threshold go to stderr
  -v value
    	log level for V logs
  -vmodule value
    	comma-separated list of pattern=N settings for file-filtered logging

Example run: ./airlock username ExampleBucket ExampleFile will export file ExampleFile to bucket ExampleBucket.

Troubleshooting

See troubleshooting for fixes to known issues.

License

Data Gateway is released under MIT, see LICENSE.

Wails is released under MIT

CgoFuse is released under MIT

sda-filesystem's People

Contributors

Stargazers

Watchers

Forkers

o-alex

sda-filesystem's Issues

Remove .c4gh extension from encrypted files

At the moment files that are encrypted have .c4gh extension in mounted filesystems. This causes issues with applications that try to detect file format from filename extension. Especially, this will cause problems Windows where this is default behaviour.

We should also try to find a way to deal with filename conflicts as that might happen when we remove .c4gh extension.

improve error message shown to the user

This issue is based on #25 (comment)
We need to distinguish between in the UI:

user has access and no datasets
user does not have access
we also need to identify 500 errors when the API is unavailable
for other errors we default to Unknown error

Depends on: neicnordic/sda-download#125

move selecting directory from main view

The view below should change as follows:

Choose directory should be remove from Home tab s well as the title for the 2. Mount Directory
We will default to user's Home/Projects directory
keeping choosing the directory functionality, by introducing a Settings tab after Home which will allow users to change the directory
if an error occurs and the Home/Projects directory has already data or is already mounted we will redirect users to Settings tab to change it

Run unit tests in github actions on every push

Description

we want to use github actions to run unit tests automatically and report the coverage at https://coveralls.io

DoD (Definition of Done)

~~enable coveralls to take coverage from the repo and place its token so that it can report the coverage~~ @blankdots did this
create a github action to run unit tests in go example: https://github.com/neicnordic/sda-pipeline/blob/master/.github/workflows/test.yml

Testing

Unit tests run on every PR

Refactor GUI

Refactor qt to wails

Head request should check status code

https://github.com/CSCfi/sda-filesystem/blob/master/internal/api/api.go#L198-L200

We only check the error from a HEAD request, but it seems, that we also need to check the status code from response, it looks like the error is nil, because the server responded, but it still has an error code, e.g. 404

change default window size

Default window size should not be full screen.
pending information what should be the default window size.

Refresh access

Design: https://www.figma.com/file/L7lldtlYvpXNZrriYueZVC/DRAFT-2---Data-Gateway-with-compute-(Copy)?node-id=2948:2962

SD-Connect API merger

SD-Connect Data-API and Metadata-API are going to be merged into a single app. Old endpoints stay the same. Instead of two environment variables we're going to need only one.

Old

export FS_SD_CONNECT_METADATA_API=https://connect-metadata-api-test.sd.csc.fi
export FS_SD_CONNECT_DATA_API=https://connect-data-api-test.sd.csc.fi

New, e.g.

export FS_SD_CONNECT_API=https://connect-api-test.sd.csc.fi

Filter logs

Give user the possibility to filter logs in the UI based on the levels (info, warning, error)

change main services names

change the names of the services:

SD-Submit- > SD Apply - probably we should change folder name to SD Apply instead of SD-Submit
SD-Connect- > SD Connect

Modal

Design: https://www.figma.com/file/L7lldtlYvpXNZrriYueZVC/Data-Gateway-Prototype?node-id=3135%3A5057

Refresh FUSE layer

As a user i need to be able to refresh the fuse layer with new projects/datasets that have been added while i keep the UI open

It is assumed that we need to check whether projects/datasets have been added/removed and what is inside projects/datasets

Disable buttons in the UI if the required envs are missing

if FS_SD_CONNECT_METADATA_API and FS_SD_CONNECT_DATA_API are missing we can disable the SD Connect button
if FS_SD_SUBMIT_API is missing disable SD Apply button in the UI

User should identify the software as Data Gateway

Wherever we say SDA-Filesystem we change to Data Gateway

Application title
main window
logged window

Increase fuse speed with max_pages

If and when cgofuse supports libfuse3, test if increasing max_pages will speed up fuse.

New versions of SD connect might require sending access token

new versions of the SD Connect might require sending SDS_ACCESS_TOKEN due to additional checks being done, by reverse proxy in front of the API

More details required

Select or create folder

Design: https://www.figma.com/file/snbixszUdiXemsRZF4ura6/Data-Gateway?node-id=2425%3A2409

Given that user wants to create folder
when user clicks Folder name field
then user can type in folder name.

As a project manager I want to upload my project's files or folders to SD-Connect from SD-Desktop so I can store and share them safely

Design: https://www.figma.com/file/snbixszUdiXemsRZF4ura6/Data-Gateway?node-id=2425:2387

Export

Design: https://www.figma.com/file/L7lldtlYvpXNZrriYueZVC/DRAFT-2---Data-Gateway-with-compute-(Copy)?node-id=2434%3A3668

Given that user wants to select files
when user Select files button
then user will see file browser and can select files
and when user has selected files
then user will see files in the Export list.

Given that user wants to drag and drop files
when user drags files to area
then user will see that area is highlighted
and when user drops files
then user will see files in the Export list.

Given that user wants to remove files from the Export list
when user selects Remove link next to the file
then user will see that file was removed from the Export list.

Given that user wants to export files
when user selects Export button
then user will see progress bar and file list with progress bars for individual files and can follow progress
and when export is ready
then user will see that progress bar is full
and also progress bars of individual files are full.

Prevent thumbnail creation for files

Updating fuse is not always possible even if user is not using files. This is because, when opening folders in a file browser, some process begins reading files for the intention of creating thumbnails for them (QuickLook Thumbnailing on macOS). This chould be possibly prevented by checking the pid of the program that reads the file.

cscfi / sda-filesystem Goto Github PK

sda-filesystem's Introduction

SDA-Filesystem / Data Gateway

Requirements

Graphical User Interface

Dependencies

Build and Run

Deploy

Command Line Interface

SDA-Fileystem

Build and Run

User input

Airlock

Build and Run

Troubleshooting

License

sda-filesystem's People

Contributors

Stargazers

Watchers

Forkers

sda-filesystem's Issues

Description

DoD (Definition of Done)

Testing

Recommend Projects

Recommend Topics

Recommend Org