This codebase could potentially be used for reference data storage, allowing us to keep a standardized, centralized place to upload & download RE import data, SDK app data, etc. I propose adding an endpoint that provides a JSON-RPC 2.0 api for uploading files:
Methods:
upload_file
- Upload a new file and get its ID. Returns a hex-encoded content hash of the file, which represents its ID. If the file already exists, then the user is given the file ID and this is a no-op. The user can provide some optional additional metadata which is stored in Minio's object metadata.
download_file
- Download a file from the store using its ID, which is its content hash.
The server can gzip files if they have not already been compressed. We can also have a Python client module that takes care of creating tarballs, gzip compression/extraction, proper low-memory streaming, and checking downloaded data against its content hash.
Optionally, we could have a policy to expire files if they haven't been accessed after a certain time period, such as 500 days.
Blake2b hashing should be used.
Users can upload directories if they pack them into a tarball, and the client can help with this. The server can detect if uploaded files are already gzipped using this