blazebit / blaze-storage Goto Github PK

The idea is to introduce something like a bucket object retention plan which can be assigned to bucket objects. If needed we could introduce a default assignment to objects based on e.g. bucket object prefix.

The plan defines a retention period e.g. 30 days and an action. Interesting actions are

• Move to storage + set new retention plan
• Delete
• Publish a bucket object event

The use case is to provide a temporary storage for e.g. attachments that are used when sending an email. An attachment can be put into the bucket with a retention of e.g. 30 days. An e-mail is sent with the bucket objects as attachment. The bucket object is kept around for a while in case it is necessary to debug or re-send the e-mail.

Right now we only set the state of a file to deleted but don't actually delete the file.
A custom task that runs periodically should delete files and on complection also remove the database entries and update pending statistics.
That task could also be triggered in an after-transaction-synchronization-callback (CDI event)

Would be great to use updatable entity views to reduce the amount of queries.

Use https://github.com/dCache/nfs4j for implementing access to buckets through NFS mounts.

We have to think about how to assign "persistent" inode numbers to bucket objects and also to their "virtual directories" as that is what NFS needs to identify objects.

One possible implementation would be to simply not make inodes persistent, but just cache them in memory. This would be ok for uses that only do reads and creates, but even then this could be painful. All clients have to reset their local inode caches after a restart.

A much better implementation would be to allow a global configuration, that when enabled, will maintain a inode number on the bucket_object table based on a sequence through Blazebit/blaze-persistence#365

For directories, we would need a separate table where inodes are managed.

We need an API that allows streaming data. Currently, we expect an input stream to be available, which could be lazy, but usually is not. Being able to write to the final outputstream within the correct context would be nice if possible with JAX-RS

Implement a copy operation that allows to do efficient copying of files on the same storage.

Add some more tests to cover the admin/owner security model.

See https://www.ovh.com/blog/dealing-with-small-files-with-openstack-swift-part-2/ for how we could structure files on the file system to avoid running into filesystem limitations.

Implement uploading of multiple files at once.

Asynchronous copying of bucket objects between storages should be supported e.g. for archivation. Currently, this is only supported in a synchronous fashion.

Right now we test the REST API manually by using it in a front end web application. We should create end-to-end tests for it to catch possible regressions.

Currently we don't publish the sources JAR for WAR projects which results in not being able to reuse the Java classes used for the admin UI. The maven WAR plugin should be adapted to publish this artifact as well.

Like S3, we should implement support for bucket events with support for the following sinks

• SNS
• SQS
• Kafka

We should also implement the alerting based on these events.

The event publishing options are

• Publish event in a PreCommitListener => Might trigger an event although the TX failed
• Publish event in a PostCommitListener => Might miss an event in case of outage
• Source events via Debezium => Requires proper setup

Implement the possibility to download multiple files at once in a multipart response.

Finish the simple object delete operation and also allow to delete multiple objects at once.

Currently we do not really use the checksums. This should change.

Implement ACL security

It should be possible to publicly share files via a link. We could use a UUID for that purpose.

Right now we only collect statistics on storage and bucket level. We should also do that on a per-tag basis.

Implement support for extended feature for HTTP GET of objects like

• range request
• ETag, Last-Modified, If-None-Match support
• Download Speed throttling

Also see: https://www.nurkiewicz.com/2015/06/writing-download-server-part-ii-headers.html