Comments (5)
I would tend to be in favor of making it be behind a feature flag.
Since the user already has to choose a database implementation using a feature flag, I would say that it is a price that we already payed.
Also, I think that handling features in workspaces is going to improve in usability in the future, according to this issue.
from alexandrie.
i think using artifactory for binary storage would be useful for a lot of enterprise users.
from alexandrie.
Hi, and thanks for this project! I'm evaluating it for internal use at my company, and I'd like to add support for S3 storage. I've started by trying to use rusoto_s3. Implementing the "get" methods is no problem, but implementing the "store" methods seems to be pretty painful:
- The
put_object
method wants aStreamingBody
as its payload. - Constructing a
StreamingBody
(which is a type alias forrusoto_core::ByteStream
) appears to need aStream<Item = Result<bytes::Bytes, std::io::Error>> + Send + Sync + 'static
, which already seems difficult to satisfy with the current store type bound ofdata: T where T: io::Read
, both because of the inherent pain of converting blocking -> streaming and the missing extraSend + Sync + 'static
type bounds. But on top of that... - Apparently S3 puts actually require you to know the size up front. So not only do we have all the type bound restrictions + sync -> async, we also have to know the full size ahead of time.
It seems like the only way to satisfy that is to read the full data into a Vec<u8>
, which can trivially be converted into a ByteStream
. I found another S3 library that has a method that takes a blocking std::io::Read
er for puts, and that's exactly what it does.
Looking higher up the call stack, the store methods are called from alexandrie/src/crates/publish.rs
, and it looks like they have already read the full data into memory. So passing a T: Read
down to the store methods which would read them into a new vec would just be making a copy. That doesn't seem terrible, but doesn't seem ideal either. It looks like with a small amount of work, the caller could pass an owned Vec<u8>
down to the store methods, which would avoid the double copy.
Do you have any interest in changing the store methods to take a Vec<u8>
instead of a T: Read
? Or would you rather keep T: Read
but need to make a copy when storing to S3? (Or if you see another avenue out of this, that'd be great too!)
from alexandrie.
@jgallagher-cs Hello, and thank you for the interest and willingness to help !
It indeed would be really cool to have the ability to leverage S3 as a storage option.
Reading your comment made me realize that the Storage
abstraction is trying to do more than what is actually needed from it.
You're completely right that we have no use for streaming bytes into the storage without having buffered the entire crate already.
I (most-likely) made the the mistake of adding this trait method as a way-too-early guess that it could be useful one day, but, as you've shown, even if it was something that is needed/desired, the current implementation is not the right one.
I am in favor of changing the API as you suggested, by simply taking a Vec<u8>
.
I think it models better what the registry really does and needs.
from alexandrie.
Do you want S3 support gated behind a Cargo feature? It ends up pulling in quite a few dependencies, so I would assume "yes", but features don't really play nicely with workspaces (e.g., rust-lang/cargo#7507). The docs for building with S3 support would be something like
cd alexandrie
cargo build --release --features s3
:(
from alexandrie.
Related Issues (20)
- Is it repository Deprecated? Is the code still updated? HOT 2
- Support for Azure AD single-sign-on HOT 1
- add additional details to error logs HOT 1
- locked sqlite db? HOT 4
- Apologies for the inactivity period HOT 1
- "The published version is too low" - why is that an error? HOT 3
- Some potential data racing issues
- Self-modifying login form password field interferes with the Firefox remember password feature
- ERROR: insert or update on table "sessions" violates foreign key constraint "sessions_author_id_fkey" HOT 1
- After updating to Merge pull request #116 ,i can't publish new version of crates
- Question about database and build process
- Best method to remove/delete a crate HOT 1
- Enable git HTTPS authentication via username/password
- TLS support HOT 3
- Cargo checksum verification fails HOT 14
- Running error๏ผ๏ผ๏ผ[frontend] missing field `origin` HOT 4
- cargo owner --add error HOT 4
- Consider switching to tokio/axum/hyper stack HOT 9
- Allow configuring the crate upload size limit HOT 5
- Could you provide a simple demo ?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from alexandrie.