Ensure all ObjectStores have publicly available client about object_store_rs HOT 5 CLOSED

I meant adding CopyObject support to ObjectStore itself. A rename is just a copy followed by a delete.

Sounds good. I've started implementing that in #14. I will close #11.

from object_store_rs.

I'm a bit wary of exposing the clients used internally as I suspect we may want to change these down the line. Both Rusoto and cloud_storage are not official crates, with somewhat patchy support.

Perhaps we could add copy support directly to the trait, it seems like a reasonable addition to me?

from object_store_rs.

Perhaps we could add copy support directly to the trait, it seems like a reasonable addition to me?

As in, copy the client rather than returning a reference / shared pointer?

I'm a bit wary of exposing the clients used internally as I suspect we may want to change these down the line.

Yeah, I agree. The alternative would be crates should bring their object store extensions into this crate as optional features. So delta-rs would bring in the rename_noreplace() method as an optional trait. In the S3 case, that might mean bringing over dynamodb lock, which may be more than you wish to accept (or maybe not!).

Or even a final alternative would be that in delta-rs we wrap the ObjectStore implementors instead of using them directly.

from object_store_rs.

As in, copy the client rather than returning a reference / shared pointer?

I meant adding CopyObject support to ObjectStore itself. A rename is just a copy followed by a delete.

The alternative would be crates should bring their object store extensions into this crate as optional features

I think an optional feature that adds CreateIfNotExists and CopyIfNotExists operations would be really cool. It's a real pain that S3 doesn't natively support it, and if we can somehow hide that from users that would be awesome.

from object_store_rs.

In case of Adls Gen2 the *IfNotExist methods are natively supported. In delta-rs we also have an implementation for local file systems, which only on windows needs some extra care - I think... for blob and S3 afaik some external service is needed.

I'd be happy to contribute adls gen2 to this crate 🙂.

from object_store_rs.