streamlining DB api with regard to CFs about rust-rocksdb HOT 15 CLOSED

I like the idea of making the Column Family stuff easier to use. Though I don't have good ideas how to do it :)

How would DB::open() know about the existing Column Families? In RocksDB you need to pass in their name and options.

from rust-rocksdb.

There is a DB::list_cf (rocksdb_list_column_families in the C API) function which just takes a path and lists all of the column families stored in that file path.

It actually represents one of my annoyances with the original API. There is a function to enumerate column families, but when you create a DB object, you have to explicitly specify the CFs (and if you specify one that doesn't exist, it doesn't create it, it just errors). I mean.. I can appreciate forcing people to be explicit when it comes to data storage, but that's just a bit wonky.

In rocks-cache I have a Builder style object which enumerates all CFs, and then lets you add attitional ones if you like, and it will load all of these into the RocksDB::DB (this requires a &mut DB) then after you build() it returns a DB object which doesn't allow mutation of CFs, that way you don't need a &mut DB.

from rust-rocksdb.

The problem is that with ListColumnFamilies() you'll only get the names, but not the options. So your application needs to keep the options somewhere and then it's easy to store the name with it as well.

from rust-rocksdb.

I see your point.

It seems to me that the problem is that you need to supply the options twice. Persisting them to disk with the CF seems like the only safe thing to do.

from rust-rocksdb.

I think the original post of issue facebook/rocksdb#1055 sums up the problem with RocksDBs Column Family API pretty nicely.

from rust-rocksdb.

What I get from that is their excuse for why they designed their API so poorly (needing to associate options with CFs) should actually be the exact reason to design an API that is more ergonomic and also foolproof.

from rust-rocksdb.

While reading the RocksDB source I found: https://github.com/facebook/rocksdb/wiki/RocksDB-Options-File. Though one problem surely are the "unsupported options".

from rust-rocksdb.

As there is the create_missing_column_families option how about an API like this:

let cf1_options = ColumnFamilyOptions::new("cf1_name");
cf1_options.set_max_write_buffer_number(16);

let options = Options::default();
options.set_column_families(vec![cf1_options])

// `create_missing_column_families = true` is default
let db = DB::open(&db_opts, "path/for/rocksdb/storage_with_cfs");

let cf1 = db.get_cf("cf1_name");
cf1.put("hello", "world");

from rust-rocksdb.

That looks good to me. I think that'd be an excellent place to start. I'll have a look at #136 as well.

After that,
We might want to experiment with paramaterizing the ColumnFamily by key and value types.
The hard part is that the types would need to be Deserialize<'a> + Serialize which is a bit of a pain.
The neat thing is that we could associate MergeOperators with the CFs and ensure that they work with the types.

Thoughts?

from rust-rocksdb.

@rrichardson I don't understand what you mean with "paramaterizing the ColumnFamily by key and value types".

from rust-rocksdb.

something like :

impl<K: Serialize + Deserialize, V: Serialize + Deserialize>   ColumnFamily<K, V> {
  fn put(K, V) -> Result {}
  fn get(K, V) -> Result {) 
  fn merge(K, V) -> Result {}
}

so we could have type safe serialization and deserialization as well as merging that could support a specific set of operations (sets, counters, etc)

I'm experimenting with something like this in rocks-cache at the moment.

from rust-rocksdb.

Oh, I see. That would be pretty cool.

from rust-rocksdb.

It's been over a year since this was discussed. Suggesting that we close.

I share your frustration with the abstractions. Much of the ergonomics when using column families is no doubt a result of column families being a later addition to underlying library. However, I think the direction with this library has historically always been a thin Rust wrapper over the underlying RocksDB. Drastically changing the Rust API abstractions in comparison to the native RocksDB abstractions would require a greater documentation burden because of how different the libraries would become. Creating an opinionated wrapper around RocksDB I think should be pushed into a different crate, using this one as a dependency.

from rust-rocksdb.

As it's been over a year, I don't remember the details. Though IIRC it would be quite difficult to create a new crate which just chnages how column family work, without being essentially a fork. I'd still believe that a simpler API is desirable, but as I don't have any time for helping out at the moment, I'm OK with having it closed as "won't fix".

from rust-rocksdb.

@vmx Sounds good. I've been spending a lot of time trying to improve the ergonomics of the API while staying true to the underlying API (see #253 and #255). I'm going to close this, but I'm open to more suggestions on how we can improve the ergonomics of the API.

I've got another PR planned to make better use of traits for the definition of "PIGMED" operations. It will simplify the existing functionality as well as make it easier to implement #144, #178, and we might be able to implement some of the traits on ColumnFamily, getting some of the ergonomics asked for here.

from rust-rocksdb.