loro-dev / columnar Goto Github PK

View Code? Open in Web Editor NEW

21.0 3.0 0.0 238 KB

ergonomic columnar storage encoding crate

License: MIT License

Rust 100.00%

columnar columnar-storage rle serde compression backward-compatibility forward-compatible

columnar's People

Stargazers

Watchers

columnar's Issues

[LORO-278] Replace macro attribute "original_type" to "copy"

null

_LORO-278

Integrate columnar to Loro

Generics trait bound & lifetime bound

#[columnar(ver, ser,de)]
struct Data<T, P>{
     t: T,
    #[columnar(skip)]
     p: P
}

we need add trait bound for T

#[columnar(ser, de)]
struct B<'a> {
    a: &'a str
}

Implement macro in nested case

#[columnar(vec)]
struct Data{
    id: u64
}

#[columnar(vec)]
struct Store{
    #[columnar(type="vec", warp)]
    data: Vec<Data>,
    #[columnar(strategy="DeltaRle")]
    id: u64
}

#[columnar]
struct NestedStrore{
    #[columnar(type="vec")]
    stores: Vec<Store>
}

I think maybe we could still impl nested case without wrap attribute, by always wrapping vec or map.

Use an iterator to iterate over each `row struct` when deserializing

Add #[columnar(skip)] to macro

pass &T instead of Cow<T> so that DeltaRle can distinguish num type by size of<T>

Feature: Output the size of each column after encoding

check whether `IT` of `ColumnarVec` needs `Clone`

[LORO-371] The behavior of richtext's `applyDelta()` is not entirely correct

const doc = new Loro();
const text = doc.getText("text");
const doc2 = new Loro();
const text2 = doc2.getText("text");
text.subscribe(doc, (event)=>{
  const e = event.diff as TextDiff;
  text2.applyDelta(e.diff);
});
text.insert(0, "foo");
text.mark({start: 0, end: 3, expand: "none"}, "link", true);
// doc.commit();   // multi delta
text.insert(3, "baz");
doc.commit();
await new Promise((r) => setTimeout(r, 1));
expect(text2.toDelta()).toStrictEqual([ { insert: 'foo', attributes: { link: true } }, { insert: 'baz' } ]);

If uncomment first doc.commit(), the result will be wrong

_{From SyncLinear.com | LORO-371}

Build a benchmark to compare performance

candidate：

columnar
postcard
bincode

Use izip! macro to zip more than two iterator

There is a bug

// when there are more than two iterator
let (a, b,c ) = iter_a.into_iter().zip(iter_b.into_iter()).zip(iter_c.into_iter()).collect();
// it will get  `((a, b), c)`   not  `(a, b, c )` 

let (a, b, c) = izip!(iter_a.into_iter(), iter_b.into_iter(), iter_c.into_iter())

Split VecRow into SerVecRow DeVecRow (MapRow is the same)

Generally, when encoding, we build some intermediate struct that holds value‘s borrow. And we definitely do not decode into this intermediate struct, which means it is inflexible to require the implementation of serialize and deserialize at the same time.

Add support for `FieldAnalyze` with lifetime

Support enum type as container

#[columnar(ver, map, ser, de)]
struct Data{
    id: 64,
    name: String
}

#[columnar]
enum EnumStore{
     #[columnar(type="vec")]
     Data(Vec<Data>),
     #[columnar(type="map")]
     MapData(HashMap<u64, Data>)
}

Add `#[columnar(borrow)]` support

#[columnar(ser, de)]
struct Data<'a>{
    #[columnar(borrow)]
    a: Cow<'a, str>
}

The number of fields is variable because of supporting forward and backward compatibility, so we could not reuse the borrow attribute from serde directly.

_{From SyncLinear.com | LORO-287}

Tree move

[LORO-257] Transaction decode

cache the encoded inputs then decode them all at once when needed.

remove the decode_batch method from Loro and make importing a lot of things within a transaction pretty fast.

loro-dev/loro#83 (comment)

_{From SyncLinear.com | LORO-257}

add #[columnar(no_strategy)] to reduce unnecessary PartialEq

For example

#[columnar(vec, ser, de)]
#[derive(Clone, Serialize, Deserialize)]
struct Data{
    #[columnar(strategy="Rle")]
    id: u64
    #[columnar(no_strategy)]
    other_data:  T: !PartialEq,
}

#[columnar(ser, de)]
struct Store{
    #[columnar(type="vec")]
    data: Vec<Data>,

}

Add strategy for keys in BTreeMap

#[columnar(ser, de)]
struct Store{
    #[columnar(class="map", key_strategy="DeltaRle")]
    mapping: BTreeMap<u64, String>
}

Forward and backward compatible support proposal

Considering cases like:

// old version
struct Data{
    id: u64,
    name: Option<String>
}


// new version
struct Data{
    id: u64,
    // delete name option field
    // add a new option field
    count: Option<u64>
}

cases:

delete option field
add a new option field

Both Table Struct and Row Struct

loro-dev / columnar Goto Github PK

columnar's People

Stargazers

Watchers

columnar's Issues

Recommend Projects

Recommend Topics

Recommend Org