josephg / diamond-types Goto Github PK

Thank you for making this advanced document store. But how to use or how to start using this. Please make a md tutorial (with nodejs as example). Thank a lot

As support for a wider array of data structures grows in diamond types, would a derive macro be worth it? For example, the shelf merging algorithm that I contributed a few months ago could be derived for a struct without too much bother, and CRDTs for other data structures (e.g. strings and lists) could be swapped in with attributes. In essence, I think it would be interesting if we could make something similar to the following happen:

#[derive(CRDT)]
struct Article {
    #[crdt(algo = "rich")]
    content: String,
    #[crdt(algo = "list")]
    tags: Vec<String>,
    slug: String,
    is_ready: bool
}

This would apply shelf logic to the is_ready and slug field (e.g. last write wins) and apply CRDT logic to the list of tags and setup the content field for collaborative rich-text merging. Perhaps it would produce something like:

enum ArticleShelfType {
    String(String),
    Bool(bool)
}

struct ArticleCrdt {
    content: TextCrdt,
    tags: ListCrdt,
    slug: shelf::Item<ArticleShelfType>,
    is_ready: shelf::Item<ArticleShelfType>,
}

impl Into<Shelf<ArticleShelfType>> for ArticleCrdt {
    fn into(self) -> Shelf<ArticleShelfType> {
        // ...
    }
}

impl ArticleCrdt {
    // Generate methods for merging two Article structs.
    // Fields with specific CRDT algos are merged using those rules.
    // All other fields are merged using the shelf algorithm.
}

This is my first stab at an API interface without any significant thought into what it would need to look like – I'd like to test the water first and see if this is a direction in which this library wants to go. I'd be happy to hack around with the process if it's something that would be worth exploring! I think it would break the barrier of entry for CRDTs and provide a comfortable on-ramp for developers to include them in a wider variety of projects (long live collaborative applications!).

I've been meaning to experiment with diamond-types in Helix which already uses Ropey + an OT Transaction to apply changes.

Would it be possible to use diamond-types by feeding it edits then extracting a list of operations to replay on the rope? (I think Atom's teletype-crdt worked in a similar way.) That way we can only use CRDTs when a collaboration session is started and just use the plain rope when editing locally. I saw you had a module that supported this prior to the rewrite, but it's gone now.

I wonder whether you thought of extending diamond types to more complex structures - especially something like rich text?

I just got notification about the article https://www.inkandswitch.com/peritext/?utm_source=geoffreylitt&utm_medium=email (co-authored by @geoffreylitt ) and must say it was an interesting read.

I sense there might be some (smaller or bigger, IDK) overlap with ORDTs, but didn't have time to investigate it now.

Hi, thanks for the amazing library. I have a quick question: can undo/redo be implemented on top of this library and what is a general strategy of doing so?

Hi there!

First of all, thank you for the amazing project and write ups! I'm diving into the world of CRDTs and your content has been immensely helpful to improve my understanding.

Going through the INTERNALS.md file, I've noticed this warning:

This was written for an earlier version of diamond types when I persisted the merge structure like yjs and automerge do. This has much worse performance when there are no concurrent changes, and a bigger file size. TODO: Bring this entirely up to date with the current DT version!

Could you provide more details on the current persistence strategy? I'd be so interested to learn more about it

Keep rocking!

Hi, I'm starting to explore the API and internals of DT starting from this simple example, but I must be using the API incorrectly since it panics at the first unwrap().

// Insert "ab" at site 1 and send the document over to 2 other sites.

let mut oplog1 = OpLog::new();

let site1 = oplog1.get_or_create_agent_id("1");

let ab = oplog1.add_insert(site1, 0, "ab");

let ab = oplog1.encode_from(EncodeOptions::default(), &[ab]);

let mut oplog2 = OpLog::new();

oplog2.decode_and_add(&ab).unwrap(); // -> BaseVersionUnknown

let mut oplog3 = OpLog::new();

oplog3.decode_and_add(&ab).unwrap();

// At this point all the documents should contain "ab".

let doc1 = oplog1.checkout_tip();
let doc2 = oplog2.checkout_tip();
let doc3 = oplog3.checkout_tip();

assert_eq!(doc1.content(), "ab");
assert_eq!(doc2.content(), "ab");
assert_eq!(doc3.content(), "ab");

Unfortunately the BaseVersionUnknown error variant is not documented and a first (albeit cursory) look at the source code wasn't illuminating.

I'm using the latest version published on crates.io which should be 8fc685d.

Hi, Seph!

I'm curious about the way you plan to implement different data types within diamond (maps, graphs?, etc.), do you already have something specific in mind in this regard? Would you implement an approach similar to the one of Yjs where users can create different data types with different "names" under the main CRDT document, or you have something different in mind?

Hello, I am learning CRDT recently and trying to implement my own crdt。After reading your article, I have a few doubts.

1. Why do I think RGA in the paper looks more like yjs rather than automerge? Did I misunderstand ？

interface Item {
    data: string
    /**
     * [agentId, sequence]
     */
    id: [string, number]
    /**
     * [agentId, sequence]
     */
    left: [string, number]
}

more like a list of Item (order by id)?

2. If only need to implement plain text, is it necessary to record the parent field?

Yjs just puts all the items in a single flat list:

state = [
  { item: 'a', id: ['seph', 0], seq: 0, parent: null },
  { item: 'X', id, seq, parent: ['seph', 0] },
  { item: 'b', id, seq, parent: ['seph', 0] },
  { item: 'c', id, seq, parent: [..] }
]

Just need item and id and 'left` to implement the plain text crdt ? or parent and right are equivalent ?

const state = [
    {
        "item": "a",
        id: ['a', 0]
    },
    {
        "item": "x",
        id: ['b', 0],
        left: ['a', 0]
    },
    {
        "item": "b",
        id: ['a', 1],
        left: ['a', 0]
    },
    {
        "item": "c",
        id: ['a', 2],
        left: ['a', 1]
    }
]

I see you're a long-time expert on CRDTs and such. So it's highly probable you'll be already familiar with this.

Nevertheless let me ask what you think about "ORDT"s when compared to the CRDTs implemented here in diamond-types?

About 3 years ago an article about ORDTs (and Causal Trees) explained a (rather highly successfull) attempt to solve the bunch of the hard problems I see diamond-types is trying to tackle. Including generalization for other data types (maybe even hierarchical ones).

I'm also highly interested in the pruning (space & bandwidth) & reconstruction (speed) strategies. Could you elaborate on how that's (meant to be - in case it's pluggable/configurable) solved in diamond-types?

I myself thought about a hybrid strategy I outlined in HigherOrderCO/Kind1#167 (comment) (compare it to automerge/automerge#253 ).

Btw. there seem to be some example implementations of ORDTs - e.g. https://github.com/courajs/referent .

Hi! I had a question while browsing the automerge folder in this repository.

In CRDTLocationExternal, there's a field called seq. From seeing how this is used, I don't think this is an Automerge sequence number, right?

https://github.com/josephg/text-crdt-rust/blob/ba20b6386c0472958f33024ce0b806e75470e1ca/src/automerge/mod.rs#L22

But I'm assuming insert_seq_start represents an Automerge sequence number?

https://github.com/josephg/text-crdt-rust/blob/ba20b6386c0472958f33024ce0b806e75470e1ca/src/automerge/mod.rs#L51

Just wanted to clarify this. 🙂

I also see that there's a JS version of (at least some of) the Rust code?

https://github.com/josephg/text-crdt-rust/blob/ba20b6386c0472958f33024ce0b806e75470e1ca/src/automerge/txn.rs#L359

I checked https://github.com/josephg/reference-crdts, but I don't see any JS code that corresponds to the the branchContainsVersion checks.

In the benchmark_data folder there are a bunch of traces with a .dt extension which I'm assuming where serialized using a custom binary version of the data format described here.

I got the friendsforever trace in json from that repo, and I was wondering if you also had the json sources for the git-makefile and node_nodecc traces.

First, thanks for development such a library. And I try to use in my opensource project feakin, here is the issue behavior:

1. In JS/Wasm side, when new OpLog() and .toBytes() with return:

Uint8Array(36) [
  68, 77, 78,  68, 84,  89,  80,  83,  0,
   1,  2,  3,   0, 10,   7,  12,   2,  0,
   0, 13,  1,   4, 20,   6,  21,   0, 22,
   0, 23,  0, 100,  4, 108, 206, 107,  0
]

2. but in Rust side, when:

let mut oplog = OpLog::new();
let bytes = oplog.encode(ENCODE_FULL);
println!("{:?}", bytes);

// [68, 77, 78, 68, 84, 89, 80, 83, 0, 1, 2, 3, 0, 10, 0, 20, 6, 21, 0, 22, 0, 23, 0, 100, 4, 134, 119, 77, 106]

the bytes.len() will be 29.

When I generate in backend side (Rust side), and return it to frontend, it will be throw error with ParseError::MissingChunk(12), 12 means ChunkType::Version.

I thinking it means WASM version different to latest Rust side. Maybe some API change, but I'm not have double check. Here is the last published time of JS/WASM in npm and Rust in crate.

• http://npmjs.com/diamond-types-node and https://www.npmjs.com/package/diamond-types-web was publish in 5 months ago
• https://crates.io/crates/diamond-types publish in half month ago.

After I recompiled WASM version in my machine, it works.