My impression is that gouged is the service that provides the SQLite abstraction?

If this is correct, is there any interest in writing a very simple python client to this service/gouged?

When attempting to start the example server, I trip over:

penberg@vonneumann chiselstore % cargo run --example gouged -- --id 1 --peers 2 3
    Updating crates.io index
   Compiling event-listener v2.5.3
   Compiling chiselstore v0.1.0 (/Users/penberg/src/chiselstrike/chiselstore/core)
   Compiling async-mutex v1.4.0
    Finished dev [unoptimized + debuginfo] target(s) in 6.63s
     Running `target/debug/examples/gouged --id 1 --peers 2 3`
RPC listening to 127.0.0.1:50001 ...
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 61, kind: ConnectionRefused, message: "Connection refused" })))', core/src/rpc.rs:55:52
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 61, kind: ConnectionRefused, message: "Connection refused" })))', core/src/rpc.rs:55:52

I bisected the problem to 75ec0c7

Currently, RPCs establish a new connection for every invocation. Let's use a connection pool to make this faster.

Raft state machine advances at the same rate as heartbeat interval (500 ms), which limits throughput.

The connection pool currently grows without bounds. Let's enforce some limit to it.

Currently, the replicated Raft log grows unbounded. First, we need to add Raft snapshot support to Little Raft, which allows truncating the log. We then need to add snapshot support to ChiselStore with, for example, SQLite online backups: https://www.sqlite.org/backup.html. IOW, normal reads and writes could go to an in-memory database, and at snapshot time, an on-disk backup is created. When a node is restarted, the on-disk backup could be read to an in-memory database.

Non-deterministic SQL function such as date() and random() must be evaluated only once. We can do this by evaluating the functions only on leader and replicate the evaluated values to followers.

@glommer suggests:

Pekka, please add a variation of #![warn(missing_docs, missing_debug_implementations, rust_2018_idioms)] to the top-level lib file so we make sure that all public interfaces have Debug, documentation, etc (likely you will want to enforce 2021 idioms)

Currently gouged assumes that all nodes are running on local machine and uses a mapping between nodes and ports. Make the peer IP addresses configurable to make testing with multiple machines easier.

Suggested by Andy Pavlo: turn on SQLite strict mode to avoid some idiosyncrasies.

We currently let SQL transaction commands to replicate, which can result in a dead-lock.

For example, if a writer issues a BEGIN TRANSACTION command, but dies after it has replicated, the cluster is in a situation where all nodes have an active transaction, but there's now writer to commit or roll back.

Currently, ChiselStore performs reads using the Raft consensus protocol -- just like writes -- which provides strong consistency. However, strong consistency also makes reads pretty expensive. One way to optimize reads is to provide a "local read" option, which performs reads on any node. The "local read" option relaxes the consistency model by allowing stale reads in exchange for higher performance.

Let's delegate client request to the leader instead of erroring out.

If we have registered a pending transition on the leader, but there's a leader change, the the client request has to either be rejected or restarted.

To fix this, we need a new Abandoned transition state in Little Raft:

andreev-io/little-raft#13

If a leader crashes after applying a command to the state machine, but before responding to a client, the client will re-try the command, which breaks exactly-once semantics. One way to fix this is to make the client include an ID for every command. A leader can then check for the ID in its log before applying a command to its state machine to detect stale commands.

We currently use file-backed SQLite databases because the basic in-memory SQLite does not seem to support concurrent reads:

75ec0c7

Let's look at ways to turn on in-memory again. For example, the "memdb" VFS should support concurrent reads.

We currently use a single thread for all threads, which limits SQLite read performance for local reads.

If we have an I/O error, we must not resume completion, but instead crash the process so that node with partial results does not continue in the same Raft cluster.

We don't need to replicate SQL read commands to other nodes if they don't have side-effects.

With relaxed reads, there's currently no guarantee on read-your-write consistency. This is because a write will be acknowledged when the write is applied to the state machine of the leader, but not on the local replica.

Example:

• A follower node F receives a write request, which is delegated to node L, which is the leader.
• The write is replicated to the logs of all nodes (but not necessarily applied).
• The write is applied to the state machine of L.
• L acknowledges the write to F and F acknowledges the write to the client.
• A relaxed read request arrives on node F, which does not yet have the write applied to its state machine, violating read-your-writes consistency.