[feature request] go-memdb backend about kine HOT 15 OPEN

fwiw, I tried using sqlite in memory mode and it's pretty flakey, due to changes in table locking semantics.

docker run --rm -it --privileged --name k3s-server-1 rancher/k3s server --datastore-endpoint 'sqlite://file:memdb1?mode=memory&cache=shared'

from kine.

yep.

Some of the requirements to use the sqlite in-memory store are described here: https://www.sqlite.org/inmemorydb.html - I don't remember specifically what I ran into with locking, but it should be easy enough to reproduce.

from kine.

That might be hard, since kine expects a SQL compatible storage engine. Seems like if you want an ephemeral dataatore you could get the same result out of sqlite with the special :memory: database file path?

from kine.

yep, at the same time I also hope not to introduce other components to accomplish this.

from kine.

maybe we can using https://github.com/araddon/qlbridge to translate it. https://github.com/araddon/qlbridge/tree/master/datasource/memdb

from kine.

Introduce an SQLite with the in-memory file path is still a bit heavy for us, we need some further discussions.

from kine.

What specifically are you trying to achieve?

from kine.

fwiw, I tried using sqlite in memory mode and it's pretty flakey, due to changes in table locking semantics.

Is this still the case? I've recently started exploring kcp-dev/kcp#54 (comment) how to use kine with kcp as a means to support a light-weight binary CLI program that uses controllers. I was hoping to leverage an ephemeral, in-memory DB like sqlite in-mem for this purpose.

from kine.

You could just use kine but store the SQLite database in a temporary directory that's not persistent?

from kine.

You could just use kine but store the SQLite database in a temporary directory that's not persistent?

True, but part of the goal is a super portable, self-contained means of implementing controllers in CLI programs. I'll take your suggestion under advisement. Based on the response, can I assume that in-memory sqlite still does not work well?

from kine.

You're welcome to try it and see if it works for you. It didn't work very well for a whole Kubernetes cluster using the example command I gave.

from kine.

Ack. Thank you. So just to be clear, to your knowledge no one has a) tried it since you did last July or b) worked on it since then either?

from kine.

Seems to be the same issue described here: mattn/go-sqlite3#50 (comment)

from kine.

That might be hard, since kine expects a SQL compatible storage engine.

@brandond Is this statement still valid?

The reason I think it may not, is: here https://github.com/k3s-io/kine/blob/master/pkg/endpoint/endpoint.go#L59 kine creates a raw etcd listener without any sql specific wrapper. It looks simple task to create any backend, not just SQL. As i see a backend only has to implement this interface: https://github.com/k3s-io/kine/blob/master/pkg/server/types.go#L20 and has to start it's own listener.

🙏

from kine.

We now have a non-sql backend in https://github.com/k3s-io/kine/tree/master/pkg/drivers/nats, but it was fairly non-trivial for the Synadia folks to implemement. You might look at that as an example though.

from kine.