Bitsnap

Status

Sorting things out.

Components

Bitsnap

Bitsnap is a wonder wuzzle data processing application platform and a relational API mapper, revolves around the idea of self-optimization and lean designs with minimal longterm ownership costs. It's based on Streamline remoting and eventual consistency models, Quiver serialization and uses Skippy language to define the respective relational mapping rules onto multiple relational, column and time-series storages.

Streamline

Bx IO Monad implementation for cats-effect using Monocle with distributed persistance capabilities.

The general idea is to make IO monad replicated across network using strong eventual consistency on top of CRDT's and Vector Clocks. Think of it as your monolithic codebase becoming a distributed one, without much added effort, with volumetric ton of codegen and dotty wizardry...

Streamline introduces remote observable and fault-tolerant Spools of threads and fibers, tracks the data dependencies and builds a concurrent execution plan, similarly to the existing database engines, to get some estimates if it would make any sense to perform any remote execution and to decide on an optimal batch sizes to lower the IO amplification. It operates by providing remoting-enabled cats-effect scheduling instances, and tracks out data dependencies using common Parallel cats type class.

Protocol-wise it makes sense to get well-polished protocol on board, so Apache Arrow on top of Google's flatbuffers would be really a wise choice and pretty conservative pick. Streamline uses Quiver, for the respective connectivity, serialization and in-mem data storage.

Streamline kubernetes operator and the respective knative-serving / knative-eventing support is under development.

Observability, execution state replay and state logging capabilities are also under development.

Quiver

Quiver is a simplified Google FlatBuffers implementation in Typelevel Scala, with moderate Apache Arrow compatibility. Flat Buffers are much leaner compared to Protocol Buffers and the existing GRPC implementation, although there's some networking overhead and minor IO amplification involved. Quiver is similar to ScalaPB by design, and has Cloud Events compatibility for better future knative-eventing interoperability, although doesn't focus on offheap and the respective performance optimizations yet, because it would require a lot of dotty tinkering, which is a bit out of scope now.

Skippy

bum bum bee dum bum bum bee dum-bum

Data transformation and relational projection language - defines domain bounded contexts, borderline consistency, data retention and availability requirements and builds denormalization strategies for the most common application queries.

About a half of spec had been designed, and the other half should be ready after the GA.

FAQ

1. Why it took you so long ?

• Waited for scala3 and Typelevel ecosystem to mature enough, so I won't be forced to publish much more novel designs, - everything in here is a backport of sorts, after almost a decade of R&D

2. Why Scala3 ?

• Had some plans for golang as well, but with maturing scala-native it would require less effort, dotty flexibility and a fairly concise design reduced complexity a lot

3. Do you plan to Highload anything ?

• scala-native DPDK and SPDK bindings with a lightweight TCP/IP stack port would be nice, cozy and really tempting, but definitely something in unforeseeable future

4. PostgreSQL, Scylla and RedPanda, huh ?...

• well, had been planning to release a ready to use DevOps stack for this as well, to turn Bitsnap into a complete Cloud Native Application platform, and there's just right amount of DPDK/SPDK wizardry to backup some of the performance concerns.

5. When it'll be ready ?

• It's a non-profit activity now, and definetely not a full-time engagement. I'm not expecting contributions, but there are couple more products based on bitsnap under development which should provide further R&D and support cash flow

6. How Bitsnap compares to Akka and disney's Smithy IDL ?

• Don't think that humans should decide and define what an Actor IS, so it's just yet another optimization problem for the compiler to estimate and decide which computational states form up a distributed FSM, not necessarily parallelized one
• IDL wise Bitsnap won't be fully compatible with FlatBuffers - it's using "just the right amount" of primitives to map onto the existing Scala3 type system, and that's it

7. Will you make it into a database ?

• Probably, and I don't want to hear anything about nosql or #hatesql nonsense.

8. Are you planning to contribute to Typelevel ecosystem, and won't it hurt in the long run ?

• There's enough scala in the world, but not enough clean and lean Functional Solutions, Typelevel is the most Concise and Stable one which actually should reduce the long term ownership risks, as long as it's growing and viable

9. Why MPL-2.0 ?

• Prevents potential legal headache

10. What about Scala Frontend ?

• Not sure right now, but I'm not that confident about Scala.js ecosystem, yet partial code reuse would totally make sense.

License

Bitsnap projects are released under the terms of the MPL License.

Contact

Bitsnap has a Discord server.

Support

Enterprise Support and Consulting is planned after the first public GA release.