bigeasy / compassion Goto Github PK

Writing software in C, shell, Node.js and Go. Managing infrastructure with Kubernetes, ELK, Kafka, Consul/Vault, Terraform and Pulumi. Formerly Wink, currently available for contract.

My most popular project is Timezone. A IANA DB timzone library for Node.js or the browser that does timezone aware date math and localized time formatting.

Currently working on documentation for these featured projects. They are the most interesting projects in this account.

The projects in this account come together to create four primary projects. Memento a pure-JavaScript relational database, Compassion a Paxos-based atomic log, Packet a binary packet parser and serializer generator and Prolific a logging system for Node.js applications that won't lose messages.

As of October 2021 my primary focus is on documentation of all this work. Please be patient as I work though the README.mds.

Strata gets a lot of attention because it is a pure-JavaScript b-tree for Node.js. A proper file-backed b-tree that can write to either a directory tree or a write-ahead log. The write-ahead log is itself useful and you can find it in WriteAhead.

Memento is a practical application of Strata. It is a pure-JavaScript noSQL relational database for Node.js. It is ACID. It is atomic with all or nothing transactions, isolated in that changes made during a transaction are only visible to that transaction, and durable in that there are no parietal writes and committed writes will survive a system crash. It is based a multi-version concurrency control model and I've a collection of libraries to build MVCC databases off of Strata.

IndexedDB is a pure-JavaScript implementation of IndexedDB for Node.js. It is build on top of Memento and is primarily a proof of concept and is used to leverage the Web Platform Test suite for IndexedDB to get a through testing of Memento.

Compassion is an atomic log based on a Paxos. People are drawn to the Paxos implementation via NPM but repo it is the core Paxos algorithm without networking components. If you want to use that Paxos implementation you should instead use Compassion. Compassion provides the service discovery and network communication. It implements an async/await interface you can use to build applications. You can use Conference to implement a map/reduce, which ends up being a useful concept in consensus applications, allowing you to take actions based on whether or not all participants have been notified. Currently sketching out Addendum as a proof-of-concept for Compassion that implements the etcd v2 API in Node.js using Paxos instead of Raft.

Packet is a binary parser generator for Node.js that generates pure-JavaScript whole or incremental parsers and serializers from a syntax-bashed JavaScript definition language. These generated parsers and serializers ought to be as performant as any parser or serializer you would write by hand.

Prolific is more than yet another logging library. It is both performant due to asynchronous message processing during normal operation, and durable due to synchronous message processing upon uncaught exception. Unlike other logging libraries Prolific will not lose your parting stack trace. The final messages will be captured and redirected to the same logging stream as your runtime messages. This is kind of a big deal.

Destructible is a structured concurrency library I created before I learned about structured concurrency. It has a different interface than the Python libraries like Trio with it's nurseries, but the underlying concepts are the same, launching multiple concurrent paths of execution and having them rendezvous in a single promise upon completion. Destructible manages multiple async/await call stacks and when an exception occurs in any one of them it will gather all exceptions from an async/await call combine them into an elaborate report of all stack traces through that single promise.

Turnstile performs parallel concurrency with an explicit work queue that can be monitored rather than the implicit work queue of spawning an arbitrary number of async function calls. Fracture is a more elaborate parallel concurrency library with the ability to enqueue work in the work queue from the work queue in a way that will avoid deadlock.

Avenue is a module that implements what are essentially Go channels, channels that can be processed synchronously or asynchronously by zero, one or more consumers (i.e. multiplexed.) The consumption decisions are deferred so that the producer does not have to concern itself with the concurrency considerations of the consumer.