Giter VIP home page Giter VIP logo

cirrus's Introduction

Cirrus

GoDoc Go report

Cirrus lets you manage your Snowflake node IDs.

Etymology

High-level clouds, called cirrus clouds, can reach heights of 20,000 feet (6,000 meters) and are typically thin. They do not produce rain and often indicate fair weather. They are usually made up of ice.

Not Production Ready

See TODO list below.

What's Snowflake?

Snowflake IDs are special 8-byte IDs generated by Twitter's Snowflake algorithm.

Each Snowflake ID is a 63 bit int.

  • 41 bits for a timestamp (milliseconds since a custom epoch)
  • 10 bits for a node ID (ranging from 0 to 1023)
  • 12 bits for a sequence number (ranging from 0 to 4095)

Why Cirrus?

If you're planning on horizontally scaling (i.e., replicating) your app, you'll need a way of doling out node IDs.

In the era of Docker and Kubernetes, containers and pods are expected to be ephemeral. They can go down at any moment. If that happens, they have no way of relinquishing their node IDs, and so that node ID is essentially lost.

Fortunately, checking for "stale" IDs is super easy, so we can ensure we have a dynamic and reliable way of re-requisitioning and reallocating node IDs.

What does Cirrus do?

Cirrus maps node IDs to app IDs.

An app ID is a string that can be anything (e.g., an int, a UUID, a hash, etc).

A node ID is an int ranging from 0 to 1023.

Apps must periodically send heartbeats to Cirrus, otherwise their node IDs get requisitioned.

TODO

Questions

  • How do we know if a node has actually died? What if it's just busy and stays alive? There might be a moment where 2 apps generate PKs with the same node ID? That would be disastrous.

Single point of failure

Cirrus needs to be able scale out so it's not a single point of failure.

We need a couple things:

  • Replace AvailableNodeIds channel with a distributed queue system (e.g., RabbitMQ, Cherami, etc)
  • Replace Heartbeats map with a distributed store (e.g., etcd)

Usability

  • Usage section in README
  • Configurable durations
  • Dockerfile
  • Helm chart

cirrus's People

Contributors

kevinmichaelchen avatar

Watchers

 avatar  avatar

Forkers

theroka

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.