Giter VIP home page Giter VIP logo

elixir_riak_core's Introduction

Riax Build

Riax is an Elixir wrapper for Riak Core. Riak Core is a building block for distributed and scalable systems in the form of an Erlang Framework.

To learn more about Riak you can check the Riak Core and useful links section sections. To learn more about Riax, check the setup, the tutorial or the API Reference.

If you want to set it up with Erlang, we also have an up-to-date (OTP 25) tutorial.

Use example

iex(dev1@127.0.0.1)1> #### Check the Ring Status
iex(dev1@127.0.0.1)2> Riax.ring_status
==================================== Nodes ====================================
Node a: 64 (100.0%) dev1@127.0.0.1
==================================== Ring =====================================
aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|aaaa|
:ok
iex(dev1@127.0.0.1)3> #### Join an already running Node
iex(dev1@127.0.0.1)4> Riax.join('[email protected]')
13:51:21.258 [debug] Handoff starting with target: {:hinted, {913438523331814323877303020447676887284957839360, :"[email protected]"}}
iex(dev1@127.0.0.1)5> #### Send a command to a VNode
iex(dev1@127.0.0.1)6> Riax.sync_command(1, "riax", {:ping, 1})
    13:13:08.004 [debug] Received ping command!
    {:pong, 2, :"[email protected]", 822094670998632891489572718402909198556462055424}

Installation

Add Riax to your dependencies:

defp deps do
  [
    {:riax, ">= 0.1.0", github: "lambdaclass/elixir_riak_core", branch: "main"}
  ]
end

And follow the setup

Riak Core

What is it?

It is based on the Dynamo architecture, meaning it is easy to scale horizontally and distributes work in a decentralized manner. The great thing about Riak it's that it provides this architecture as a reusable Erlang library, meaning it can be used in any context that benefits from a decentralized distribution of work.

Why Riax?

You must be thinking "ok, fine, this is an Erlang lib, I'll use it directly". The setup of Riak Core can be tricky, specially from Elixir, this library takes care of all the gory details for you - we suffered so you don't have to.

What's so great about it?

The key here is that Riak Core provides Consistent Hashing and Virtual Nodes. Virtual Nodes distribute work between them, and Consistent Hashing lets us route commands to these Virtual Nodes. Note that many Virtual Nodes can run in a Physical Node (i.e. a physical server) and can be easily set up or taken down. Plus, the only thing that you have to do using this library is giving them names and implement a behaviour, Riak handles the rest for you.

Use cases

The most intuitive and straight-forward use case is a key-value store in memory, we've actually implemented one here for our tests.

A game server which handles requests from players could partition players through said hashing to handle load, and ensure that players requests are always handled on the same Virtual Node to ensure data locality.

A distributed batch job handling system could also use consistent hashing and routing to ensure jobs from the same batch are always handled by the same node, or distribute the jobs across several partitions and then use the distributed map-reduce queries to gather results.

Another example: Think about serving a dataset which you want quick access to, but It's too big to fit in memory. We could distribute said files (or file) between Virtual Nodes, use an identifier (say, like an index) hash it and assign it to a Virtual Node. Riak fits really well here as it scales easily horizontally. This last use case is actually explained below.

More about Hashing and VNodes

Before performing an operation, a hashing function is applied to some data, a key. The key hash will be used to decide which node in the cluster should be responsible for executing the operation. The range of possible values the key hash can take (the keyspace, usually depicted as a ring), is partitioned in equally sized buckets, which are assigned to Virtual Vodes.

The Ring

Virtual Nodes share what is called a keyspace. The number of VNodes is fixed at cluster creation and a given hash value will always belong to the same partition (i.e. the same VNode). The VNodes in turn are evenly distributed across all available physical nodes. Note this distribution isn't fixed as the keyspace partitioning is: the VNode distribution can change if a physical node is added to the cluster or goes down.

After this, be sure to check the tutorial to see this in action

Useful links

elixir_riak_core's People

Contributors

fkrause98 avatar loco1982 avatar unbalancedparentheses avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

Forkers

cjimison

elixir_riak_core's Issues

Library Objectives

  • Provide a module (Riax, probably) to communicate with a VNode that handles:

    • Sync Commands
    • Async Commands
    • Preferred Node Names
    • Coverage Commands
    • Ring Status
  • Provide a supervisor that receives a VNode module implementation (i.e. a module that implements the
    vnode behaviour)

  • For coverage commands, a FSM + Supervisor provided by us should be enough. If it is needed to be more general, we'll see it.

  • Hide VNode weird details like described here. Probably something like the use macro will be helpful here.

Consider removing fork dependencies

Riak Core is hosted on GitHub, and so are its dependencies, so we can't upload Riax to hex.pm for the time being.
We have some options for this:

  • Upload Riak Core and its dependencies to hex.pm by ourselves. Some dependencies like Exometer are actually hosted on hex.pm, but Riak's upstream is using a fork of its own,
  • Use a fork, like Riak Core Lite. This is actually a really good fork,
    and it's being used by AntidoteDB but it's lacking some features like Coverage Commands.

Ease Riak Setup and usage

There are some things that we can do better and that would be good to address as we're planning to provide
an elixir library for riak, like:

  • Avoiding weird Erlang macros like this one:
handle_handoff_command(?FOLD_REQ{foldfun=FoldFun, acc0=Acc0}, _Sender,
                       State = #{data := Data}) ->
  log("Received fold request for handoff", State),
  Result = maps:fold(FoldFun, Acc0, Data),
  {reply, Result, State};

Currently I'm using this as a workaround:

  require Record

  Record.defrecord(
    :fold_req_v2,
    :riak_core_fold_req_v2,
    Record.extract(:riak_core_fold_req_v2, from_lib: "riak_core/include/riak_core_vnode.hrl")
  )
  • See If we can make the VMaster setup more straight forward.
  • See If we can make the FSM Coverage setup more straight forward.
  • Maybe provide an Elixir specific VNode behaviour that wraps the original one.

Add ex_docs

Although I'm adding documentation to each module and function,
I'd like to add ex_docs to generate html documentation.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.