Giter VIP home page Giter VIP logo

haskell-tools's Introduction

Haskell-based CoinMetrics.io tools

These tools are used by CoinMetrics.io team for exporting data from blockchains into analytical databases and monitoring full nodes synchronization state.

Utilities

  • coinmetrics-export - utility for exporting data from blockchains in formats suitable for inserting into analytics databases (SQL, Avro).
  • coinmetrics-monitor - utility for monitoring blockchain nodes and providing Prometheus-compatible metrics.

Status

The project is being used in production at Coin Metrics, but many things are in flux or fragile, and docs may be outdated. Command line interface is more or less stable but may change. Please use with caution.

Supported blockchains:

Binaries

There're no stable releases yet. All binaries are "bleeding edge".

One easy way to run the tools is to use docker.

Pull the latest version:

docker pull coinmetrics/haskell-tools

Run e.g. coinmetrics-export tool:

docker run -it --rm --net host coinmetrics/haskell-tools coinmetrics-export <arguments>

Building from source

Building with Nix is recommended, because dependencies are fetched and built automatically.

With Nix

nix build -Lf ./release.nix bins

With Stack + Nix

stack build --nix

With Stack

stack build

Required dependencies: zlib, libpq.

Stellar export also requires xdrpp and Stellar XDR headers built with XDR compiler, which may be hard to build manually. By default Stellar support is enabled, so build will fail without them. You can disable Stellar support if you don't need it: stack build --flag coinmetrics-all-blockchains:-stellar. If you do need it, please use one of the Nix build methods above (with Stack or without).

The code is only tested on Linux (but maybe works on other OSes too).

haskell-tools's People

Contributors

centromere avatar nikitarzm avatar quyse avatar whateverpal avatar znack avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

haskell-tools's Issues

Generate docs

Produce generated documentation with up-to-date export schemas and man pages.

Support unified output format

Implement unified form of data for all blockchains, support for every output format (via additional flag --unified).

Refactor storage

Refactor storage-specific code from export utility, introduce abstractions for all concepts (--continue, --heal, etc). Also give developer a choice of supported storages (it affects set of runtime dependencies).

Support for flat schema

Initial support for flat schema (i.e. with separate table for blocks, transactions, logs, etc) is implemented. This issue is to track implementation status of this feature.

Flat schema is very experimental and very briefly tested at the moment. Enabled with --flat.

Should work for blockchains:

  • Bitcoin
  • Cardano
  • EOS
  • Ethereum
  • N/A IOTA
  • Monero
  • Nem
  • Neo
  • Ripple
  • Stellar
  • Waves

With output types:

  • AVRO
  • ElasticSearch
  • PostgreSQL

print-schema command is not implemented yet for flat schemas.

Output to ElasticSearch

  • Force all block types to have JSON representation.
  • Support for print-schema
  • Support for --flat

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.