blockchain-etl / blockchain-etl-streaming Goto Github PK

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

Home Page: https://medium.com/google-cloud/live-ethereum-and-bitcoin-data-in-google-bigquery-and-pub-sub-765b71cd57b5

License: MIT License

Shell 11.93% Python 59.40% Mustache 28.67%

real-time bitcoin ethereum cryptocurrency apache-beam blockchain blockchain-analytics crypto data-analytics data-engineering

blockchain-etl-streaming's Introduction

Blockchain ETL Streaming

Streams the following Ethereum entities to Pub/Sub or Postgres using ethereum-etl stream:

blocks
transactions
logs
token_transfers
traces
contracts
tokens

Streams blocks and transactions to Pub/Sub using bitcoin-etl stream. Supported chains:

bitcoin
bitcoin_cash
dogecoin
litecoin
dash
zcash

Deployment Instructions

Create a cluster:

gcloud container clusters create ethereum-etl-streaming \
--zone us-central1-a \
--num-nodes 1 \
--disk-size 10GB \
--machine-type custom-2-4096 \
--network default \
--subnetwork default \
--scopes pubsub,storage-rw,logging-write,monitoring-write,service-management,service-control,trace

Get kubectl credentials:

gcloud container clusters get-credentials ethereum-etl-streaming \
--zone us-central1-a

Create Pub/Sub topics (use create_pubsub_topics_ethereum.sh). Skip this step if you need to stream to Postgres.

"crypto_ethereum.blocks"
"crypto_ethereum.transactions"
"crypto_ethereum.token_transfers"
"crypto_ethereum.logs"
"crypto_ethereum.traces"
"crypto_ethereum.contracts"
"crypto_ethereum.tokens"

Create GCS bucket. Upload a text file with block number you want to start streaming from to gs://<YOUR_BUCKET_HERE>/ethereum-etl/streaming/last_synced_block.txt.
Create "ethereum-etl-app" service account with roles:
- Pub/Sub Editor
- Storage Object Admin
- Cloud SQL Client

Download the key. Create a Kubernetes secret:

kubectl create secret generic streaming-app-key --from-file=key.json=$HOME/Downloads/key.json -n eth

Install [helm] (https://github.com/helm/helm#install)

brew install helm
helm init  
bash patch-tiller.sh

Copy example values directory to values dir and adjust all the files at least with your bucket and project ID.
Install ETL apps via helm using chart from this repo and values we adjust on previous step, for example:

helm install --name btc --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/bitcoin/values.yaml
helm install --name bch --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/bitcoin_cash/values.yaml
helm install --name dash --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/dash/values.yaml
helm install --name dogecoin --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/dogecoin/values.yaml
helm install --name litecoin --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/litecoin/values.yaml
helm install --name zcash --namespace btc charts/blockchain-etl-streaming --values values/bitcoin/zcash/values.yaml

helm install --name eth-blocks --namespace eth charts/blockchain-etl-streaming \ 
--values values/ethereum/values.yaml --values values/ethereum/block_data/values.yaml
helm install --name eth-traces --namespace eth charts/blockchain-etl-streaming \ 
--values values/ethereum/values.yaml --values values/ethereum/trace_data/values.yaml

Ethereum block and trace data streaming are decoupled for higher reliability.

To stream to Postgres:

helm install --name eth-postgres --namespace eth charts/blockchain-etl-streaming \ 
--values values/ethereum/values-postgres.yaml

Refer to https://github.com/blockchain-etl/ethereum-etl-postgres for table schema and initial data load.

Use describe command to troubleshoot, f.e.:

kubectl describe pods -n btc
kubectl describe node [NODE_NAME]

Refer to blockchain-etl-dataflow for connecting Pub/Sub to BigQuery.

blockchain-etl-streaming's People

Contributors

Stargazers

Watchers

blockchain-etl-streaming's Issues

Current block lags

Current block is not being updated on the 10 seconds interval?

Implement Kubernetes manifests for ethereum-etl-streaming

A single Pod runs:

download-last_synced_block - Job that downloads last_synced_block.txt from GCS bucket on node start up. See also init containers.
streaming_service.py - streams blockchain data to PubSub
health_checker.py - listens on HTTP port for health check requests
gcs-syncd - uploads last_synced_block.txt periodically to GCS bucket and has a preStop hook that uploads the file to GCS bucket before Pod is terminated.
fluentd daemon - uploads logs to GCS bucket

====================

Some code can be found here https://github.com/airswap/ethereum-etl/commit/cc7f86e139d317452c9cf0c975b99feb6ed8d7a0#diff-aa0ae5de65a67b108e5a26ddde9d3adf

minor - gcs_prefix inconsistent with readme.md

GCS_PREFIX in example_values/ethereum/trace_data/values.yaml inconsistent w/ readme step 4 (ethereum-etl/streaming/last_synced_block.txt)

GCS_PREFIX should read ethereum-etl/streaming

Investigate PubSub push subscription error: The supplied HTTP URL is not registered in the project that owns the subscription

There is no way to connect a cloud function with a PubSub in a different project:

Note: The Cloud Pub/Sub topic that your function is subscribed to must be in the same Google Cloud Platform project as your Cloud Function.

There is a way to use HTTP-triggered subscription to invoke a cloud function:

Note: You can also use HTTP-triggered functions to listen to Pub/Sub push subscriptions. This allows a single Cloud Function to subscribe to multiple topics.

However when trying to create an HTTP-triggered subscription an error occurs:

PubSub push subscription error: The supplied HTTP URL is not registered in the project that owns the subscription

It seems there is a need to register the endpoint: https://cloud.google.com/pubsub/docs/push#other-endpoints

This task is to follow the registration flow and see how simple/hard it is.

Implement streaming_service.py

Streaming service

Continuously polls for new blocks using Ethereum JSON RPC API (e.g. every 10 seconds).
Outputs data to GooglePubSubItemExporter, which publishes data to PubSub.
Saves the last synchronized block to last_sync_block.txt file on every period.

This script can be used as a basis https://github.com/blockchain-etl/ethereum-etl/blob/feature/streaming/stream.py

================================

PubSub publisher guide https://cloud.google.com/pubsub/docs/publisher

There is some code in this fork https://github.com/airswap/ethereum-etl/commit/cc7f86e139d317452c9cf0c975b99feb6ed8d7a0#diff-a8e6a25bc32464a59378bac1befac9eb

TODO: Consider using geth subscriptions to handle chain reorganizations https://github.com/ethereum/go-ethereum/wiki/RPC-PUB-SUB#newheads

Add AWS compatibility

Is it possible to add S3 storage, kinesis data pipeline, as output or storage of current state ?

Implement health_checker.py

Health checker:

Listens on an HTTP port for health check requests.
Checks last_sync_block.txt file produced by streaming_service.py, looks at its last modified date .
Reports UNHEALTHY if last_sync_block.txt was modified more than 2 minutes ago.
Has minimal dependencies.

Delete everything related to EOS

EOS is not used at this point.

When the pod is deleted in Kubernetes it can't terminate gracefully and is terminated forcefully after 30 seconds timeout

Steps to reproduce:

Deploy one of streamers e.g. ethereum block data.
Delete the pod.
Observe that it takes 30 seconds to delete the pod and the return status is 127.

The suspected cause is the upload-last-synced-block-file container can't terminate gracefully for some reason.

blockchain-etl / blockchain-etl-streaming Goto Github PK

blockchain-etl-streaming's Introduction

Blockchain ETL Streaming

Deployment Instructions

blockchain-etl-streaming's People

Contributors

Stargazers

Watchers

Forkers

blockchain-etl-streaming's Issues

Recommend Projects

Recommend Topics

Recommend Org