Giter VIP home page Giter VIP logo

hive-plug-play's Introduction

Hive Plug & Play

Customizable block streaming and parsing microservice for custom_json ops on Hive.

Documentation

Development

Dependencies:

  • Python 3.6 required (with pip, libpq-dev, and Psycopg2)
  • PostgreSQL 10+

Install depencencies

  • Python3 and PostgreSQL :
sudo apt install python3 python3-pip libpq-dev python3-psycopg2 postgresql

Configure PostgreSQL:

  • Configure postgresql for remote [Authentication]
  • OR
    For default postgres installs on localhost, at a minimum CHANGE THE PASSWORD! a) Update the password for the Linux postgresql User acount
    sudo passwd postgres
    
    b) change to the postgres user
    sudo -i -u postgres
    
    c) start the postgresql database/service
    sudo systemctl start postgresql
    psql # Launch posgresql cli
    
    d) change the password in postgresql
    \password
    \q 
    

Restart the PosgreSQL service

```
exit # quit to linux original user account
sudo systemctl stop postgresql
sudo systemctl start postgresql
# Check status with 
sudo systemctl status postgresql
```

Create empty database

sudo -i -u postgres
psql # Launch posgresql cli
CREATE DATABASE plug_play;
GRANT ALL PRIVILEGES ON DATABASE plug_play TO postgres;
# Verify database was created with list
\l 
# Exit postgresql cli
\q

Configure Hive Plug & Play

In terminal login to your postgres account sudo -i -u postgres

TLDR build config.ini file: This one-liner from the terminal creates the required config.ini file, sets the environment variable and opens nano to edit.
Make your updates then use ctl+s to save and ctl+x to close.

mkdir -p ~/.config/hive-plug-play && export PLUG_PLAY_HOME=~/.config/hive-plug-play && ([ -f ~/.config/hive-plug-play/config.ini ] || echo $'db_username=postgres\ndb_password=password\nserver_host=127.0.0.1\nserver_port=5432\nssl_cert=\nssl_key=\nstart_block=54000984\nop_ids=["podping","hive-hydra"]' > ~/.config/hive-plug-play/config.ini) && nano ~/.config/hive-plug-play/config.ini

OR step by step build config.ini file:

  1. Hive Plug & Play requires a config.ini file to exist in either: - Default file location of /etc/hive-plug-play - Or use any custom folder by setting an environment variable: export PLUG_PLAY_HOME=~/.config/hive-plug-play.
  2. Build the file directory:
mkdir -p ~/.config/hive-plug-play
  1. Create the config.ini file - Any text editor should do:
db_username=postgres
db_password=password
server_host=127.0.0.1
server_port=5432
ssl_cert=
ssl_key=
start_block=54000984
op_ids=["community","notify"]

Configure data collection for a specific project

Note The following are some examples of know op_ids per project, this list is not all inclusive, and is likely missing ids for the projects listed, additional fields are required to be reviewed to get the correct project.

Project Name op_ids
reblog post reblog
podping earliest valid block:54000984 podping,hive-hydra
3speak 3speak-publish
actifit actifit
peakd peakd_notify
DCity dcity, dcity-bg-save, dcitystats
splinderlands sm_accept_challenge, sm_add_wallet, sm_advance_league, sm_burn_cards, sm_cancel_match, sm_cancel_sell, sm_card_update, sm_claim_airdrop, sm_claim_reward, sm_combine_all, sm_combine_cards, sm_create_tournament, sm_decline_challenge, sm_delegate_cards, sm_edit_guild, sm_enter_tournament, sm_external_payment, sm_find_match, sm_gift_cards, sm_gift_packs, sm_guild_accept, sm_guild_brawl_settings, sm_guild_contribution, sm_guild_decline, sm_guild_invite, sm_guild_promote, sm_guild_remove, sm_join_guild, sm_leave_guild, sm_leave_tournament, sm_lock_assets, sm_market_purchase, sm_open_all, sm_open_pack, sm_price_feed, sm_purchase, sm_purchase_dice, sm_purchase_land, sm_purchase_skin_set, sm_refresh_quest, sm_sell_cards, sm_set_authority, sm_start_match, sm_start_quest, sm_submit_team, sm_surrender, sm_team_reveal, sm_token_award, sm_token_transfer, sm_undelegate_cards, sm_unlock_assets, sm_update_authority, sm_update_price, sm_upgrade_account
CBM cbm__backpack__drink_beer, cbm__balance__deposit, cbm__building__rent, cbm__building__restore_condition, cbm__craft__claim, cbm__craft__finish_now, cbm__craft__start, cbm__daily_quests__claim, cbm__daily_quests__finish_now, cbm__daily_quests__start, cbm__enhancer__claim, cbm__enhancer__start, cbm__market__completed_purchase, cbm__market__completed_sale, cbm__market__placed_a_sell_order, cbm__pub__drink_beer, cbm__pub__sold_beer, cbm__referral__claim
many more exist (hundreds)

Installation:

  • Download from github and install as a python package from directory
    sudo -i -u postgres
    cd ~/
    # This example is for the podping project
    # git clone [email protected]:imwatsi/hive-plug-play.git
    git clone https://github.com/seakintruth/hive-plug-play.git
    cd ~/hive-plug-play
    git checkout podping
    pip3 install -e .
    
    

Run:

From command: Change user to your postgres account su postgres (this shouldn't be required if authenticating via certificate) hive_plug_play

Or from dir:

  • cd hive_plug_play
  • python3 run.py

Resource Links (Developer documentation)

hive-plug-play's People

Contributors

imwatsi avatar seakintruth avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

seakintruth

hive-plug-play's Issues

Custom views fail to create

On build/run the custom (podping) views fail to create in the database.

Additionally creation of these custom views should be tried to either a new configuration variable or maybe just build the custom app views based on logic of what's in op_ids.

sudo required for first setup on new

Updating README.md with these dependencies removes the sudo requirement:

Dependencies:

  • Python 3.6 required (with pip, libpq-dev, and Psycopg2)
  • PostgreSQL 10+

Install depencencies

  • Python3 and PostgreSQL :
sudo apt install python3 python3-pip libpq-dev python3-psycopg2 postgresql

these are the errors I get on a new install if attempting to install without a sudo command on Ubuntu 20.04

  • Building wheels for collected packages: psycopg2
postgres@<MACHINENAME>:~/hive-plug-play$   pip3 install -e .
Obtaining file:///var/lib/postgresql/hive-plug-play
Requirement already satisfied: aiohttp in /var/lib/postgresql/.local/lib/python3.8/site-packages (from hive-plug-play==0.2.1) (3.7.4.post0)
Requirement already satisfied: jsonrpcserver in /var/lib/postgresql/.local/lib/python3.8/site-packages (from hive-plug-play==0.2.1) (4.2.0)
Collecting psycopg2
  Using cached psycopg2-2.8.6.tar.gz (383 kB)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from hive-plug-play==0.2.1) (2.22.0)
Requirement already satisfied: attrs>=17.3.0 in /var/lib/postgresql/.local/lib/python3.8/site-packages (from aiohttp->hive-plug-play==0.2.1) (21.2.0)
Requirement already satisfied: chardet<5.0,>=2.0 in /usr/lib/python3/dist-packages (from aiohttp->hive-plug-play==0.2.1) (3.0.4)
Requirement already satisfied: yarl<2.0,>=1.0 in /var/lib/postgresql/.local/lib/python3.8/site-packages (from aiohttp->hive-plug-play==0.2.1) (1.6.3)
Requirement already satisfied: typing-extensions>=3.6.5 in /var/lib/postgresql/.local/lib/python3.8/site-packages (from aiohttp->hive-plug-play==0.2.1) (3.10.0.0)
Requirement already satisfied: async-timeout<4.0,>=3.0 in /var/lib/postgresql/.local/lib/python3.8/site-packages (from aiohttp->hive-plug-play==0.2.1) (3.0.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /var/lib/postgresql/.local/lib/python3.8/site-packages (from aiohttp->hive-plug-play==0.2.1) (5.1.0)
Requirement already satisfied: apply-defaults<1 in /var/lib/postgresql/.local/lib/python3.8/site-packages (from jsonrpcserver->hive-plug-play==0.2.1) (0.1.4)
Requirement already satisfied: jsonschema<4,>=2 in /var/lib/postgresql/.local/lib/python3.8/site-packages (from jsonrpcserver->hive-plug-play==0.2.1) (3.2.0)
Requirement already satisfied: idna>=2.0 in /usr/lib/python3/dist-packages (from yarl<2.0,>=1.0->aiohttp->hive-plug-play==0.2.1) (2.8)
Requirement already satisfied: pyrsistent>=0.14.0 in /var/lib/postgresql/.local/lib/python3.8/site-packages (from jsonschema<4,>=2->jsonrpcserver->hive-plug-play==0.2.1) (0.17.3)
Requirement already satisfied: six>=1.11.0 in /usr/lib/python3/dist-packages (from jsonschema<4,>=2->jsonrpcserver->hive-plug-play==0.2.1) (1.14.0)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from jsonschema<4,>=2->jsonrpcserver->hive-plug-play==0.2.1) (45.2.0)
Building wheels for collected packages: psycopg2
  Building wheel for psycopg2 (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kjvyaap9/psycopg2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-kjvyaap9/psycopg2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-sdn3zenw
       cwd: /tmp/pip-install-kjvyaap9/psycopg2/
  Complete output (40 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/psycopg2
  copying lib/_json.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/errorcodes.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/extras.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/sql.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/__init__.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/extensions.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/tz.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/_range.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/compat.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/_ipaddress.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/_lru_cache.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/pool.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/errors.py -> build/lib.linux-x86_64-3.8/psycopg2
  running build_ext
  building 'psycopg2._psycopg' extension
  creating build/temp.linux-x86_64-3.8
  creating build/temp.linux-x86_64-3.8/psycopg
  x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSYCOPG_VERSION=2.8.6 (dt dec pq3 ext lo64) -DPG_VERSION_NUM=120007 -DHAVE_LO64=1 -I/usr/include/python3.8 -I. -I/usr/include/postgresql -I/usr/include/postgresql/12/server -c psycopg/psycopgmodule.c -o build/temp.linux-x86_64-3.8/psycopg/psycopgmodule.o -Wdeclaration-after-statement
  In file included from psycopg/psycopgmodule.c:28:
  ./psycopg/psycopg.h:36:10: fatal error: libpq-fe.h: No such file or directory
     36 | #include <libpq-fe.h>
        |          ^~~~~~~~~~~~
  compilation terminated.
  
  It appears you are missing some prerequisite to build the package from source.
  
  You may install a binary package by installing 'psycopg2-binary' from PyPI.
  If you want to install psycopg2 from source, please install the packages
  required for the build and try again.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <https://www.psycopg.org/docs/install.html>).
  
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for psycopg2
  Running setup.py clean for psycopg2
Failed to build psycopg2
Installing collected packages: psycopg2, hive-plug-play
    Running setup.py install for psycopg2 ... error
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kjvyaap9/psycopg2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-kjvyaap9/psycopg2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-o59l1obs/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /var/lib/postgresql/.local/include/python3.8/psycopg2
         cwd: /tmp/pip-install-kjvyaap9/psycopg2/
    Complete output (40 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/psycopg2
    copying lib/_json.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/errorcodes.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/extras.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/sql.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/__init__.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/extensions.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/tz.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/_range.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/compat.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/_ipaddress.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/_lru_cache.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/pool.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/errors.py -> build/lib.linux-x86_64-3.8/psycopg2
    running build_ext
    building 'psycopg2._psycopg' extension
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/psycopg
    x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSYCOPG_VERSION=2.8.6 (dt dec pq3 ext lo64) -DPG_VERSION_NUM=120007 -DHAVE_LO64=1 -I/usr/include/python3.8 -I. -I/usr/include/postgresql -I/usr/include/postgresql/12/server -c psycopg/psycopgmodule.c -o build/temp.linux-x86_64-3.8/psycopg/psycopgmodule.o -Wdeclaration-after-statement
    In file included from psycopg/psycopgmodule.c:28:
    ./psycopg/psycopg.h:36:10: fatal error: libpq-fe.h: No such file or directory
       36 | #include <libpq-fe.h>
          |          ^~~~~~~~~~~~
    compilation terminated.
    
    It appears you are missing some prerequisite to build the package from source.
    
    You may install a binary package by installing 'psycopg2-binary' from PyPI.
    If you want to install psycopg2 from source, please install the packages
    required for the build and try again.
    
    For further information please check the 'doc/src/install.rst' file (also at
    <https://www.psycopg.org/docs/install.html>).
    
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kjvyaap9/psycopg2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-kjvyaap9/psycopg2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-o59l1obs/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /var/lib/postgresql/.local/include/python3.8/psycopg2 Check the logs for full command output.

On first run, expects 'plug_play' database to exist.

--- Hive Plug & Play started ---
No database found. Please create a 'plug_play' database in PostgreSQL.

Initial Python script could check if the database exists, if not then create it:
from stackoverflow

cursor.execute("SELECT 1 FROM pg_catalog.pg_database WHERE datname = 'python_db'")
exists = cursor.fetchone()
if not exists:
    cursor.execute('CREATE DATABASE python_db')

Microfork recovery

Currently, Hive Plug & Play trails the head_block by 10 blocks, to give a safe allowance for last_irreversible_block. I'm working on microfork recovery to allow it to trail as close to head_block as possible.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.