Giter VIP home page Giter VIP logo

elasticsearch-postgres's Introduction

PostgreSQL To ElasticSearch in Go

Requirements

Project in Go + SQL to create an ElasticSearch index from PostgreSQL database.

This procedure is for a Linux operating system.

The following dependencies needs to be previously installed :

  • PostgreSQL (database)
  • ElasticSearch (index)
  • Docker (containerizer)
  • PG Admin (Sql manager)

Two ways to start listening:

  1. Via PostGresToES.go: this will save the raw data from the database
  2. Via PostGresToES_ApiGateway.go: this will query the API in order to retrieve JSON which will be indexed in ES

Nothing changes in terms of triggers

Installation

  1. Docker : https://docs.docker.com/install/
  2. PostgreSQL
docker pull postgres
docker run --name abcd -e POSTGRES_PASSWORD=abcd -d postgres
  1. PG Admin
docker pull thajeztah/pgadmin4
docker run --restart=always -p 5050:5050 thajeztah/pgadmin4
  1. ElasticSearch
docker pull elasticsearch
docker run --restart=always -d -p 9200:9200 -p 9300:9300 -it -h elasticsearch --name elasticsearch elasticsearch

Configure :

  1. Create index :
PUT http://localhost:9200/<INDEX_NAME>
{
    "settings" : {
        "index" : {
            "number_of_shards" : 5,
            "number_of_replicas" : 2
        }
    }
}
OR :
curl -XPUT  'http://localhost:9200/<INDEX_NAME>' -d '{"settings" : {"number_of_shards" : 5, "number_of_replicas" : 2}}'
  1. Create the PostgreSql function :
CREATE OR REPLACE FUNCTION public.notify_event()
    RETURNS trigger
    LANGUAGE 'plpgsql'
    COST 100
    VOLATILE NOT LEAKPROOF
AS $BODY$

    DECLARE
        data json;
        notification json;
        id integer;
    BEGIN

        -- Convert the old or new row to JSON, based on the kind of action.
        -- Action = DELETE?             -> OLD row
        -- Action = INSERT or UPDATE?   -> NEW row
        IF (TG_OP = 'DELETE') THEN
            data = row_to_json(OLD);
            id = OLD.id;
        ELSE
            data = row_to_json(NEW);
            id = NEW.id;
        END IF;

        -- Contruct the notification as a JSON string.
        notification = json_build_object(
                          'table',TG_TABLE_NAME,
                          'action', TG_OP,
                          'id', id,
                          'data', data);

        -- Execute pg_notify(channel, notification)
        PERFORM pg_notify('events',notification::text);

        -- Result is ignored since this is an AFTER trigger
        RETURN NULL;
    END;
$BODY$;
  1. Create the PostgreSql trigger :
CREATE TRIGGER products_notify_event
    AFTER INSERT OR DELETE OR UPDATE
    ON public.users
    FOR EACH ROW
    EXECUTE PROCEDURE public.notify_event();
  1. Install Go https://go.dev/doc/install
  2. Init go.mod go mod init PostGresToES
  3. Install dependencies go mod tidy
  4. Setup (auth, indexName) : https://github.com/120dev/ElasticSearch-postgres/blob/master/PostGresToES.go#L20
  5. if PostGresToES_ApiGateway.go : setup conf.yaml
  6. Build go build PostGresToES.go && chmod +x ./PostGresToES or go build PostGresToES_ApiGateway.go && chmod +x ./PostGresToES_ApiGateway
  7. Run Go script
./PostGresToES or ./PostGresToES_ApiGateway

And wait, all events are logged.

UML diagrams

You can render UML diagrams using Mermaid. For example, this will produce a sequence diagram:

sequenceDiagram
PostgreSQL ->> Trigger: Event crud in Json
Trigger ->> Go: Read and set ES _id
Go ->> ES: Post Json to ES
Loading

elasticsearch-postgres's People

Contributors

120dev avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.