Giter VIP home page Giter VIP logo

taas-es-processor's Introduction

Taas ES Processor

Dependencies

  • Nodejs(v12+)
  • ElasticSearch
  • Kafka

Configuration

Configuration is at config/default.js. The following parameters can be set in config files or in env variables:

  • PORT: port number the health check dropin listnering on

  • LOG_LEVEL: the log level

  • KAFKA_URL: comma separated Kafka hosts

  • KAFKA_CLIENT_CERT: Kafka connection certificate, optional; if not provided, then SSL connection is not used, direct insecure connection is used; if provided, it can be either path to certificate file or certificate content

  • KAFKA_CLIENT_CERT_KEY: Kafka connection private key, optional; if not provided, then SSL connection is not used, direct insecure connection is used; if provided, it can be either path to private key file or private key content

  • KAFKA_GROUP_ID: the Kafka group id

  • topics.TAAS_JOB_CREATE_TOPIC: the create job entity Kafka message topic

  • topics.TAAS_JOB_UPDATE_TOPIC: the update job entity Kafka message topic

  • topics.TAAS_JOB_DELETE_TOPIC: the delete job entity Kafka message topic

  • topics.TAAS_JOB_CANDIDATE_CREATE_TOPIC: the create job candidate entity Kafka message topic

  • topics.TAAS_JOB_CANDIDATE_UPDATE_TOPIC: the update job candidate entity Kafka message topic

  • topics.TAAS_JOB_CANDIDATE_DELETE_TOPIC: the delete job candidate entity Kafka message topic

  • topics.TAAS_RESOURCE_BOOKING_CREATE_TOPIC: the create resource booking entity Kafka message topic

  • topics.TAAS_RESOURCE_BOOKING_UPDATE_TOPIC: the update resource booking entity Kafka message topic

  • topics.TAAS_RESOURCE_BOOKING_DELETE_TOPIC: the delete resource booking entity Kafka message topic

  • topics.TAAS_WORK_PERIOD_CREATE_TOPIC: the create work period entity Kafka message topic

  • topics.TAAS_WORK_PERIOD_UPDATE_TOPIC: the update work period entity Kafka message topic

  • topics.TAAS_WORK_PERIOD_DELETE_TOPIC: the delete work period entity Kafka message topic

  • topics.TAAS_WORK_PERIOD_PAYMENT_CREATE_TOPIC: the create work period payment entity Kafka message topic

  • topics.TAAS_WORK_PERIOD_PAYMENT_UPDATE_TOPIC: the update work period payment entity Kafka message topic

  • topics.TAAS_INTERVIEW_REQUEST_TOPIC: the request interview entity Kafka message topic

  • topics.TAAS_INTERVIEW_UPDATE_TOPIC: the update interview entity Kafka message topic

  • topics.TAAS_INTERVIEW_BULK_UPDATE_TOPIC: the bulk update interview entity Kafka message topic

  • topics.TAAS_ROLE_CREATE_TOPIC: the create role entity Kafka message topic

  • topics.TAAS_ROLE_UPDATE_TOPIC: the update role entity Kafka message topic

  • topics.TAAS_ROLE_DELETE_TOPIC: the delete role entity Kafka message topic

  • esConfig.HOST: Elasticsearch host

  • esConfig.AWS_REGION: The Amazon region to use when using AWS Elasticsearch service

  • esConfig.ELASTICCLOUD.id: The elastic cloud id, if your elasticsearch instance is hosted on elastic cloud. DO NOT provide a value for ES_HOST if you are using this

  • esConfig.ELASTICCLOUD.username: The elastic cloud username for basic authentication. Provide this only if your elasticsearch instance is hosted on elastic cloud

  • esConfig.ELASTICCLOUD.password: The elastic cloud password for basic authentication. Provide this only if your elasticsearch instance is hosted on elastic cloud

  • esConfig.ES_INDEX_JOB: the index name for job

  • esConfig.ES_INDEX_JOB_CANDIDATE: the index name for job candidate

  • esConfig.ES_INDEX_RESOURCE_BOOKING: the index name for resource booking

  • esConfig.ES_INDEX_ROLE: the index name for role

  • auth0.AUTH0_URL: Auth0 URL, used to get TC M2M token

  • auth0.AUTH0_AUDIENCE: Auth0 audience, used to get TC M2M token

  • auth0.AUTH0_CLIENT_ID: Auth0 client id, used to get TC M2M token

  • auth0.AUTH0_CLIENT_SECRET: Auth0 client secret, used to get TC M2M token

  • auth0.AUTH0_PROXY_SERVER_URL: Proxy Auth0 URL, used to get TC M2M token

  • zapier.ZAPIER_COMPANYID_SLUG: your company id in zapier; numeric value

  • zapier.ZAPIER_CONTACTID_SLUG: your contact id in zapier; numeric value

  • zapier.ZAPIER_SWITCH: decides whether posting job related message to zapier or not; possible values are ON and OFF, default is OFF

  • zapier.ZAPIER_WEBHOOK: the remote zapier zap webhook url for posting job related message

  • zapier.ZAPIER_JOB_CANDIDATE_SWITCH: decides whether posting job candidate related message to zapier or not; possible values are ON and OFF, default is OFF

  • zapier.ZAPIER_JOB_CANDIDATE_WEBHOOK: the remote zapier zap webhook url for posting job candidate related message

Local Kafka and ElasticSearch setup

  1. Navigate to the directory local

  2. Run the following command

    docker-compose up -d
  3. initialize Elasticsearch, create configured Elasticsearch index:

    npm run delete-index # run this if you already created index
    npm run create-index

Local deployment

  1. Make sure that Kafka and Elasticsearch is running as per instructions above.

  2. From the project root directory, run the following command to install the dependencies

    npm install
  3. To run linters if required

    npm run lint

    To fix possible lint errors:

    npm run lint:fix
  4. Start the processor and health check dropin

    npm start

Local Deployment with Docker

To run the processor using docker, follow the below steps

  1. Navigate to the directory docker

  2. Rename the file sample.api.env to api.env

  3. Set the required Kafka url and ElasticSearch host in the file api.env.

    Note that you can also add other variables to api.env, with <key>=<value> format per line. If using AWS ES you should add AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables as well.

  4. Once that is done, run the following command

    docker-compose up
  5. When you are running the application for the first time, It will take some time initially to download the image and install the dependencies

Unit Tests and E2E Tests

Unit Tests

  • Run npm run test to execute unit tests.
  • Run npm run test:cov to execute unit tests and generate coverage report.

E2E Tests

Before running e2e tests, make sure index are created and the processor app is not running. Existing documents will be remove from ES before and after tests.

  • RUN npm run e2e to execute e2e tests.
  • RUN npm run e2e:cov to execute e2e tests and generate coverage report.

Verification

see VERIFICATION.md

taas-es-processor's People

Contributors

cagdas001 avatar eisbilir avatar imcaizheng avatar maxceem avatar narekcat avatar nikolay83 avatar nkumar-topcoder avatar sushilshinde avatar urwithat avatar veshu avatar xxcxy avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.