Giter VIP home page Giter VIP logo

project-ias's Introduction

Setup up locally

Setting up React Frontend

  1. Change the directory to frontend - cd frontend
  2. Install node modules using yarn - yarn
  3. Start the react server - yarn start

Setting up Node JS backend

  1. Change the directory to backend
  2. Install node modules using npm - npm install
  3. Start the Node JS server using nodemon - nodemon index.js

Note: The frontend is connected to the MeiliSearch hosted on the Server, which is filled with the Updated Data. This is done so that developer doesn't have to refill the Database on his/her own. If you want to setup the database, here are the instructions

Setting up MeiliSearch

MeiliSearch is the Search Database, where we store all our PYQs, DNS and Reading Content. To intall and Launch Meilisearch

# Install MeiliSearch
curl -L https://install.meilisearch.com | sh

# Launch MeiliSearch
./meilisearch

Adding Documents to MeiliSearch

The data has been stored in the form of JSON files in the backend/pyq-scrapers folder. There are some JS scripts in backend folder in the format of add_*_to_ms.js. You can run the scripts individually to add documents to MeiliSearch. If you wish to add more documents in one of the existing indexes

  1. Scrape the Data and store it in a JSON file in the backend/pyq-scrapers
  2. Find the appropirate file in backend. For example, if you want to add dns documents the filename would be add_dns_to_ms.js
  3. Change the JSON file name in the file found in the above step.
  4. Run the JS file to add the data in MeiliSearch - node FILENAME.js Note: Please check to which MeiliSearch DB you are adding the DB - Local MeiliSearch DB or Production MeiliSearch DB

Deleting an index

  1. Change the indexName in this file
  2. Run the file - node delete_index.js

Deployment

  • Frontend, backend and MeiliSearch all are deployed on t2 micro instance.
  • Frontend build is served by Nginx
  • Backend runs in background via pm2
  • Meilisearch is hosted via ngrok

To deploy

  • If you want your changes(both frontend and backend) to be deployed. SSH to the server. Run the deploy.sh script present in the home directory.

Project Structure

  • In frontend directory
    • src/components contains Components to be displayed
    • All constants are in src/cnstants/constants.js
  • In backend directory
    • index.js contains the server code
    • Other *.js files add JSON data to MeiliSearch
    • pyq_scrapers contains python files to scrape data, along with JSON files containing the data

Cronjob

Flow:

  1. Using Github Actions, an endpoint is hit once a day
  2. The endpoint runs a bash script
  3. The bash scripts first runs a Python Script, then a JS script
  4. The Python Scripts creates a JSON file
  5. The JS Script adds the data in JSON file to MeiliSearch

Below are the related files for 2 cronjobs - Rau DNS, Dhristi Content

Updating Rau DNS Daily

Updating Dhristi Content Daily

project-ias's People

Contributors

chirag-ghosh avatar rakaar avatar sahil-shubham avatar twistingtwists avatar hargup avatar

Watchers

James Cloos avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.