Giter VIP home page Giter VIP logo

twitter-indexer's Introduction

Small project to index archived trends and tweets

The data is from Twitter API V1.0, this code is not compatible with V2.0.

Internal archive format for Tweets : [timestamp_archived_date, tweet_id/internal_code, tweet_data from REST API ] \

Internal archive format for Trends : [timestamp_archived_date, internal_code, trends_data from REST API ]

Running elasticsearch and kibana on docker :

#create a network named elastic
docker network create elastic

docker run -d  --name elasticsearch --network elastic  -p 9200:9200 -p 9300:9300 -v  /rex/ssd/zpehlivan/twitter_2018-03/elastic/data:/usr/share/elasticsearch/data -e xpack.security.enabled=false -e discovery.type=single-node    docker.rech.ina.fr/elasticsearch:7.5.2
docker run -d  --name kb-container --network elastic  -p 5601:5601 -e ELASTICSEARCH_HOSTS=http://elasticsearch:9200 docker.rech.ina.fr/kibana/kibana:7.5.2

Running indexer on docker :

  • Update config file

  • Run temp-docker https://github.com/ZeynepP/DockerTemp with correct mounted volumes

    #create a network named elastic docker run -d -v /rex/ssd/zpehlivan/twitter_2018-03/trends/:/usr/src/temp/data -v /rex/ssd/zpehlivan/twitter_2018-03/twitter_indexer/:/usr/src/temp -ti --rm --name tindexer docker.rech.ina.fr/zpehlivan/temp-docker:latest

TODO:

  • Add index name, es etc. as params to script.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.