Java app from Pluralsight course to get streaming tweets used with AWS Kinesis
- Go to Twitter Development Platform: https://developer.twitter.com/content/developer-twitter/en.html
- Log into the website
- Click on Apps > Create an App
- Complete the following THEN add the 4 keys to the app:
- app name
- website url: http://www.example.com
- enable sign-in with twitter
- callback url: http://www.example.com
- app usage: brief description
- go to tab - permissions: select: Read and Write
- go to tab - keys and tokens: generate 4 keys: Consumer API keys (2), Access token & access token secret (2)
*Currently, when this app is run some tweets stream but there are intermittent errors: JSONObject["text"] not found.
I. AWS Set-up - permissions
- AWS - create account
- IAM (identity and access management) > Users > User name = kinesis-user
- AWS Access type = Programmatic access [next permissions]
- Attach exisiting policies directly > search for Kinesis and add
- AmazonKinesisAnalyticsFullAccess
- AmazonKinesisFirehoseFullAccess
- AmazonKinesisFullAccess
- AWSLambbdaKinesisExecutionRole
- CloudWatchFullAccess
- AmazonDynamoDBFullAccess
II. Create Kinesis
- Kinesis > Create Kinesis stresm: tweet-stream
- Number of shards 1
- Run Java Kinesis-connect branch
- See Tweets consumed after adding below - run TwitterProducerMain then run TwitterConsumerMain
- TweetsProcessor
- TweetsProcessorFactory
- TwitterConsumerMain
- TwitterProducerMain
- Kinesis Connector Library
- Creating a Pipeline with Kinesis
*Create AWS S3 bucket with name "pluralsight-kinesis-course" *Run ConnectorsMain.java *In AWS - go into buckets and see a file - select checkbox and download - inspect file