Comments (2)
kevinmeredith
commented on June 30, 2024
So, I just ran the sample project successfully - I have a better idea.
It uploads the Users.txt contents to an S3 bucket, where they're separated by |.
$aws s3 cp s3://kinesis-bucket-kevin/49547565461333514253711404634954902460693951254611623938-49547565461333514253711404634993588086921619525641175042 ~
$head ~/49547565461333514253711404634954902460693951254611623938-49547565461333514253711404634993588086921619525641175042
338|BAM32FRT|Anastasia|Stephens|West Hollywood|PE|[email protected]|(623) 388-5271|false|false|false|false|false|false|false|false|true|false
I also could query Redshift via:
SELECT * FROM kinesisbasictable, seeing a bunch of results matching the users.txt file.
Currently, I have the following setup:
S3 bucket (each object is a single JSON) --> AWS Lambda (removes time-zone on a date's value since Redshift doesn't like time-zones when defining a TIMESTAMP) --> Kinesis --> ??? --> Redshift
Can I use this project to convert the JSON in Kinesis into the pipe-delimited format, and then put it into Redshift?
from amazon-kinesis-connectors.
buzzstop03
commented on June 30, 2024
At a high level, Redshift basic connector takes data from the users.txt file, puts into kinesis, reads data from kinesis and puts into s3 and finally reads data from s3 and puts it in redshift.
from amazon-kinesis-connectors.
Related Issues (20)
- AWS ES service version conflict HOT 1
- Add a fail callback when the message can not be transformed. HOT 1
- HTTP proxy support HOT 4
- Emitters should be more extensible
- Application logs
- Logic for auto scaling
- ElasticSearch connector does not work against AWS-ES service HOT 1
- Hibernate - Redshift.
- UnmodifiableBuffer equals method does not return false when buffers contain different records
- is Amazon KCL 1.7.5 compatible with elasticSearch 5.1.1? HOT 1
- Script is still using default values even after modifying the properties file HOT 2
- No suitable driver found for JDBC
- Understanding Number of records emitted to S3 HOT 3
- Not able to run the amazon-kinesis-connectors-samples-1.0.0-SNAPSHOT.jar
- finish Application after consuming n records from the stream with graceful shutdown HOT 2
- Setting Kinesis Client Library DynamoDB properties fails
- The input line is too long. The syntax of the command is incorrect.
- How can I migrate to use KCL 2.x and new AWS SDK, what's the future of kinesis-connectors?
- S3 Sample Error - Caught exception when uploading file s3://pfifer-connector-test HOT 3
- S3 and Kinesis access and secret keys can be different.Support multiple access and secret key.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazon-kinesis-connectors.