Comments (4)
Yes, if you reprocess the batch it will use the configuration entry that you have set when the reprocess occurs, which if unchanged would load both clusters. You could create a single cluster configuration that you link to the prefix and change the old one if you want to change the load target.
from aws-lambda-redshift-loader.
The s3Prefix is a primary key in config table, so it won't let me to create two separate load configurations, with different Redshift clusters, but for the same s3 prefix.
I had to trick the config a bit and created two separate entries with the single Redshift cluster config for each; one prod and one dr. I hope it is going to work:
1. bucket/source/schema/table_name/year=*/month=*/day=*/hour=*
2. bucket/source/schema/table_name/year=*/month=*/day=*/*
from aws-lambda-redshift-loader.
I think in that case it will potentially use both, or may just select the first one. I would instead change the prefix to a dummy value for the single cluster, and then flip it back to the dual cluster configuration once the reprocess is done.
from aws-lambda-redshift-loader.
I used your suggestion, Ian. The approach that I described above did not work, and as you said, it picked up the first matching prefix configuration.
from aws-lambda-redshift-loader.
Related Issues (20)
- Remaining duration of <negativenumber> insufficient to load cluster
- make moduleKeyName configurable in kmsCryto.js HOT 2
- Unable to validate the following destination configurations HOT 7
- node setup-file.js throw err;
- Unable to batch and COPY .parquet files via manifest file HOT 12
- batchOperations.js error HOT 7
- Add ttl to auto-expire entries in LambdaRedshiftBatches and LambdaRedshiftProcessedFiles
- What is the easiest way to delete historical LambdaRedshiftProcessedFiles HOT 1
- How do load the same file with a different timestamp? HOT 1
- Dataloss issue with this lambda redshift loader HOT 7
- Cluster Security VPC
- Option to disable hive partitioning wild cards HOT 8
- files not loaded occasionally due "another thread has locked it for loading" HOT 22
- files not loaded because they may be in 'locked' state (though they aren't) HOT 2
- ERR_TLS_CERT_ALTNAME_INVALID HOT 3
- Loader doesn't work on node 14 HOT 1
- LambdaRedshiftBatches in "locked" status - solved
- Manifest missing file returned in describeBatch.js HOT 4
- Shouldn't lambda-redshift-loader be released? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-lambda-redshift-loader.