Comments (3)
@seajhawk I dont think this is something that the tap would handle because it does already support a fluctuation of columns on each sync if you send a different catalog file each time, so I'm going to close the issue.
Maybe you could handle it with how you call Meltano but is there a way to not fluctuate columns? It seems like a data problem that would ideally be fixed in the source but I know its out of your control sometimes.
I'd double check this via the docs and slack community but I think youre catalog file is being re-used, if you did a full sync and used a new catalog each time you might be able to get away with it (depending on the target). I think if you run with the --full-refresh
argument you'd get a new catalog file for each run. Hope that help!
from tap-csv.
@pnadolny13 - Thanks for reviewing this!
I agree that this doesn't seem like the job of the general use tap-csv plugin (unless lots of people need this capability), because it's moving beyond the primitive capability that most reasonable people would expect.
I'm not familiar with sending a catalog file, but if I do that and this tap is feeding to a target like target-postgres, will the target be able to handle the change in the catalog file? (am I thinking about this correctly?) I'm guessing it may not unless it has an additive mode where it can add new columns, but not remove columns that are not in the new catalog.
I was chatting with Douwe last week and he suggested there is a way to run some arbitrary code between the tap and target via a custom mapper. I need to look into this further as time permits.
from tap-csv.
I'm not familiar with sending a catalog file, but if I do that and this tap is feeding to a target like target-postgres, will the target be able to handle the change in the catalog file? (am I thinking about this correctly?) I'm guessing it may not unless it has an additive mode where it can add new columns, but not remove columns that are not in the new catalog.
I was chatting with Douwe last week and he suggested there is a way to run some arbitrary code between the tap and target via a custom mapper. I need to look into this further as time permits.
@seajhawk yeah those are good questions! I'm not sure the answers though 😄 , I'd follow up in slack and others can help. Sounds like mappers might be able to do the job though.
from tap-csv.
Related Issues (15)
- Remove State Capability HOT 4
- feat: add source file metadata columns
- Fix bug / add feature for recursive folder scan
- Implement State Capability HOT 3
- fix: set proper types for metadata columns
- Support batch messaging
- Allow using tap-csv with Python 3.10
- unable to get the version when invoking in Meltano HOT 1
- UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 HOT 5
- CSV seperators & dev activity HOT 5
- Hard to find errors HOT 2
- Default to `utf-8` encoding HOT 3
- Improve config definitions and defaults
- INFO:tap-csv:Skipping deselected stream 'payers'. HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tap-csv.