Need a way to handle files with transient columns about tap-csv HOT 3 CLOSED

@seajhawk I dont think this is something that the tap would handle because it does already support a fluctuation of columns on each sync if you send a different catalog file each time, so I'm going to close the issue.

Maybe you could handle it with how you call Meltano but is there a way to not fluctuate columns? It seems like a data problem that would ideally be fixed in the source but I know its out of your control sometimes.

I'd double check this via the docs and slack community but I think youre catalog file is being re-used, if you did a full sync and used a new catalog each time you might be able to get away with it (depending on the target). I think if you run with the --full-refresh argument you'd get a new catalog file for each run. Hope that help!

from tap-csv.

@pnadolny13 - Thanks for reviewing this!

I agree that this doesn't seem like the job of the general use tap-csv plugin (unless lots of people need this capability), because it's moving beyond the primitive capability that most reasonable people would expect.

I'm not familiar with sending a catalog file, but if I do that and this tap is feeding to a target like target-postgres, will the target be able to handle the change in the catalog file? (am I thinking about this correctly?) I'm guessing it may not unless it has an additive mode where it can add new columns, but not remove columns that are not in the new catalog.

I was chatting with Douwe last week and he suggested there is a way to run some arbitrary code between the tap and target via a custom mapper. I need to look into this further as time permits.

from tap-csv.

I'm not familiar with sending a catalog file, but if I do that and this tap is feeding to a target like target-postgres, will the target be able to handle the change in the catalog file? (am I thinking about this correctly?) I'm guessing it may not unless it has an additive mode where it can add new columns, but not remove columns that are not in the new catalog.

I was chatting with Douwe last week and he suggested there is a way to run some arbitrary code between the tap and target via a custom mapper. I need to look into this further as time permits.

@seajhawk yeah those are good questions! I'm not sure the answers though 😄 , I'd follow up in slack and others can help. Sounds like mappers might be able to do the job though.

from tap-csv.