Support tap-dbt-artifacts about dbt_artifacts HOT 4 CLOSED

Hey @andrewcstewart this is a great idea. I can see this package being configurable with a variable like 'artifact_source' which could take on values along the lines of:

• 'single_artifact_table' (We call this the V1 upload method)
• 'flattened_artifact_tables' (We call this the V2 upload method)
• 'tap-dbt-artifacts' (This new thing)

The staging models would then be compiled differently depending on this variable to match the source data. What do you think?

from dbt_artifacts.

Hi @NiallRees - Yeah that sounds like a good way to handle multiple sources.

I just noticed that this plugin is primarily angled towards Snowflake - I'm wondering (w/o having yet scoured the SQL dialect of the package models) if that is primarily due to the current upload method being Snowflake specific? For example, would tap-dbt-artifacts open up the possibility of using different adapters?

from dbt_artifacts.

The reason it's currently only Snowflake compatible is that the org it was originally built for were using Snowflake, the artifact upload method using PUT came later. Additional loading techniques absolutely opens up the possibility of adding compatibility with other warehouses though!

from dbt_artifacts.

We've been busy reimplementing the package, opening up this package to be compatible with more adapters (adding databricks compatibility out of the gate). We'd love any feedback you have: https://github.com/brooklyn-data/dbt_artifacts/releases/tag/1.0.0b1

from dbt_artifacts.