dbt invocation_id changes on upload_dbt_artifacts about dbt_artifacts HOT 8 CLOSED

Hey Alex, you're right that this is confusing. Currently, joins attempt to match on either invocation_id or dbt_cloud_run_id, see here. A couple of ideas to overcome this while still being able to use the upload_dbt_artifacts run-operation:

1. Use an environment variable which can be used to match the manifest.json to the run_results.json. You'd have to come up with a way to make sure the value was changed whenever you modified the project, and add the environment variable as an additional join key in the models. A PR would be very welcome for this, perhaps the env var could be called DBT_ENV_CUSTOM_ENV_DBT_ARTIFACTS_JOIN_KEY.
2. (Change to dbt-core required) A new metadata key in run_results.json and manifest.json is added as a hash of the entire project. This is definitely the best solution as it wouldn't require any manually configured environment variables.

from dbt_artifacts.

Another effect of this issue is that in dbt Cloud where one job has multiple dbt runs that need to have artifacts recorded, we can't reliably identify the pairs of eg. manifest + run_results.

from dbt_artifacts.

Another effect of this issue is that in dbt Cloud where one job has multiple dbt runs that need to have artifacts recorded, we can't reliably identify the pairs of eg. manifest + run_results.

In this instance, you could actually join any of the manifest.jsons which match on the dbt_cloud_run_id, as the manifest.json generated by each dbt run command within the same dbt Cloud run will be identical. It would require a couple modifications to the package to deduplicate manifests on the dbt_cloud_run_id.

from dbt_artifacts.

@NiallRees - I think that #81 solves this. If you agree can you close this issue?

from dbt_artifacts.

@NiallRees - I think that #81 solves this. If you agree can you close this issue?

I'm not sure that it does, but the specific issue of the invocation ID changing can't be resolved by dbt_artifacts. I've outlined a couple of ways forward in #53 (comment), would be happy to see a follow-up issue or PR for those.

from dbt_artifacts.

@DVAlexHiggs See the linked pull request above for a proper solve!

from dbt_artifacts.

Hi! Thanks so much. I'm gonna try this out. Any idea when it will hit release?

from dbt_artifacts.

Using --no-write-json should solve the problem you were encountering today. The work going on currently should be released in the next week.

from dbt_artifacts.