Empty transient tables about dbt_artifacts HOT 6 CLOSED

We are seeing the same issue and just arrived at this root cause today. We were thinking along the same lines of adjusting the incremental condition. Has that worked for y'all?

from dbt_artifacts.

@davidshapir I haven't had a chance to test this but am hoping to later this week - will let you know!

from dbt_artifacts.

@timothyjang123 - Sounds good! Can you share whatever macro or other operation you use in var_row_count if it ends up working?

from dbt_artifacts.

@timothyjang123 - I'm curious if making the template dynamic with var_row_count might suffer from the "parse time" vs "run time" issue explained in this dbt article. I was wondering if this could be done in SQL with something like:

where artifact_generated_at > (select max(artifact_generated_at, '1970-01-01 00:00:00 +0000') from {{ this }})

Note: this code is 100% untested, but I will try to look into this today. :)

context: I'm @davidshapir's coworker.

from dbt_artifacts.

Hmm I think I see what you're saying - I haven't seen that article before and I'd have to test out the differences between dbt compile/run to see how the issue would work with the change mentioned. From what I read, I think it might have unexpected behavior (strange logs) but during execution it could work as expected - partly because the dependent view is just a saved query so it wouldn't have that same ref() issue mentioned in the article.

where artifact_generated_at > (select ifnull(max(artifact_generated_at), '1970-01-01 00:00:00 +0000') from {{ this }})

Regardless, I think the above is more precise for what we're trying to do - thanks @tommyh! FYI, I modified the above a bit for my own use in Snowflake.

from dbt_artifacts.

@timothyjang123 - the edgecase to test with var_row_count approach would be: when the fct_dbt__test_executions does have rows in it, is the var_row_count variable greater than 0? Because, if I'm reading that blog article correctly which I totally might not be, then var_row_count will be 0 during "parse" time when the model gets compiled.

ifnull - yes, that looks great!

from dbt_artifacts.