Decide how Cosmos should treat dbt views about astronomer-cosmos HOT 5 CLOSED

🤖

Hey @jlaneve! Good to see you here again, your insights are always thought-provoking. Thanks for this detailed issue! I'm diving into your proposal now and will get back to you with a comprehensive response shortly.

from astronomer-cosmos.

from @tatiana: another workaround in the meantime could be creating manual dependencies between table A and C, using something like #577

from astronomer-cosmos.

I really like your proposal, @jlaneve ! Once we work on this, follow up in #airflow-astronomer:
https://apache-airflow.slack.com/archives/C03T0AVNA6A/p1702309820139329

from astronomer-cosmos.

Love it!

from astronomer-cosmos.

Nothing wrong with this as a feature that users can enable, and as long as I'm aware of this feature I'll just do whatever I need in my own setup, so default behavior discussions decisions won't impact me much. That said, if I may, my $0.02 would be that it is probably better to not have Cosmos skip views as a default behavior.

Main reasons:

• Some amount of astonishment as it is non-standard dbt behavior: The default behavior for dbt is that it will attempt to recreate any view by default. I think by default Cosmos should attempt to more or less emulate would happen if you just ran dbt build, and this would include recreating any views.
• Unclear what problem this is solving: If the issue is compile time, I think there are much better ways to solve this. For example, #785. (<- Although this issue doesn't address performance issues for small, "just getting started" style projects, it does help for larger more complex projects that have dbt compile as part of their CICD, which are the ones more likely to hit performance issues anyway.)
• Not skipping is likely better behavior for new users: It seems like the trade-off being made here between setting this as a default behavior vs not is potentially reducing complaints about performance / DagRun landing times, at the risk of potentially increasing complaints that Cosmos is unexpectedly skipping things. For users just getting started with Cosmos, performance shouldn't be an issue as their projects are likely to be small. So not skipping should optimize the new user experience.
• Backwards compatibility: This will be a huge gotcha for a lot of people who are upgrading Cosmos without looking at the release notes.

A few other reasons:

• Some users update their views a lot: I think the assumption is that views don't change, but a lot of workflows may involve making many changes to views. Obviously not on each run, but many users may be constantly making code changes 5 days a week and those code changes may be woven through the DAG.
• Lack of Params in existing DbtDag, combined with being a non-standard dbt feature: The proposal here suggests adding a Param, presumably to the DbtDag. I think adding Params to the DbtDag would be fine, but it feels like there should be more of an all-or-nothing; most notably, --full-refresh is a Param that makes a lot of sense to include as well.

Again, this doesn't really impact me, I'm just chiming in on what I think would be the better way to proceed for Cosmos's sake.

from astronomer-cosmos.