Use preinstalled deps from cicd instead of rerunning about astronomer-cosmos HOT 5 CLOSED

Also seems like dbt 1.7 behavoiour might of changed. When I have any form dependancy I get the below error:

from astronomer-cosmos.

Update think I found the issue the issue is caused when the create symlinks it deliberately ignores the dbt_packages folder as part of the project.py script. I think if we set dbt_deps to false then by default it should syslink the dbt_packages folder.

from astronomer-cosmos.

🤖

Hello @DanMawdsleyBA,

I'm Dosu, a friendly bot here to assist you with your issues, answer questions, and guide you in contributing to the Cosmos project. While we wait for a human maintainer, feel free to ask me anything related to the repository. Let's get your problem sorted!

Based on a similar issue I found in the Cosmos repository, you can configure Cosmos to recognize preinstalled packages from your CI/CD pipeline by setting the RenderConfig.dbt_deps to False when initializing the DbtGraph class. This will prevent Cosmos from rerunning the dbt deps command. Here's an example:

from cosmos.config import RenderConfig, ProjectConfig, ExecutionConfig, ProfileConfig
from cosmos.dbt.graph import DbtGraph

# Initialize the configs
render_config = RenderConfig(dbt_deps=False)  # Set dbt_deps to False
project_config = ProjectConfig(...)  # Fill in the necessary details
execution_config = ExecutionConfig(...)  # Fill in the necessary details
profile_config = ProfileConfig(...)  # Fill in the necessary details

# Initialize the DbtGraph
dbt_graph = DbtGraph(
    project=project_config,
    render_config=render_config,
    execution_config=execution_config,
    profile_config=profile_config
)

# Load the dbt project
dbt_graph.load(...)

Please replace the ... with the necessary details for your project. You can find more details about this in the DbtGraph class.

However, it seems like Cosmos might not be handling the copying of the dbt_packages folder. You might need to manually ensure that the dbt_packages folder is present in the correct location for Cosmos to recognize it. I wasn't able to find specific details on how Cosmos handles the dbt_packages folder during the execution of the DAGs and Task Groups within the repository.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

• Run dbt deps when using load_mode=LoadMode.DBT_LS
• cosmos/dbt/graph.py

from astronomer-cosmos.

I have already set the deps flag to False. I can see the cosmos recognizes the original folder and can see the dbt_packages. I think it might be something to do with the temp directory. Any ideas what might be the issue @tatiana happy to implement a fix for this

from astronomer-cosmos.

Just read this, @DanMawdsleyBA, yes - I think it is a good assumption. Thanks for the PR!

from astronomer-cosmos.