Comments (3)
🤖
Hello @Santhin! I'm Dosu, a friendly bot here to assist you in resolving issues, answering queries, and helping you become a contributor. While we wait for a human maintainer to assist further, I'm here to help you as best as I can!
Thank you for providing such a detailed description of the issue you're facing. It seems like the TestBehavior.AFTER_ALL
option is not behaving as expected due to the way the build_airflow_graph
function is structured in the Cosmos codebase.
In the current implementation, the TestBehavior.AFTER_ALL
option creates a single test task after all the "leaf" tasks in the DAG. A leaf task is one that does not have any downstream tasks. The test task is created and then added as a downstream task to all the leaf tasks. However, the seeds are also considered as leaf tasks, which means that the test task is being run concurrently with the seed tasks, leading to the errors you're experiencing.
Here is the relevant code snippet from the build_airflow_graph
function:
if test_behavior == TestBehavior.AFTER_ALL:
test_meta = create_test_task_metadata(
f"{dbt_project_name}_test",
execution_mode,
test_indirect_selection,
task_args=task_args,
on_warning_callback=on_warning_callback,
)
test_task = create_airflow_task(test_meta, dag, task_group=task_group)
leaves_ids = calculate_leaves(tasks_ids=list(tasks_map.keys()), nodes=nodes)
for leaf_node_id in leaves_ids:
tasks_map[leaf_node_id] >> test_task
(Source: cosmos/airflow/graph.py)
To resolve this issue, you would need to modify the logic in the build_airflow_graph
function to ensure that the test task is only run after all other tasks, including the seed tasks, have completed. This could involve creating a separate group for the seed tasks and ensuring that the test task is only run after this group has completed.
I also found a similar issue that was resolved in the Cosmos repository: Configure testing behavior. This issue was closed by a commit on January 10, 2023, which refactored the parsing and generation of tasks to support different testing types (after_each
, after_all
, none
). This might provide some insights into how to handle your issue.
I hope this helps! If you have any further questions or need more clarification, please let me know.
Sources
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.
from astronomer-cosmos.
Hi, @Santhin,
I'm helping the Cosmos team manage their backlog and am marking this issue as stale. From what I understand, you raised an issue regarding unexpected behavior of the TestBehavior.AFTER_ALL
option in the astronomer-cosmos
on K8s using kind and airflow helm chart. Dosubot provided a detailed analysis of the issue and suggested a modification to the logic in the build_airflow_graph
function to ensure the test task runs after all other tasks, including the seed tasks, have completed. You later inquired about any updates on the issue.
Could you please confirm if this issue is still relevant to the latest version of the Cosmos repository? If it is, please let the Cosmos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.
Thank you for your understanding and cooperation. If you have any further questions or need assistance, feel free to reach out.
from astronomer-cosmos.
Any updates on this?
from astronomer-cosmos.
Related Issues (20)
- Community Support HOT 1
- Release Cosmos 1.5.0
- Bugs HOT 1
- New Features
- Support docs generated with `dbt docs generate --static`
- Test task is generated though it is not defined HOT 2
- Reduce task queueing latency when using Cosmos HOT 3
- Emit Airflow metrics to support analysing Cosmos performance HOT 1
- `LoadMode.AUTOMATIC` does not seem to work as documented HOT 4
- Using EMR transient cluster with Cosmos HOT 1
- AthenaAccessKeyProfileMapping does not work as expected locally HOT 3
- [Bug] cosmos 1.4.1 does not create virtualenv when using `ExecutionMode.VIRTUALENV` HOT 1
- Decouple `LoadMode.AUTOMATIC` from `load()` method in `DbtGraph`
- Cosmos tasks randomly getting marked as zombie tasks HOT 2
- Fix hard to read AirflowException error logs from dbt command HOT 2
- bug please help me this is my project (ubuntu) HOT 4
- Permission issue with Cosmos cache in some restricted environments HOT 1
- [Bug]: Trial Issue using Bug report template HOT 1
- This a new feature request using Feature request Issue template HOT 1
- Link to dbt docs from task
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from astronomer-cosmos.