Comments (1)
🤖
Hey @joppevos! Great to see you contributing to the Cosmos repository again. Your suggestions are always insightful.
Thank you for your detailed request and suggestions, joppevos. I agree that adding support for Airflow operator extra links in Cosmos could significantly improve task discoverability and debugging.
To implement this, we could extend the BaseOperator class and override the get_extra_links
method. This method should return a dictionary where the keys are the names of the links and the values are the URLs for those links.
Here's a basic example of how you could implement this:
from airflow.models import BaseOperator
class DbtOperatorWithExtraLinks(BaseOperator):
def get_extra_links(self, dttm, link_name):
if link_name == 'dbt_run_results':
# Here you would generate the URL for the dbt run_results.json file
# This could involve reading the invocation_id from the file and using it in the URL
return 'http://path/to/dbt/run_results.json'
elif link_name == 'dbt_python_api_results':
# Here you would generate the URL for the dbt Python API results
# This could involve calling the dbt Python API and getting the results
return 'http://path/to/dbt/python/api/results'
else:
return super().get_extra_links(dttm, link_name)
Then, in your calculate_operator_class
function, you would return the path to this new operator class instead of the existing one:
def calculate_operator_class(
execution_mode: ExecutionMode,
dbt_class: str,
) -> str:
"""
Given an execution mode and dbt class, return the operator class path to use.
:param execution_mode: Cosmos execution mode (e.g. local, virtualenv, docker, kubernetes)
:param dbt_class: The dbt command being used (e.g. DbtSnapshot, DbtRun, DbtSeed)
:returns: path string to the correspondent Cosmos Airflow operator
(e.g. cosmos.operators.localDbtSnapshotLocalOperator)
"""
return f"cosmos.operators.{execution_mode.value}.DbtOperatorWithExtraLinks"
Please note that this is a simplified example and you would need to adapt it to your specific use case, including handling the generation of the URLs for the extra links.
Regarding your suggestions about leveraging the invocation_id written by dbt in run_results.json or using dbt's Python API, both approaches seem feasible. The choice between them would depend on the specific requirements and constraints of your use case.
I hope this helps! If you have any further questions or need more clarification, please let me know.
Sources
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.
from astronomer-cosmos.
Related Issues (20)
- Decouple `LoadMode.AUTOMATIC` from `load()` method in `DbtGraph`
- Cosmos tasks randomly getting marked as zombie tasks HOT 3
- Fix hard to read AirflowException error logs from dbt command HOT 2
- bug please help me this is my project (ubuntu) HOT 6
- Permission issue with Cosmos cache in some restricted environments HOT 2
- [Bug]: Trial Issue using Bug report template HOT 1
- This a new feature request using Feature request Issue template HOT 1
- Link to dbt docs from task HOT 1
- [Cosmos CI] 2.7 unit tests taking a long time to run
- Remove dependency on Pydantic
- Improve how we authorise non-commiters to run Cosmos integration tests
- Review dbt in Airflow Survey results
- Support docs generated with `dbt docs generate --static`
- Test task is generated though it is not defined HOT 2
- Reduce task queueing latency when using Cosmos HOT 5
- Emit Airflow metrics to support analysing Cosmos performance HOT 2
- `LoadMode.AUTOMATIC` does not seem to work as documented HOT 4
- Using EMR transient cluster with Cosmos HOT 1
- AthenaAccessKeyProfileMapping does not work as expected locally HOT 3
- [Bug] cosmos 1.4.1 does not create virtualenv when using `ExecutionMode.VIRTUALENV` HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from astronomer-cosmos.