fugue-project / adagio Goto Github PK

Is your feature request related to a problem? Please describe.
I find task execution graphs really useful in communicating workflows to non-technical audiences, specifically they are interested in exactly what is being performed on the data to get it from A to B. However, this can get complex fast for distributed workflows!

dask:

prefect:

The dask graph is really hard to follow!

Describe the solution you'd like
The dask graph is for the nerds, I'd like a higher-level prefect style graph instead for non-tech audiences and documentation. prefect creates uses graphviz to create nodes for each task in its task graph and then plot them (I'm slowly learning @kvnkho :) - https://github.com/PrefectHQ/prefect/blob/5de58efaba956b431335d99acab07eaf6a362e1b/src/prefect/core/flow.py#L1290) which I suspect could be adapted here

Describe alternatives you've considered
It's already possible to leverage the dask graph by writing a dask function and using the dask execution engine, and so perhaps there is no need to do this natively!