Add caching to improve runtime performance about astronomer-cosmos HOT 1 CLOSED

I'm not overly familiar with how the scheduler heartbeat works but I assume that it executes the DAG Python file and then collects the contents of globals to understand the shape of the DAG. After that I expect that the file closes.

If that is the case then I don't think we can use any internal memoization techniques like functools lru_cache, please correct me if I'm wrong as I don't have a deep understanding.
That leaves us with either implementing an external cache e.g. memcached or writing the contents to file.
Perhaps we could use PyYaml with a classmethod of to_yaml and from_yaml on the operators.
If we compute a single deterministic hash of the things that should trigger a change (such as the cosmos library version, dag files, relevant dbt directories) and use that to determine if we should re-process everything.
If we need to re-process then we use the standard constructor to initialize and then write out to file, otherwise we use the classmethod to read from file.

Either way we'd need to profile the current implementation and the cost/benefit of such an approach for a small, medium and large project. The expectation being that for smaller projects it's much simpler to use the current approach.

There is also always the option to look at threading to speed up the rendering of the project, but that's not really addressing the waste of processing, only trying to ensure that it can happen within the DAG heartbeat interval for larger projects.

from astronomer-cosmos.