Transformation: Pass data references about ods HOT 3 CLOSED

I would suggest the following implementation:
Scheduler triggers transformation (synchronous RPC request) -> transformation service executes transformation and triggers storage (blocking the scheduler) -> transformation notifies scheduler if transformation and storage was successful and transmits storage location (synchronous RPC response) -> scheduler triggers notification using the location

After refactoring the ods to an event-driven architecture, we can get rid of the blocking part and use events instead of RPC. But for now, this seems to be the easiest solution.
Are you okay with this approach @georg-schwarz?

from ods.

For now, I would rather let the storage (which is better named query in my opinion) stay unknown to the transformation.
Does this feature have any benefits for us right now? I agree, letting the transformation fetch the data via reference from adapter completes kind of one step we started implementing, so we should finish that. Also makes sense because we might build on that feature by enabling multiple pipelines per datasource.
However, I don't see the benefit in storing the data of the transformation result right now.. Or am I missing an important point here?

from ods.

Yes, I mixed something up here. What I described above is actually the next step of passing references instead of actual data. Thanks for pointing that out to me :D
You are right, the first step would be passing a reference to the data instead of the data itself from the scheduler to the transformation service.

After implementing that, the transformed data (itself, not a reference) would still be returned to the scheduler which passes it on to the scheduler.
In my opinion we should use references there, too. For example by implementing a similar approach by passing a reference to the storage service from the scheduler to the transformation service. Like this, the transformation service does not need to know the storage service, does not need an own database and we minimize unnecessary internal data transmission.
However, we can discuss this afterward and I start with the first step first.

from ods.