How different execution events are related between them about spline HOT 2 CLOSED

With the last question I mean that for example for a process that has different output files written and therefore different executions the target our output file is IL_CLE_2_1_2 with a URL of file:/fastdisk3/flight_searches/2_1_2/IL_CLE_2_1_2
This is the picture

Inside the processing node this is the node where it is written

Maybe the partitions are cuasing the change in data source name?

Then another execution that uses the same output data source as input source and has the same URL the data source name is different (it is called by the last partition).

Any reason for this? I want to have the complete view in the lineage diagram of both executions.

Thanks in advance

from spline.

how do different executions get related? By the data source name?

By the data source URI - this can be different for each data source type, but at the end it's a String in the database.

how does a data source node get named?

There are multiple plugins of difference data source types in Spark Spline Agent. Each Plugin is responsible for extracting the URIs from its data sources.

Maybe the partitions are cuasing the change in data source name?

Yes, this is a known issue.

Generally, this is an unsolvable problem. Consider OS path and one file having multiple aliases or server that is accessible from different IP addresses from different networks.

We want to solve this eventually by allowing Spline admin to define which URIs should be considered the same data source, but work on this haven't even began.

See the issue here: #689

from spline.