[Bug]: Python expansion with multiple SqlTransforms is extremely slow about beam HOT 5 OPEN

The workaround looks good for the Python Direct Runner. @tvalentyn

from beam.

cc: @chamikaramj

from beam.

The ResolveArtifacts call will unconditionally download the 300MB beam-sdks-java-extensions-sql-expansion-service-2.55.0.jar to a temporary directory for each ste

The downloaded jars should be cached. Probably this caching doesn't work for your environment ?

You also have the option of manually specifying the jar [1] or manually starting up an expansion service [2].

[1] --beam_services="{\":sdks:java:extensions:sql:expansion-service:shadowJar\": \"$EXPANSION_SERVICE_JAR\"}"
[2] https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/#choose-an-expansion-service

from beam.

@chamikaramj The cache hit will never be detected for the downloaded JARs because of this line:

It always evaluates to False.

A worse problem though that, as mentioned above ArtifactRetrievalService.ResolveArtifacts() call takes 1.5s per SQL query even without the downloading of the actual files.

from beam.

@robertwb can you check this?

from beam.