Error: ConnectError: [unknown] network error on the package execution logs screen about kurtosis HOT 7 CLOSED

I just hit this today while trying to give a demo

from kurtosis.

we are discussing this issue here: https://kurtosistech.slack.com/archives/C05DZ4UABDZ/p1708114238397159

it could be related to this one: #1768

another possible relation: connectrpc/connect-es#907

from kurtosis.

It seems there is a timeout somewhere because it always fails when the call lasts more than 1 minute:

And it seems the timeout came from Connect http client when calling to the GRPC server, we can see the "Stream was closed by client" in the APIC's logs

from kurtosis.

I don't think that the problem is in ConnectGo or in the Kurtosis Engine because I've tested it locally with the UI and it runs the MERN package completely without any problem and even downloading the container images (I removed the images before the test).

If we see the following image the RunStarlarkPackage calls took more than 2 min and it was success while in the cloud it always is interrupted in 1min

I think that the problem, in cloud, could be in some layer above of this.

from kurtosis.

I see this error log in the cloud gateway when the call fails:

[error] 35#35: *16517 upstream timed out (110: Connection timed out) while reading upstream, client: 10.0.8.135, server: ~^(?<port_number>[0-9]{2,5})-(?<service_short_uuid>[0-9a-f]{12})-(?<enclave_short_uuid>[0-9a-f]{12})-(?<instance_short_uuid>[0-9a-f]{12}).*$, request: "POST /gateway/ips/54-88-173-250/ports/8081/kurtosis_enclave_manager.KurtosisEnclaveManagerServer/RunStarlarkPackage HTTP/1.1", upstream: "http://10.0.5.188:8081/kurtosis_enclave_manager.KurtosisEnclaveManagerServer/RunStarlarkPackage", host: "cloud.kurtosis.com", referrer: "https://cloud.kurtosis.com/gateway/ips/54-88-173-250/ports/9711/enclave/2bf955f3b89a/logs"

from kurtosis.

The error was fixed in the PR linked above

I was able to run the Jupyter package (including the images download part) without any issue, and the "RunStarlarkPackage" calls took ~2.5 minutes

Same result for the MERN package:

from kurtosis.

@barnabasbusa could you confirm if this fix also fix the use case that you reported on this ticket: #2231 ?

from kurtosis.