Prlimit Unit Test fails on s390x about containerized-data-importer HOT 12 CLOSED

We discussed this in the sig-storage call, what we can do as a quick fix is to bump the memory limit to 2Gi
depending on the arch.

However, ideally, we would like to understand specifically why this fails on s390x;
We have some ideas, like, maybe the coverage tracking is really expensive and badly optimized for s390x,
or, maybe we were always close to exhausting the memory in the unit tests.

from containerized-data-importer.

• CDI version : v1.58.1

So this branch of CDI is using older golang, maybe that is the problem?

from containerized-data-importer.

This was tested with go 1.22.3 and 1.22.4.
The original Issue (#3226) also needed to omit the coverprofile flag as a workaround

from containerized-data-importer.

The unit tests run is containerized, this is why I am implying the branch matters

from containerized-data-importer.

yep, thanks for pointing that out
I noted the CDI version of the instance deployed on my cluster by mistake, but the branch I'm on is latest(1.59), when running the Unit tests here

from containerized-data-importer.

Go 1.22 only made it to main, so it's not in 1.59

from containerized-data-importer.

Oh meant main
What could be arguments against increasing that limit, at least architecture dependend to 2GB?
As long as I understand it, this 1GB stems from a recommandation made at OpenStack to prevent the allocation of an arbitrary amount of memory when using qemu-img

from containerized-data-importer.

Yeah that sounds about right, but, ideally I would like to understand what is it exactly that s390x doesn't like in this particular case? Is there a nonlegitimate request in that? seems completely fine to me

from containerized-data-importer.

I wasn't successful yet in troubleshooting the issue through memory profiling or valgrind, but I found a value, where the tests gets flaky.
At a limit of 1509000000 roughly 40-50% of the time the test succeeds

from containerized-data-importer.

Additionally the same Error can happen under amd64:
https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_containerized-data-importer/3349/pull-cdi-unit-test/1818300000342380544#1:build-log.txt%3A1206-1240

A further investigation is neccesary here

from containerized-data-importer.

Additionally the same Error can happen under amd64: https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_containerized-data-importer/3349/pull-cdi-unit-test/1818300000342380544#1:build-log.txt%3A1206-1240

A further investigation is neccesary here

Interesting.. I would run the unit tests locally and follow the memory consumption of the container,
but if I'm not mistaken this simply means that the 8Gi limit on the CI job for unit tests does not cut it,
Which is surprising to me even if this unit test alone is taking 1Gi.

from containerized-data-importer.

Additionally the same Error can happen under amd64: prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_containerized-data-importer/3349/pull-cdi-unit-test/1818300000342380544#1:build-log.txt:1206-1240

A further investigation is neccesary here

Small correction here to pull back a little:
The one coming up on that flaky amd64 run is not the same error as for the original s390x one.

<string>: fatal error: runtime: cannot allocate memory
vs
<string>: unexpected fault address 0x180000\nfatal error: fault\n[signal SIGSEGV: segmentation violation

The first one can easily be reproduced by lowering the limit to 1<<29 (~500MB) or below.

from containerized-data-importer.