Endurance Test! about data-prep-kit HOT 6 CLOSED

I tested with a set of 1483 files on (32GB, 4CPUs)and used traceback library to check for memory leak. I ran 10 iterations and observed the memory usage which was peaking around 4 GB. There were no obvious signs of a memory leak in this test.

from data-prep-kit.

I ran another test of 1483 files on podman VM with different memory configurations. The results are below.
It seems that it needed around 4GB of available memory to run successfully for these 1483 files.

from data-prep-kit.

@shahrokhDaijavad, @shivdeep-singh-ibm This is an important piece of info not only for us but also for potential users. Can we please:

1. Bring all of the results together in a separate document inside the project
2. Once 1 is complete, remove this issue

from data-prep-kit.

I agree, @blublinsky. I will create an md file called memorytest under doc with this information and link from the mac.md file to it.

from data-prep-kit.

@shahrokhDaijavad great. Should it be memory? or endurance?

from data-prep-kit.

@blublinsky It's a combination of testing for memory leak (which peaks and flattens around 4GB, i.e., no leak) and endurance that shows with smaller memory (4GB and 6GB total memory), it is still possible to process 500 or 900 files successfully before it crashes. I will explain in the readme.

from data-prep-kit.