High memory consumption from two modules about jpo-ode HOT 5 CLOSED

Were the logs of the ODE and the modules collected? Please share the logs. Thanks.

from jpo-ode.

We didn't collect the logs, but I have been noticing the PPM running high on memory before. I can collect logs in the future if you would like.

from jpo-ode.

Please do so we can focus on the right area for remediation. Since multiple modules are affected, chances are Kafka is running out of resources. The logs can potentially show us what type of resource (memory or disk) and where so the exception can be caught and handled properly.

Are huge data dumps going to be a regular occurrence?
What was the size of the data dump? How many files, how big each file? What type of data file?
Is it reproducible?

Thanks

from jpo-ode.

@kimPerry Please comment on the status of this issue. Has the new PPM module fix by ORNL team resolved the PPM issue. What about the CVPEP (SDC) BSM Depositor? Is it still exhibiting high memory consumption?

from jpo-ode.

The PPM module has had low memory consumption since the update, I think that one is fine now. I've only seen the CVPEP BSM module use a lot of memory during a major data dump, but it seems to be on par with the other modules during high load times. The PPM module was the main module that was concerning.

from jpo-ode.