Severe CPU degradation after vizier-pem update about pixie HOT 2 OPEN

Hey @jack-hernandez, I don't believe this is a known issue. Could you please provide information on the types of services you're running as well as PEM logs and any other guidance to reproduce this issue would really help.

from pixie.

Hi @kpattaswamy, most of our services running are PHP 7.4 using apache, supervisord or crontab. We install this on top of an Ubuntu 20.04 base image. We don't have Pixie running on production anymore but I captured some pem logs (attached) from our old cluster which does still have it running (though the cluster is not in use now, so not sure how helpful this will be).

Just to also provide additional context to the symptoms we saw, we noticed an increase in CPU on each individual process running within our application services, particularly when PHP code was being executed. It looks as though the CPU degradation happens very gradually over time and will continue to climb until everything comes to a halt. The same happened when we removed Pixie, and noticed it took a few hours for everything to eventually settle back down. It's also worth noting that even a rollout restart of pods with high cpu utilisation didn't alleviate the problem, they would come straight back up with high CPU rather than climbing from zero again (they were also not being throttled by any CPU limits, or processing anything more compute heavy than we'd expect).

We carried out other troubleshooting steps, working with AWS and other support partners to investigate our nodes, application changes and metrics, networking (specifically DNS), and our apache configuration, but the only thing that seems to have any real correlation with the behaviour we've seen is this change in the vizier-pem image version. We've also performed stress tests with and without Pixie and there is a clear impact on performance. I'm currently in the process of trying to replicate on a simplified cluster so I'll send any findings through once done.

nri-bundle__vizier-pem-dhc2z__pem.log
nri-bundle__vizier-pem-rtmn5__pem.log
nri-bundle__vizier-pem-v4bpm__pem.log

from pixie.