Significant memory usage about sentry-java HOT 12 OPEN

Hey @adinauer , thank's for looking into that!

Do you do any manual capturing of errors, breadcrumbs etc.?

We don't create/capture/use breadcrumbs. We catch exceptions above the controller's layer in the @ControllerAdvice class -> @ExceptionHandler methods.

Other interaction with the Sentry API?

Nope.

Are any/many exceptions thrown and recorded by Sentry?

Not many, during a few days, we got <100 occurrences of 3 different Exceptions

Roughly how many requests is the server seeing?

Single pod/instance handles roughly up to 150 RPS

Do you use CRONS?

Nope, at least not directly in our code (not sure if it's used inside Spring itself)

I just set sentry.enable-aot-compatibility=true, will report back tomorrow with the result.

from sentry-java.

@RobicToNieMaKomu thanks for opening this issue. We'll take a look and update here.

from sentry-java.

@RobicToNieMaKomu I can confirm, we're using up more memory on startup but after manually triggering GC, it ends up roughly the same.

How I tested

• I fired up our Spring Boot 3 Webflux sample
• let it startup for a bit then manually triggered GC in visualvm a couple times
• sent some requests in using wrk -t2 -c4 -d30s -H "Authorization: Basic dXNlcjpwYXNzd29yZA==" http://127.0.0.1:8080/todo-webclient/1. This sends ~ 4k requests in 30s:

Running 30s test @ http://127.0.0.1:8080/todo-webclient/1
  2 threads and 4 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    28.21ms    9.48ms 174.51ms   93.47%
    Req/Sec    72.24     11.44   101.00     68.91%
  4325 requests in 30.03s, 532.18KB read
Requests/sec:    144.01
Transfer/sec:     17.72KB

• manually triggered GC a couple times again
• reran wrk
• manually triggered GC a couple times again

The endpoint does not record any errors but did create transactions and spans (when Sentry was enabled).

Results

Here's memory with Sentry:

And memory without Sentry:

So with Sentry enabled the server is consuming a lot more memory on startup but it is able to free that back up. Will keep investigating.

Questions

Can you please tell us a bit about your usage of Sentry on the screenshots you provided?

• Do you do any manual capturing of errors, breadcrumbs etc.?
• Other interaction with the Sentry API?
• Are any/many exceptions thrown and recorded by Sentry?
• Roughly how many requests is the server seeing?
• Do you use CRONS? There's another potential memory problem there, see #3165

from sentry-java.

@RobicToNieMaKomu if you're not using any of our AOP features (@SentryTransaction, @SentrySpan, @SentryCheckIn annotations) you could try to disable those using sentry.enable-aot-compatibility=true and see if that helps lower the initial memory consumption. From my testing it only used ~ 1/3 of memory on startup this way.

from sentry-java.

Here's another screenshot showing memory consumption with above mentioned test procedure and sentry.enable-aot-compatibility=true:

from sentry-java.

@RobicToNieMaKomu you can also try playing with these settings:

sentry.logging.minimum-event-level=info
sentry.logging.minimum-breadcrumb-level=debug

and you could drop all breadcrumbs from BeforeBreadcrumbCallback. Should be as simple as providing a spring bean that always returns null from Breadcrumb execute(@NotNull Breadcrumb breadcrumb, @NotNull Hint hint).

from sentry-java.

This is what it looks like after applying these properties:

sentry.enable-aot-compatibility=true
sentry.logging.minimum-event-level=info
sentry.logging.minimum-breadcrumb-level=debug

and BeforeBreadcrumbCallback -> null:

Maybe memory consumption became slightly smaller, but it still has a significant impact.

from sentry-java.

@RobicToNieMaKomu can you please provide a sample that allows us to reprodue the issue, maybe by forking the repo and adding to our Spring Boot 3 Webflux Sample. I'm assuming you're using additional Spring features, settings or other libraries that could cause this when combined with Sentry.

and BeforeBreadcrumbCallback -> null

I'm not sure we're on the same page here. Did you provide a callback that returns null (this is what is needed to drop all breadcrumbs) or did you set the callback to null (this is the default behaviour of not dropping breadcrumbs)?

sentry.logging.minimum-breadcrumb-level=debug

I meant to suggest you can increase the level to not have as many breadcrumbs but it's not that relevant if you're dropping all breadcrumbs anyways or limit breadcrumb count to 1.

from sentry-java.

Oh and are you using any special garbage collector settings?

from sentry-java.

Hey @RobicToNieMaKomu, did you had some time to look into the questions from above^, by any chance?

from sentry-java.

@adinauer I'm sorry, I can't fork the repo (NDA) and don't have enough time to come up with PoC now.
The simplified stack we're using in that service:

        <java.version>17</java.version>
        <lombok.version>1.18.30</lombok.version>
        <springboot-cloud.version>2022.0.5</springboot-cloud.version>
        <resilience4j.version>2.1.0</resilience4j.version>
        <protobuf-java.version>3.22.2</protobuf-java.version>
        <awaitility.version>4.2.0</awaitility.version>

Besides, it heavily uses Redis (spring-boot-starter-data-redis-reactive dependency of SB 3.1.8).

I'm not sure we're on the same page here. Did you provide a callback that returns null (this is what is needed to drop all breadcrumbs) or did you set the callback to null (this is the default behaviour of not dropping breadcrumbs)?

I provided a callback that returns null.

are you using any special garbage collector settings?

Java options: -Xss512k -XX:+UseZGC -XX:MaxDirectMemorySize=64m -XX:ReservedCodeCacheSize=120M

from sentry-java.

Thanks for the update.

from sentry-java.