Client sending RST_STREAM led to exceeding maxEventsPerSecond(128) threshold about jetty.project HOT 4 OPEN

It is not normal.

The client by default does not have any rate control limit: it must be explicitly configured, and I assume it is not.

Could it be that the client application cancels every stream it creates, and the server is flooded with resets, and eventually exceeds the max events per second?

Is there a way we can reproduce the issue and take a closer look?

Can you enable client-side DEBUG logs to understand why the client resets the streams?

from jetty.project.

jetty.txt

Could it be that the client application cancels every stream it creates, and the server is flooded with resets, and eventually exceeds the max events per second?

Yes client is the one flooding server with RST_STREAM. Further investigation took me to the below code where the limit is checked.

from jetty.project.

Upon your suggestion, I enabled the logs at "org.eclipse.jetty" and got to know that there is a possibility, I am still debugging, that we may have been closing the InputStream bit early before we receive all the DataFrame and that is what, I guess, leading to send RST_FRAME. Below is the snippet of code where we are reading the InputStream.

https://github.com/apache/solr/blob/62cf3aaef34ba8f0437dbc355457e367488c8a99/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1831-L1850

from jetty.project.

@iamsanjay from the logs I came to the same conclusion, that you are not reading the whole content.

I can see from the logs that 250 bytes are received and consumed, but the next 4 bytes are not read, and are discarded because the input stream is being closed.

It is really important that you read until the InputStream reads -1.

This means that if you are parsing JSON, and the 250 bytes would represent a full JSON document (so from the point of view of the application you have all the relevant data), you still have 4 bytes to read and you still have to read the -1 (end-of-file).

Closing the InputStream earlier would lead to a reset being sent to the server.

from jetty.project.