Comments (8)
According to the code: https://github.com/http-kit/http-kit/blob/master/src/java/org/httpkit/server/HttpDecoder.java?source=c#L172
It seems that it fails to allocate a big enough buffer.
According to the code: https://github.com/http-kit/http-kit/blob/master/src/java/org/httpkit/server/HttpDecoder.java?source=c#L196
http-kit will try to reject too large body. the default value of max-body is 8M.
Does you server really busy and the memory JVM has is not big enough?
After the exception, http-kit will not process any http request. It's a bug, I will fix it.
from http-kit.
please try
[http-kit "2.1.8"]
It has this new commit: b54f860
When server is busy or something, the OOM happens, but surely, it's not the right time for the server to fail: there are so many customers.
from http-kit.
Once an OOM is thrown there's no much you can do beyond shutting down the VM.
Even if it was due to a wrong huge allocation that you would discard right away there could be a bunch of other concurrent operation failing at the same time (e.g. class loading). You cannot survive those in the general case.
There is probably 2 potential cause to this:
- some memory leak
- a misconfigured VM (like Xmx)
It would help to have a memory dump here. You could use Visual VM or the VM option -XX:+HeapDumpOnOutOfMemoryError
for this.
from http-kit.
Hi, @jeluard , I agree with you.
Since http-kit is just a very small part of the whole web application, it still a good idea to catch the exception and discard the wrong huge allocation, It's better just let that request fail, instead of the whole application fail.
Year, -XX:+HeapDumpOnOutOfMemoryError
will help a lot.
from http-kit.
Thanks very much for the quick responses! Here's some details on the server/vm setup:
It runs on a 1GB RAM/1 CPU virtual server with Ubuntu 13.04 x64 that I recently set up a few weeks ago. It doesn't have anything heavy running on it except for my http-kit app.
java version "1.7.0_21"
OpenJDK Runtime Environment (IcedTea 2.3.9) (7u21-2.3.9-1ubuntu1)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
I've been running it with the java command without any VM options: /usr/bin/java -jar my-app-standalone.jar
Here's a printout of the default settings: https://gist.github.com/cgmartin/6050892
Not experiencing any heavy traffic spikes, very low usage. The web pages it serves are cached behind a reverse proxy. It is primarily used as a WebSocket server and I don't see many concurrent connections. If curious, you can see it running here: http://cljwamp.us
I haven't been logging memory usage but whenever I spot check there's usually around 400-500 MB free. I unfortunately don't know what the memory situation was at the time of the stack traces. The app had been running without fail for 10 days straight with about 90 websocket clients that had connected over that time period. At the time of the stack traces only 1 client was connected. It logged 6 stack traces within a 10 minute timeframe.
I've restarted with the suggested HeapDump setting, and now using [http-kit "2.1.8"]
:
/usr/bin/java -XX:+HeapDumpOnOutOfMemoryError -jar my-app-standalone.jar
(I'm a newbie with VM tuning, any suggestions would be appreciated.)
I'll close this until I can capture a heap dump and provide some better details.
Much thanks for all the help!
from http-kit.
VisualVM would probably help you understand what happens to your memory usage, especially its monitor section. It's relatively easy to install/use and you can monitor remote VM with it (might require some conf).
One option you might want to set to the VM is Xmx
which defines the maximum usable memory: when this threshold is crossed you get OOM. Its default depends on OS/VM details.
from http-kit.
As far as I can think of, http-kit uses few memory ( 64K buffer, shared by all connection, other book keeping stuff). For you server, it has quite a lot of memory (1G).
I suspect there are memory leaks somewhere. Use tools or just write some code to stress test the code may help.
Stress test it for sometime, inspect the memory usage.
There are another tool, http://www.eclipse.org/mat/ , maybe helpful
from http-kit.
I've been meaning to get VisualVM set up for remote monitoring. I've used it locally to monitor one load scenario, which tests the messaging rate between many concurrent websocket connections (no memory leaks found in that test). I'll create another one that stresses the connection handling (random disconnects/reconnects) and let it run for a while. I haven't tried MAT before, I'll check that out. Thanks again for the tips!
from http-kit.
Related Issues (20)
- Race condition in TimerService.scheduleTask HOT 2
- Ring websocket API support HOT 4
- Problem in native-image and HttpUtils when using virtual threads HOT 11
- 2.8.0-beta2 has CIDER dependencies HOT 3
- CURL and finagle failing to parse :set-cookies with '\n' HOT 13
- Unix socket benchmark HOT 1
- It will turn headers into camel format HOT 2
- logger-warn gets rebound HOT 1
- logger-warn and error-warn are passed in wrong order to HttpServer constructor HOT 1
- Requests which throw java.net.ConnectException may actually succeed HOT 3
- Request Map doesn't contain information about authority HOT 8
- http-kit v2.8.0-RC1 HOT 1
- http-kit v2.8.0 final HOT 1
- Consider adding `Content-Type: text/plain` to HTTP 500 response in `org.httpkit.server.HttpHandler#run` HOT 6
- v2.7.0 SNI change broke connections to plain IP addresses with SSLHandshakeException "Hostname or IP address is undefined." HOT 8
- Add options for encoding nested form and query params a la clj-http HOT 7
- WebTransport support HOT 2
- NoSuchMethodError on projects AOT'd on java 21 but run on earlier java versions HOT 13
- [Proposal][Client] Consider more idiomatic bridges with JVM async paradigms HOT 5
- "Unmasked client to server frame" causes 502s HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from http-kit.