Connection trouble with 0.10.0 about memjs HOT 14 OPEN

OK, thanks for the report. 0.10 changed a specific way in which connections are handled so that's enough information to look.

I wish there were a way to mark versions as beta / not-yet-well-tested versions to try and catch these kinds of issues before they into production...

from memjs.

@miccolis Do you mind trying out 0.9.1 please and letting us know if everything works fine? We made some changes to timeout handling in both 0.9.1 and 0.10.0, so telling us if 0.9.1 works or fails will help narrow this down a lot for us.

from memjs.

I just tested 0.9.1 and hit a bunch of the socket timed out error message (haven't seen that in 0.9.0) plus a whole bunch of

(node) warning: possible EventEmitter memory leak detected. 11 timeout listeners added. Use emitter.setMaxListeners() to increase limit.

from memjs.

Can you share how you have memjs configured please? Ignore the warning messages, those are expected with 0.9.1.

from memjs.

@mikemorris Any update here? I can't replicate this on my end or see any problems from starring at the code so won't be able to make progress without help.

from memjs.

Hi @alevy , we are using memjs version 0.8.5 in our Heroku app and we randomly get a "MemJS: Server <..> failed after (2) retries with error - This socket has been ended by the other party ". This issue starts with a daily recycling of a server and it goes away when we restart the server manually.

As you mentioned, 0.10.0 handles the connection differently, should we update our app to use 0.10.0, will that help with this error ? given you pointed out 0.10.0 to be not well tested?

from memjs.

@azmsrwr Please try out 0.10.0 and let us know your results. We aren't able to replicate this issue right now so the more feedback we can get the better.

If you run into issues with 0.10.0, then downgrade to 0.9.0.

from memjs.

Another MemCachier customer reports:

We start getting ECONNRESET errors on all cache reads/writes.
Only way to resolve is to restart all dynos.

Error details:
/app/controllers/programs.coffee:101:24
at /app/node_modules/memjs/lib/memjs/memjs.js:149:23
at Object.errorHandler (/app/node_modules/memjs/lib/memjs/memjs.js:617:25)
at [object Object].Server.error (/app/node_modules/memjs/lib/memjs/server.js:71:18)
at Socket.\u003canonymous\u003e (/app/node_modules/memjs/lib/memjs/server.js:159:12)
at emitOne (events.js:77:13)
at Socket.emit (events.js:169:7)
at emitErrorNT (net.js:1269:8)
at nextTickCallbackWith2Args (node.js:442:9)
at process._tickCallback (node.js:356:17)

"msg": {
"msg": "Cache set error",
"err": {
"code": "ECONNRESET",
"errno": "ECONNRESET",
"syscall": "read"
}
}

Using memjs: 0.10.0 and node: v4.

from memjs.

Any update for this issue?
I also met this problem when the network was not that stable. Memjs unable to re-establish a connection to the memcached server.
Then I must restart the node process to establish a new connection, it would work again.

from memjs.

@leohihimax No update sorry, hopefully Amit or I will find some time soon to sit down with this one. The problem is we aren't able to reproduce this. Any extra information you can provide to help us there would be great.

from memjs.

We were randomly getting following issue with memjs version 0.8.5 in our Heroku app "MemJS: Server <..> failed after (2) retries with error - This socket has been ended by the other party ". We imagine this issue was around some sort of connection pooling where occasionally the server was restarted (daily planned restart) but connection was still not being renewed. This issue used to goes away when we restart the server manually.

After updating to memjs 0.10.0 we do not see this issue anymore.

from memjs.

@dterei
memjs version: 0.10.0
Actually, it is really hard to reproduce this in a good network condition or the memcached server is localhost.
Because my memcached server is running on a VPC, so the network is not that stable.

from memjs.

OK, I'll try to setup a fake network environment that simulates a poor network and see what happens.

from memjs.

I tried to reproduce this by connecting to a memcached server on the other side of the pond (I think it is called the Atlantic ocean) and setting the timeout to a value very close to the ping times I got from the server. I get occasional timeouts (which will return a timeout error for all outstanding requests) and sometimes even a ECONNRESET error (which will also return an error for all outstanding requests) but in all cases memjs reconnects fine for future requests.

from memjs.