output.heka_tcp is stuck retrying a message about hindsight HOT 7 CLOSED

Note that when I simply stop and restart hindsight (while keeping both the hindsight.cp checkpoint file and the input log files), the issue was fixed, which is surprising because I thought it would read the same buggy heka message again and reproduce the issue.

from hindsight.

It sounds like a client connection couldn't be re-established and it was just thrashing here:

if not client then return -3, err end -- retry indefinitely

But the sequence identifier indicates memory corruption. I will do a quick code review. On a related note the current lua_sandbox (actually lua_sandbox_extensions now) pulls in a much newer version of the socket and ssl library with many bug fixes, so that is worth trying. FYI: All the extensions (modules and sandboxes) are now created as separate packages so they can be maintained/upgraded/deployed independently. For convenience I have grouped the extensions Mozilla uses here: https://github.com/mozilla-services/lua_sandbox_extensions/ but they can now live anywhere and work consistently (no more special built-in modules or updating the core library for business logic changes)

from hindsight.

Thanks this sounds good!
I'd may try to create a LuaFileSystem (LFS) lua_sandbox extension which I often need in my sandboxes: should I host it myself of create a pull request to the https://github.com/mozilla-services/lua_sandbox_extensions/ repository?

from hindsight.

Either way is fine but I won't be a bottleneck if you use your own repo (I will be happy to review it in either case). In the end I will just hopefully yum install it when I need it and not be concerned with the repo :)

from hindsight.

@bbinet have you been able to reproduce this with the latest packages?

from hindsight.

I'm currently in the process of upgrading to the latest packages, but this is not functional yet.

Note that I've still not reproduced this issue with the old Hindsight version I'm using.

from hindsight.

Reopen if you can reproduce it as the heka_tcp output was reworked since this time
mozilla-services/lua_sandbox_extensions@90144b5

from hindsight.