winston-logstash will not reconnect with logstash when using --config.reload.automatic about winston-logstash HOT 8 CLOSED

Hi!

I don't have any direct advice, but I'll try to reproduce the issue and figure out the problem. I'll get back when I have some thing to share. 👍

from winston-logstash.

I started to write test cases for the config reload when I noticed an issue with the reload when there was output with TCP configuration. It doesn't reload the configuration because it can't start the TCP server because the port is still in use. Have you noticed a similar case? my draft pr #76

logstash-logstash-1 | [2023-03-18T08:20:23,286][ERROR][logstash.outputs.tcp ][main] Could not start tcp server: Address in use {:host=>"0.0.0.0", :port=>9999}

from winston-logstash.

No
I can not see your error log in my logstash console
I am using logstash 8.6.2 and elasticsearch 8.6.2 on Windows
and this is my .conf file

    input {
        tcp { port => 28777 }
    }
    output {
      elasticsearch {
        hosts => ["http://localhost:9200/"]
        user => "elastic"
        password => "PASSWORD"
        index => "dummy_index"
      }
      stdout { }
    }

    filter {
        json {
          source => "message"
        }
        json {
          source => "message"
          remove_field => [ "message","event","timestamp" ]
        }
    }

from winston-logstash.

My current test setup uses TCP output, which leads the reload case to this known bug https://discuss.elastic.co/t/tcp-output-plugin-blocking-port-across-configuration-reload/163732 . So I need to go around it to write the test case for config reloading.

from winston-logstash.

While I failed to write a failing test due to the bug, I created a sandbox setup where I reproduced the scenario. On the reload, the Logstash closes the connection without any error, which leads to a case where the code is not retrying. Next, I need to figure out if adding the retry logic might break the backward compatibility; if not, it's a straightforward change.

from winston-logstash.

A working solution is now for reconnecting to logstash after it closes the connection in draft #76. I still need to figure out the best way to write some tests and check if the change is backwards compatible.

from winston-logstash.

I just released a new version which addressed this issue. If you still have issues with the reconnect don't hesitate to re-open the issue. 😄

from winston-logstash.

@jaakkos I think this issue really describes two issues:

1. reconnect logstash after one sided disconnection <-- this one you already handled
2. all other transports are getting paralyzed, when using max_connect_retries: -1

the second issue is still relevant, easily reproducible, and very dangerous in production.
while winston-logstash tries to reconnect, all other transports do not work at all.
imagine logstash going down for 5 minutes - NOTHING is getting logged anywhere (in the first post there are TWO more transports - console and file), until it either returns, or the retry limit reached (in which case nothing further will log, unless you manually remove and add transports, in an error handler).
why is there such a strong coupling between the transports?

from winston-logstash.