Connecting non-beats source with appropriate Index about elk-docker HOT 5 CLOSED

Hi Brad,

First, a quick disclaimer: I'm not a Logstash guru by any means. 😕

I think there are two reasons why your config file isn't working:

• As far as I understand, the if ... else ... block can't go inside an elasticsearch section.
• I believe that mutate only works in a filter section, so you can't use it to dynamically alter index and document_type.

Again, not an expert so you may want to post your issue to the Logstash community to check what the "right" way to this would be, but the following config should work:

output {
  if [@metadata][beat] =~ "filebeat-" {
    elasticsearch {
      hosts => ["localhost"]
      sniffing => true
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" 
      document_type => "%{[@metadata][type]}" 
    } 
  } else {
    elasticsearch { 
      hosts => ["localhost"]
      sniffing => true
      manage_template => false
      index => "other" 
      document_type => "stuff" 
    }
  }
  stdout { 
    codec => rubydebug 
  }
}

(I see that in https://discuss.elastic.co/t/how-to-handle-multiple-inputs-with-logstash-to-different-indices/24541 the name of the index is assigned using grok at filter time – might be worth a shot.)

from elk-docker.

Looks like it's done the job thanks. I'll update if I come across any downsides. I'll tidy up our implementation down the track if we add many more input types.

Is it worth pulling something similar into your image? I guess the same issue will crop up for anything using the lumberjack input?

from elk-docker.

OK cool, thanks.

The image has indeed been strongly biased towards Beats since Lumberjack was deprecated in favour of Filebeat. To keep things clean I probably won't be updating the image (each user's situation will be different, so accommodating the most common use case here), but I'll certainly add some words in the documentation to guide users in the right direction should they need to add support for a non-Beats shipper alongside Beats.

from elk-docker.

Turns out the above if wasn't right. Here it is for completeness:

output {
  if [@metadata][beat] {
    elasticsearch {
      hosts => ["localhost"]
      sniffing => true
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" 
      document_type => "%{[@metadata][type]}" 
    } 
  } else {
    elasticsearch { 
      hosts => ["localhost"]
      sniffing => true
      manage_template => false
      index => "other" 
      document_type => "stuff" 
    }
  }
  stdout { 
    codec => rubydebug 
  }
}

from elk-docker.

Nice one. Thanks for the update.

from elk-docker.