sw-jung / logstash-filter-memoize Goto Github PK

Do you know if an update is needed to get logstash-filter-memoize to work with Logstash 6? I am using the following code snippet and it continues to error:

memoize {
key => "%{highest_registered_domain}"
fields => [ "rank" ]
filter_name => "elasticsearch"
filter_options => {
elasticsearch {
query => "highest_registered_domain:%{highest_registered_domain}"
index => "alexa-top1m"
fields => {
"rank" => "rank"
}
}
}

This code snippet works on Logstash 5. WIthout memoize (using the below code) it works.

elasticsearch {
query => "highest_registered_domain:%{highest_registered_domain}"
index => "alexa-top1m"
fields => {
"rank" => "rank"
}

The error I receive is:

[ERROR] 2017-11-20 13:40:16.780 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"NoMethodError", :message=>"undefined method get' for {\"rank\"=>\"rank\"}:Hash\nDid you mean? gem", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:210:in expr'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:202:in expr'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:133:in expr'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:98:in expr_attributes'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:76:in expr'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:69:in expr'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:48:in block in compile'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:46:in compile'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:46:in compile_ast'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:54:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:107:in compile_lir'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:215:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:35:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:335:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:332:in block in converge_state'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:319:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:362:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in block in initialize'"]}

Any thoughts? This is a truly amazing plugin and I'm doing my best to share it with the community.

The add_tag configurations on other filters will add a tag to the filter depending on the success state of the filter.

It would be useful if add_tag_on_cache_hit and add_tag_on_cache_miss configurations could be added so that we could ascertain on a large scale what percentage of our lookups are cache hits versus misses and would likely assist in troubleshooting

Is there an exposed method to expire the cached result? The Logstash memcached filter is painfully slow, hoping to solve with this plugin.