logstash-plugins / logstash-codec-plain Goto Github PK

https://travis-ci.org/logstash-plugins/logstash-codec-plain/jobs/236449417

An error occurred while loading ./spec/codecs/plain_spec.rb.
Failure/Error: # encoding: utf-8
NameError:
  missing class name (`org.apache.logging.log4j.Level')
# file:/home/travis/.rvm/rubies/jruby-1.7.25/lib/jruby.jar!/jruby/java/java_package_module_template.rb:14:in `method_missing'
# /home/travis/.rvm/gems/jruby-1.7.25/gems/logstash-core-5.4.0-java/lib/logstash/logging/logger.rb:6:in `Logging'
# /home/travis/.rvm/gems/jruby-1.7.25/gems/logstash-core-5.4.0-java/lib/logstash/logging/logger.rb:5:in `LogStash'
# /home/travis/.rvm/gems/jruby-1.7.25/gems/logstash-core-5.4.0-java/lib/logstash/logging/logger.rb:4:in `(root)'
# /home/travis/.rvm/gems/jruby-1.7.25/gems/logstash-core-5.4.0-java/lib/logstash/util/loggable.rb:1:in `(root)'
# /home/travis/.rvm/gems/jruby-1.7.25/gems/logstash-core-5.4.0-java/lib/logstash/util/loggable.rb:2:in `(root)'
# /home/travis/.rvm/gems/jruby-1.7.25/gems/logstash-core-5.4.0-java/lib/logstash/settings.rb:1:in `(root)'
# /home/travis/.rvm/gems/jruby-1.7.25/gems/logstash-core-5.4.0-java/lib/logstash/settings.rb:2:in `(root)'
# /home/travis/.rvm/gems/jruby-1.7.25/gems/logstash-core-5.4.0-java/lib/logstash/environment.rb:1:in `(root)'
# /home/travis/.rvm/gems/jruby-1.7.25/gems/logstash-core-5.4.0-java/lib/logstash/environment.rb:5:in `(root)'
# /home/travis/.rvm/gems/jruby-1.7.25/gems/logstash-core-5.4.0-java/lib/logstash/json.rb:1:in `(root)'
# /home/travis/.rvm/gems/jruby-1.7.25/gems/logstash-core-5.4.0-java/lib/logstash/json.rb:2:in `(root)'
# /home/travis/.rvm/gems/jruby-1.7.25/gems/logstash-core-5.4.0-java/lib/logstash/event.rb:1:in `(root)'
# /home/travis/.rvm/gems/jruby-1.7.25/gems/logstash-core-5.4.0-java/lib/logstash/event.rb:4:in `(root)'
# /home/travis/.rvm/gems/jruby-1.7.25/gems/logstash-core-5.4.0-java/lib/logstash/codecs/base.rb:1:in `(root)'
# /home/travis/.rvm/gems/jruby-1.7.25/gem

I'm using logstash reading data from mysql to kafka. for save the disk space, I write output config as this:

kafka {
    codec => plain {
        format => "%{key1},%{key2},%{key3}"
    }
}

as I expect, by giving such input data:

{
    key1: value1,
    key2: nil,
    key3: value3
}

the output should as this:

value1,,value3

but the acutal output is this:

value1,%{key2},value3

how to set output config to convert the nil with ""?

• Version: logstash 5.6.2
• Operating System: debian8

This is a stub issue, and needs to be fleshed out with details specific to
this plugin.

As a part of the effort to make plugins able to run in an ECS-Compatible manner
by default in an upcoming release of Logstash, this plugin needs to either
implement an ECS-Compatibility mode or certify that it does not implicitly use
fields that conflict with ECS.

Hi,

Im trying to read non UTF-8 file, it's IBM 500 file but i can't find it in the charset list provided with the plugin so i tried "US-ASCII".
Here is my config file:
input{
file {
path =>...
sincedb_path => "/dev/null"
codec => plain{charset =>"US-ASCII"}
start_position => "beginning"
}
}
output{stdout{}}

It doesn't work.
What i don't understand is that i don't have any error message, logstash starts and does nothing.
on my terminal i have only this message : "Logstash startup completed"

Any help please ?
Thanks

Setting any charset other than UTF-8 results in a call to data.encode(Encoding::UTF_8, :invalid => :replace, :undef => :replace), as seen here

This results in data loss, depending on the encoding. For example, setting a charset => Encoding::BINARY may very well contain bytes that do not map to a UTF-8 encoded character. These bytes will get replaced by the special unicode character used for invalid replacements, thus losing data.

What is the rationale behind this forced UTF-8 conversion? I think the codec should make no assumptions as to what the desired encoding is, it is up to the pipeline developer to process the data through filters if necessary.

Right now its impossible to get non-UTF-8 data into logstash, there's no codec that just gets the data through as-is and I expected the "plain" codec to be able to do just that.

here is my config:
input {
file {
path => ["/Users/luorenshu/software/es-x-pack/logstash/config/test.log"]
}
}

filter {
}

output {

#stdout {codec => rubydebug}
stdout {
    codec => plain {   
        format => "%{message}" ## format可不写，不写为空，写了就定义格式  
    }   
}   

}

when i append message "test" into test.log, the message printed is "2018-02-23T07:39:00.196Z robertdeMacBook-Pro.local test", but i just want it to be "test"