Giter VIP home page Giter VIP logo

Comments (8)

repeatedly avatar repeatedly commented on August 15, 2024

Use types parameter in in_http: http://docs.fluentd.org/articles/parser-plugin-overview

from fluent-plugin-influxdb.

o1da avatar o1da commented on August 15, 2024

Is there any other way to insert integer or float to InfluxDb? I use in_forward input plugin, because I need to read data from docker. And it seems that you have to use docker with in_forward only or am I wrong? This option doesn't work too:

<filter **>
  type record_transformer
  enable_ruby
  <record>
    value ${record["value"].to_f}
  </record>
</filter>

from fluent-plugin-influxdb.

repeatedly avatar repeatedly commented on August 15, 2024

@o1da Use auto_typecase : http://docs.fluentd.org/articles/filter_record_transformer#autotypecast-optional

from fluent-plugin-influxdb.

o1da avatar o1da commented on August 15, 2024

@repeatedly Thank you for tip, but I still have same problem with it. My filter looks like this:

<filter **>
  type record_transformer
  renew_record 
  enable_ruby 
  auto_typecast 
  <record>
    key ${/\?key=([^ ]*)/.match(record["log"]){|m| m[1]}}
    value ${/\[0m([^ ]*) ms/.match(record["log"]){|m| m[1]}}
  </record>
</filter>

Messages which I'm parsing looks like #033[0mGET /retro/7/61/[email protected]?key=D82edpaWqKQaiQSNtXrr #033[32m200 #033[0m0.708 ms - 18660#033[0m
and I get key "D82edpaWqKQaiQSNtXrr" which is OK, but value is "0.708" which is not ok, it should be 0.708 and it is the same if I add another filter:

<filter **>
  type record_transformer
  auto_typecast
  value ${value}
</filter>

What am I doing wrong?

from fluent-plugin-influxdb.

repeatedly avatar repeatedly commented on August 15, 2024

value ${/[0m([^ ]*) ms/.match(record["log"]){|m| m[1]}}

Why don't you call to_f like value ${record["value"].to_f}?

from fluent-plugin-influxdb.

o1da avatar o1da commented on August 15, 2024

I solved it with second filter.

<filter **>
  type record_transformer
  auto_typecast
  enable_ruby
  <record>
    value ${record["value"].to_f}
  </record>
</filter>

I overlooked info about limitation for single placeholder in auto_typecast before.
@repeatedly Thank you very much :-)

from fluent-plugin-influxdb.

repeatedly avatar repeatedly commented on August 15, 2024

@o1da

I solved it with second filter.

So you setup record_transformer filter chain?
On my environment, ${/\[0m([^ ]*) ms/.match(record["log"]){|m| m[1].to_f}} works.

% fluentd -c o1da_case.conf
2016-05-03 09:07:35 +0900 [info]: reading config file path="o1da_case.conf"
2016-05-03 09:07:35 +0900 [info]: starting fluentd-0.12.22
2016-05-03 09:07:35 +0900 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.0.4'
2016-05-03 09:07:35 +0900 [info]: gem 'fluentd' version '0.12.22'
2016-05-03 09:07:35 +0900 [info]: adding filter pattern="**" type="record_transformer"
2016-05-03 09:07:35 +0900 [info]: adding match pattern="**" type="stdout"
2016-05-03 09:07:35 +0900 [info]: adding source type="forward"
2016-05-03 09:07:35 +0900 [info]: using configuration file: <ROOT>
  <source>
    @type forward
  </source>
  <filter **>
    type record_transformer
    renew_record 
    enable_ruby 
    auto_typecast 
    <record>
      key ${/\?key=([^ ]*)/.match(record["log"]){|m| m[1]}}
      value ${/\[0m([^ ]*) ms/.match(record["log"]){|m| m[1].to_f}}
    </record>
  </filter>
  <match **>
    @type stdout
  </match>
</ROOT>
2016-05-03 09:07:35 +0900 [info]: listening fluent socket on 0.0.0.0:24224
2016-05-03 09:07:38 +0900 test.foo: {"key":"D82edpaWqKQaiQSNtXrr","value":0.708}

from fluent-plugin-influxdb.

o1da avatar o1da commented on August 15, 2024

This configuration doesn't work, but I have different version and another plugins and installed td-agent package from debian jessie repo - maybe it makes the difference.

2016-05-20 12:42:39 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2016-05-20 12:42:39 +0000 [info]: starting fluentd-0.12.20
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.1'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-influxdb' version '0.2.7'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-mongo' version '0.7.12'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.4'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-s3' version '0.6.5'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-td' version '0.10.28'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-05-20 12:42:39 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2016-05-20 12:42:39 +0000 [info]: gem 'fluentd' version '0.12.20'
2016-05-20 12:42:39 +0000 [info]: adding filter pattern="**" type="record_transformer"
2016-05-20 12:42:39 +0000 [info]: adding match pattern="**" type="stdout"
2016-05-20 12:42:39 +0000 [info]: adding source type="forward"
2016-05-20 12:42:39 +0000 [info]: using configuration file: <ROOT>
  <source>
    type forward
    heartbeat_type tcp
    port 24224
    bind 127.0.0.1
  </source>
  <filter **>
    type record_transformer
    renew_record 
    enable_ruby 
    auto_typecast 
    <record>
      key ${/\?key=([^ ]*)/.match(record["log"]){|m| m[1]}}
      value ${/\[0m([^ ]*) ms/.match(record["log"]){|m| m[1].to_f}}
    </record>
  </filter>
  <match **>
    @type stdout
  </match>
</ROOT>
2016-05-20 12:42:39 +0000 [warn]: parameter 'heartbeat_type' in <source>
  type forward
  heartbeat_type tcp
  port 24224
  bind 127.0.0.1
</source> is not used.
2016-05-20 12:42:39 +0000 [info]: listening fluent socket on 127.0.0.1:24224
2016-05-20 14:23:18 +0000 test: {"key":"D82edpaWqKQaiQSNtXrr","value":"70.422"}

Only configuration with second filter works and it is OK for me I don't need to tweak it any more.

2016-05-20 14:28:49 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2016-05-20 14:28:49 +0000 [info]: starting fluentd-0.12.20
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.1'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-influxdb' version '0.2.7'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-mongo' version '0.7.12'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.4'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-s3' version '0.6.5'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-td' version '0.10.28'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-05-20 14:28:49 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2016-05-20 14:28:49 +0000 [info]: gem 'fluentd' version '0.12.20'
2016-05-20 14:28:49 +0000 [info]: adding filter pattern="**" type="record_transformer"
2016-05-20 14:28:49 +0000 [info]: adding filter pattern="**" type="record_transformer"
2016-05-20 14:28:49 +0000 [info]: adding match pattern="**" type="stdout"
2016-05-20 14:28:49 +0000 [info]: adding source type="forward"
2016-05-20 14:28:49 +0000 [info]: using configuration file: <ROOT>
  <source>
    type forward
    heartbeat_type tcp
    port 24224
    bind 127.0.0.1
  </source>
  <filter **>
    type record_transformer
    renew_record 
    enable_ruby 
    <record>
      key ${/\?key=([^ ]*)/.match(record["log"]){|m| m[1]}}
      value ${/\[0m([^ ]*) ms/.match(record["log"]){|m| m[1]}}
    </record>
  </filter>
  <filter **>
    type record_transformer
    auto_typecast 
    enable_ruby 
    <record>
      value ${record["value"].to_f}
    </record>
  </filter>
  <match **>
    @type stdout
  </match>
</ROOT>
2016-05-20 14:28:49 +0000 [warn]: parameter 'heartbeat_type' in <source>
  type forward
  heartbeat_type tcp
  port 24224
  bind 127.0.0.1
</source> is not used.
2016-05-20 14:28:49 +0000 [info]: listening fluent socket on 127.0.0.1:24224
2016-05-20 14:29:11 +0000 test: {"key":"D82edpaWqKQaiQSNtXrr","value":91.37}

from fluent-plugin-influxdb.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.