funcmike / logstash-output-clickhouse Goto Github PK

View Code? Open in Web Editor NEW

97.0 7.0 51.0 31 KB

Implementation of ClickHouse output for logstash

License: Other

Ruby 100.00%

logstash logstash-plugin json-batch clickhouse

logstash-output-clickhouse's Introduction

I switched to vector -> https://github.com/timberio/vector.

Logstash Plugin

This plugin is a modified version of the Lucidworks logstash json_batch. That plugin is available here.

It has been modified to support ClickHouse JSON Format, but also supports fault tolerance.

Usage

Please note that the name of the plugin when used is clickhouse, it only supports json in its current form. If further output formats are added in the future, this might change back to json_batch.

output {
  clickhouse {
    headers => ["Authorization", "Basic YWRtaW46cGFzc3dvcmQxMjM="]
    http_hosts => ["http://your.clickhouse1/", "http://your.clickhouse2/", "http://your.clickhouse3/"]
    table => "table_name"
    mutations => {
      "to1" => "from1"
      "to2" => [ "from2", "(.)(.)", '\1\2' ]
    }
  }
}

Other custom options

save_on_failure (default: true) - enable / disable request body save on failure
save_dir (default: /tmp) - directory where failed request body will be saved
automatic_retries (default: 1) - number of connect retry attempts to each host in http_hosts
request_tolerance (default: 5) - number of http request send retry attempts if response status code is not 200
backoff_time (default: 3) - time to wait in seconds for next retry attempt of connect or request

Default batch size is 50, with a wait of at most 5 seconds per send. These can be tweaked with the parameters flush_size and idle_flush_time respectively.

Installation

The easiest way to use this plugin is by installing it through rubygems like any other logstash plugin. To get the latest versio installed, you should run the following command: bin/logstash-plugin install logstash-output-clickhouse

Building the gem and installing a local version

To build the gem yourself, use gem build logstash-output-clickhouse.gemspec in the root of this repository. Alternatively, you can download a built version of the gem from the dist branch of this repository.

To install, run the following command, assuming the gem is in the local directory: $LOGSTASH_HOME/bin/plugin install logstash-output-clickhouse-X.Y.Z.gem

logstash-output-clickhouse's People

Contributors

Stargazers

Watchers

logstash-output-clickhouse's Issues

Concurrency error occurs

Hello,

Thanks for a really useful and so good-looking plugin. Unfortunately, I'm stably receiving a concurrency error when run it.

The error looks like this:
An unexpected error occurred! {:error=>#<ConcurrencyError: Detected invalid array contents due to unsynchronized modifications with concurrent users>, :backtrace=>["org/jruby/RubyArray.java:2486:in 'map'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/manticore/client.rb:336:in 'execute!'", "/usr/share/logstash/vendor/local_gems/0626d145/logstash-output-clickhouse-0.1.0/lib/logstash/outputs/clickhouse.rb:208:in 'block in make_request'"]}

I tried to change the code and make client.execute! on line 208 not in new thead but in the current thread and the error stopped to occur and everything started to work without problems.

So, I wonder if I'm only one who experiencing such a problem and why client.execute! is made in a separate thread whereas we specify :async = true flag in the client.send() call. To me it looks like logstash-mixin-http_client is not suited to work in multiple threads, but I may be wrong.

I run logstash with -w 1 flag to limit the number of workers to 1.

Would be grateful for any comments and ideas. Thanks.

Dynamic table name

Reference: https://www.altinity.com/blog/2017/12/18/logstash-with-clickhouse

Hello Andrey

I am trying to make the table name dynamic by passing a value which is combination of JSON fields from data being ingested. For instance:

clickhouse {
                        http_hosts => ["http://172.17.0.5:8123"]
                        user => "default"
                        password => "password"
                        # table => "w46a1d81.srstream" 
                        table => "%{stream}-%{profile_id}"
                        request_tolerance => 1
                        flush_size => 1000
                        pool_max => 1000
                        mutations => {
                                "stream" => "stream"
                                "profile_id" => "profile_id"
                                "user_id" => "user_id"
                               ....
                        }
                }

This set-up when executed throws error. Ingestion URLs are constructed like:

http://172.17.0.5:8123/?query=INSERT%20INTO%20%{stream}-%{profile_id}%20FORMAT%20JSONEachRow

Any ideas how this can be solved? Does this feature exists or we need to build it?

logstash 7.4.0 empty http body

Using logstash 7.4, output plugin sends an empty json doc only to clickhouse.

Install does not run on logstash 7.4.0

I am trying on a docker container:

`
Validating logstash-output-clickhouse
Installing logstash-output-clickhouse
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-mixin-http_client":
In snapshot (Gemfile.lock):
logstash-mixin-http_client (= 7.0.0)

In Gemfile:
logstash-input-http_poller java was resolved to 5.0.1, which depends on
logstash-mixin-http_client (~> 7) java

logstash-output-clickhouse java was resolved to 0.1.0, which depends on
  logstash-mixin-http_client (>= 6.0.0, ~> 6.0) java

Running bundle update will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Service 'logstash' failed to build: The command '/bin/sh -c bin/logstash-plugin install logstash-output-clickhouse' returned a non-zero code: 1
`

Please help with new option 'mutations' )

Hi!

We are trying to use our new option "mutations" but have some problems:

out config looks like:

...
output {
    if [fields][log_type] == 'nginx_access' {
        clickhouse {
            http_hosts => ["http://xxx:8123/"],
            table => "utr.nginx_log_buffer",
            automatic_retries => 3,
            mutations => {
                'uri' => 'uri',
                'http_host' => 'http_host',
                'http_referer' => 'http_referer',
                'http_user_agent' => 'http_user_agent',
                'method' => 'method',
                'remote_addr' => 'remote_addr',
                'request' => 'request',
                'upstream_status' => 'upstream_status',
                'upstream_response_time' => 'upstream_response_time',
                'cookie_regru_utr' => 'cookie_regru_utr',
                'request_date' => [ '@timestamp', /^(\d{4}-\d{2}-\d{2}).+/, '\1' ],
                'request_datetime' => [ '@timestamp', /^(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}:\d{2}).+/, '\1 \2' ],
            }
        }
    }
}
...

We always receive the following logstash errors:
:reason=>"Expected one of #, } at line 796, column 58 (byte 26337) after output {\n if [fields][log_type] == 'nginx_access' {\n clickhouse {\n http_hosts => ["http://xxx:8123/"]"}

We tried version with and without commas (between hash items), with regexps with or without quotes, still the same error.

May be you can help as more experienced logstash user...
Thank you! )

Using additional parameters

Using additional parameters for example input_format_skip_unknown_fields

install use logstash-plugin with error: Plugin logstash-output-clickhouse does not exist

# bin/logstash-plugin install logstash-output-clickhouse
Validating logstash-output-clickhouse
Plugin logstash-output-clickhouse does not exist
ERROR: Installation aborted, verification failed for logstash-output-clickhouse

Please update for Logstash 7.2

Installation of the plugin fails on Logstash 7.2 due to the gemspec.

$ logstash-plugin install logstash-output-clickhouse                                                            1 ↵  2.6.3
Validating logstash-output-clickhouse
Installing logstash-output-clickhouse
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-mixin-http_client":
  In snapshot (Gemfile.lock):
    logstash-mixin-http_client (= 7.0.0)

  In Gemfile:
    logstash-input-http_poller java was resolved to 5.0.1, which depends on
      logstash-mixin-http_client (~> 7) java

    logstash-output-clickhouse java was resolved to 0.1.0, which depends on
      logstash-mixin-http_client (>= 6.0.0, ~> 6.0) java

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

Installation failed with logstash 6.2+

Hi!
I can not install version 0.1.0 of logstash-output-clickhouse because of failed dependencies:
`` `
$ bin / logstash-plugin install --local --no-verify /tmp/logstash-output-clickhouse-0.1.0.gem
Installing logstash-output-clickhouse
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-mixin-http_client":
In snapshot (Gemfile.lock):
logstash-mixin-http_client (= 6.0.0)

  In Gemfile:
    logstash-input-http_poller (> = 0) java depends on
      logstash-mixin-http_client (<7.0.0,> = 6.0.0) java

logstash-input-http_poller (> = 0) java depends on
logstash-mixin-http_client (<7.0.0,> = 6.0.0) java

logstash-output-clickhouse (= 0.1.0) java depends on
logstash-mixin-http_client (<6.0.0,> = 2.2.1) java

logstash-mixin-http_client (= 6.0.0) java

Running bundle update will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
`` `

So I can not downgrade logstash-mixin-http_client because of it will brake many of default plugins.

Can you please bump the version of logstash-mixin-http_client to the> = 6.0.0?

I can PR this change in gemspecs, but I'm not a ruby developer and can not test this change.