funcmike / logstash-output-clickhouse Goto Github PK
View Code? Open in Web Editor NEWImplementation of ClickHouse output for logstash
License: Other
Implementation of ClickHouse output for logstash
License: Other
I am trying on a docker container:
`
Validating logstash-output-clickhouse
Installing logstash-output-clickhouse
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-mixin-http_client":
In snapshot (Gemfile.lock):
logstash-mixin-http_client (= 7.0.0)
In Gemfile:
logstash-input-http_poller java was resolved to 5.0.1, which depends on
logstash-mixin-http_client (~> 7) java
logstash-output-clickhouse java was resolved to 0.1.0, which depends on
logstash-mixin-http_client (>= 6.0.0, ~> 6.0) java
Running bundle update
will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Service 'logstash' failed to build: The command '/bin/sh -c bin/logstash-plugin install logstash-output-clickhouse' returned a non-zero code: 1
`
Hi!
We are trying to use our new option "mutations" but have some problems:
out config looks like:
...
output {
if [fields][log_type] == 'nginx_access' {
clickhouse {
http_hosts => ["http://xxx:8123/"],
table => "utr.nginx_log_buffer",
automatic_retries => 3,
mutations => {
'uri' => 'uri',
'http_host' => 'http_host',
'http_referer' => 'http_referer',
'http_user_agent' => 'http_user_agent',
'method' => 'method',
'remote_addr' => 'remote_addr',
'request' => 'request',
'upstream_status' => 'upstream_status',
'upstream_response_time' => 'upstream_response_time',
'cookie_regru_utr' => 'cookie_regru_utr',
'request_date' => [ '@timestamp', /^(\d{4}-\d{2}-\d{2}).+/, '\1' ],
'request_datetime' => [ '@timestamp', /^(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}:\d{2}).+/, '\1 \2' ],
}
}
}
}
...
We always receive the following logstash errors:
:reason=>"Expected one of #, } at line 796, column 58 (byte 26337) after output {\n if [fields][log_type] == 'nginx_access' {\n clickhouse {\n http_hosts => ["http://xxx:8123/"]"}
We tried version with and without commas (between hash items), with regexps with or without quotes, still the same error.
May be you can help as more experienced logstash user...
Thank you! )
# bin/logstash-plugin install logstash-output-clickhouse
Validating logstash-output-clickhouse
Plugin logstash-output-clickhouse does not exist
ERROR: Installation aborted, verification failed for logstash-output-clickhouse
Reference: https://www.altinity.com/blog/2017/12/18/logstash-with-clickhouse
Hello Andrey
I am trying to make the table name dynamic by passing a value which is combination of JSON fields from data being ingested. For instance:
clickhouse {
http_hosts => ["http://172.17.0.5:8123"]
user => "default"
password => "password"
# table => "w46a1d81.srstream"
table => "%{stream}-%{profile_id}"
request_tolerance => 1
flush_size => 1000
pool_max => 1000
mutations => {
"stream" => "stream"
"profile_id" => "profile_id"
"user_id" => "user_id"
....
}
}
This set-up when executed throws error. Ingestion URLs are constructed like:
http://172.17.0.5:8123/?query=INSERT%20INTO%20%{stream}-%{profile_id}%20FORMAT%20JSONEachRow
Any ideas how this can be solved? Does this feature exists or we need to build it?
Using additional parameters for example input_format_skip_unknown_fields
Hi!
I can not install version 0.1.0 of logstash-output-clickhouse because of failed dependencies:
`` `
$ bin / logstash-plugin install --local --no-verify /tmp/logstash-output-clickhouse-0.1.0.gem
Installing logstash-output-clickhouse
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-mixin-http_client":
In snapshot (Gemfile.lock):
logstash-mixin-http_client (= 6.0.0)
In Gemfile:
logstash-input-http_poller (> = 0) java depends on
logstash-mixin-http_client (<7.0.0,> = 6.0.0) java
logstash-input-http_poller (> = 0) java depends on
logstash-mixin-http_client (<7.0.0,> = 6.0.0) java
logstash-output-clickhouse (= 0.1.0) java depends on
logstash-mixin-http_client (<6.0.0,> = 2.2.1) java
logstash-mixin-http_client (= 6.0.0) java
Running bundle update
will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
`` `
So I can not downgrade logstash-mixin-http_client
because of it will brake many of default plugins.
Can you please bump the version of logstash-mixin-http_client
to the> = 6.0.0
?
I can PR this change in gemspecs, but I'm not a ruby developer and can not test this change.
Installation of the plugin fails on Logstash 7.2 due to the gemspec.
$ logstash-plugin install logstash-output-clickhouse 1 ↵ 2.6.3
Validating logstash-output-clickhouse
Installing logstash-output-clickhouse
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-mixin-http_client":
In snapshot (Gemfile.lock):
logstash-mixin-http_client (= 7.0.0)
In Gemfile:
logstash-input-http_poller java was resolved to 5.0.1, which depends on
logstash-mixin-http_client (~> 7) java
logstash-output-clickhouse java was resolved to 0.1.0, which depends on
logstash-mixin-http_client (>= 6.0.0, ~> 6.0) java
Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Using logstash 7.4, output plugin sends an empty json doc only to clickhouse.
Hello,
Thanks for a really useful and so good-looking plugin. Unfortunately, I'm stably receiving a concurrency error when run it.
The error looks like this:
An unexpected error occurred! {:error=>#<ConcurrencyError: Detected invalid array contents due to unsynchronized modifications with concurrent users>, :backtrace=>["org/jruby/RubyArray.java:2486:in 'map'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/manticore/client.rb:336:in 'execute!'", "/usr/share/logstash/vendor/local_gems/0626d145/logstash-output-clickhouse-0.1.0/lib/logstash/outputs/clickhouse.rb:208:in 'block in make_request'"]}
I tried to change the code and make client.execute!
on line 208 not in new thead but in the current thread and the error stopped to occur and everything started to work without problems.
So, I wonder if I'm only one who experiencing such a problem and why client.execute!
is made in a separate thread whereas we specify :async = true
flag in the client.send()
call. To me it looks like logstash-mixin-http_client is not suited to work in multiple threads, but I may be wrong.
I run logstash with -w 1
flag to limit the number of workers to 1.
Would be grateful for any comments and ideas. Thanks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.