elkninja / elastic-stack-docker-part-one Goto Github PK
View Code? Open in Web Editor NEWGetting up and running with Elastic Stack on Docker-Compose
License: Apache License 2.0
Getting up and running with Elastic Stack on Docker-Compose
License: Apache License 2.0
hi . i'm beginner in elastic and i want to khow how can i deploy this compose file in swarm mod ?
I have corrected a few issues and have the docker-compose project working.
I have committed the changes to my master branch in my repo.
https://github.com/fulldeck/elastic-stack-docker-part-one/tree/master
When using the provided docker-compose.yml
, the beats containers show a gazillion errors.
It seems to me the index setup was not performed (doc):
"Running Metricbeat with the setup command will create the index pattern and load visualizations , dashboards, and machine learning jobs"
docker run \
docker.elastic.co/beats/metricbeat:8.8.2 \
setup -E setup.kibana.host=kibana:5601 \
-E output.elasticsearch.hosts=["elasticsearch:9200"]
It would be great if:
--ensureIndex
command line optionError response from daemon: no such image: docker.elastic.co/beats/metricbeat:8.7.1: No such image: docker.elastic.co/beats/metricbeat:8.7.1
this is what the compiler complains
First of all I would like to thank you these docker configuration files. This has great added value to be able to set up and run a full ELK environment for developers in minutes 👏
I have successfully installed it in my local docker environment and it works great unless I make modifications in my .env file for the passwords.
Unfortunately the configuration files of the beats has the hardcoded password 'changeme' and therefore it cannot be modified.
Would be nice if you could fix it
run on ubuntu 20.04, 22.04
spec: 2core cpu 8gb ram
~# docker -v
Docker version 24.0.4, build 3713ee1
~# docker compose version
Docker Compose version v2.19.1
docker logs:
{"@timestamp":"2023-07-16T18:52:24.587Z", "log.level": "INFO", "message":"adding index lifecycle policy [synthetics-synthetics.http-default_policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","trace.id":"73ee250f29ad38b64403dec513c3306f","elasticsearch.cluster.uuid":"-KUubkq5Q0m4mYhf_BiGng","elasticsearch.node.id":"UGU5g84rQNaHDOIFPGX0JA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2023-07-16T18:52:27.631Z", "log.level": "INFO", "message":"[.ds-metricbeat-8.7.1-2023.07.16-000001/sF0X39DOR_W9ptBFBieBbA] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","elasticsearch.cluster.uuid":"-KUubkq5Q0m4mYhf_BiGng","elasticsearch.node.id":"UGU5g84rQNaHDOIFPGX0JA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2023-07-16T18:52:30.507Z", "log.level": "INFO", "message":"adding index lifecycle policy [synthetics-synthetics.icmp-default_policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","trace.id":"73ee250f29ad38b64403dec513c3306f","elasticsearch.cluster.uuid":"-KUubkq5Q0m4mYhf_BiGng","elasticsearch.node.id":"UGU5g84rQNaHDOIFPGX0JA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
ERROR: Elasticsearch exited unexpectedly
Tested with the predefined 8.7.1
and updated to 8.8.2
with the same result:
elastic-filebeat01-1 | {"log.level":"error","@timestamp":"2023-07-07T11:27:54.208Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":258},"message":"failed to perform any bulk index operations: Post \"https://es01:9200/_bulk\": EOF","service.name":"filebeat","ecs.version":"1.6.0"}
elastic-es01-1 |
elastic-es01-1 | ERROR: Elasticsearch exited unexpectedly
elastic-es01-1 exited with code 137
Running on Debian GNU/Linux 10 (buster) server with 64GB RAM, 50GB free.
Note, this did not help:
sysctl -w vm.max_map_count=262144
From https://www.elastic.co/cn/blog/getting-started-with-the-elastic-stack-and-docker-compose
Note, the Logstash output index name is "logstash-%{+YYYY.MM.dd}". To see the data, you will need to create a Data View for the “logstash-*” pattern, as seen below.
But when I create Data View, there are only 2 sources, filebeat-8.12.0 and metricbeat-8.12.0.
Logstash container logs :
[INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"index"=>"logstash-%{+YYYY.MM.dd}"}
Is there anything wrong ?
When trying to run the logstash portion of the compose it throws an error about the formatting of the .conf file. I am using the file that is provided in this repo and haven't made any changes to it.
Logstash error:
elastic-stack-logstash01-1 | [2023-12-23T05:23:21,079][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
elastic-stack-es01-1 | {"@timestamp":"2023-12-23T05:22:36.930Z", "log.level": "INFO", "message":"loaded module [x-pack-aggregate-metric]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-eck"}
elastic-stack-logstash01-1 | [2023-12-23T05:23:21,097][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 1, column 1 (byte 1)", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:931:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}
elastic-stack-logstash01-1 | [2023-12-23T05:23:21,104][INFO ][logstash.runner ] Logstash shut down.
elastic-stack-logstash01-1 | [2023-12-23T05:23:21,107][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
elastic-stack-logstash01-1 | org.jruby.exceptions.SystemExit: (SystemExit) exit
elastic-stack-logstash01-1 | at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:808) ~[jruby.jar:?]
elastic-stack-logstash01-1 | at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:767) ~[jruby.jar:?]
elastic-stack-logstash01-1 | at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
Any recommendations for trying to fix this issue?
Docker version 20.10.22, build 3a2c30b
Logs:
2023-08-24 11:37:56 elastic-stack-es01-1 | {"@timestamp":"2023-08-24T08:37:56.837Z", "log.level": "INFO", "message":"Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#31]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","trace.id":"20a7f7b926a8447a1f1fe8ff3a7585f7","elasticsearch.cluster.uuid":"pkl8MVp-SbyQ6JyphtwfHw","elasticsearch.node.id":"VQv0yDl2TsuA7fpoxy-VhA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-08-24 11:26:09 [2023-08-24T08:26:09.374+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
2023-08-24 11:26:14 [2023-08-24T08:26:14.654+00:00][INFO ][plugins-service] Plugin "cloudChat" is disabled.
2023-08-24 11:26:14 [2023-08-24T08:26:14.655+00:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled.
2023-08-24 11:26:14 [2023-08-24T08:26:14.656+00:00][INFO ][plugins-service] Plugin "cloudFullStory" is disabled.
2023-08-24 11:26:14 [2023-08-24T08:26:14.656+00:00][INFO ][plugins-service] Plugin "cloudGainsight" is disabled.
2023-08-24 11:26:14 [2023-08-24T08:26:14.659+00:00][INFO ][plugins-service] Plugin "profiling" is disabled.
2023-08-24 11:26:14 [2023-08-24T08:26:14.706+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
2023-08-24 11:26:14 [2023-08-24T08:26:14.727+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
2023-08-24 11:26:14 [2023-08-24T08:26:14.749+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
2023-08-24 11:26:14 [2023-08-24T08:26:14.900+00:00][INFO ][plugins-system.standard] Setting up [132] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,customBranding,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,ftrApis,fieldFormats,expressions,screenshotting,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,cloudDataMigration,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,contentManagement,bfetch,data,watcher,fileUpload,ingestPipelines,ecsDataQualityDashboard,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,eventAnnotation,embeddable,reporting,uiActionsEnhanced,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,aiops,discover,observability,fleet,osquery,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]
2023-08-24 11:26:14 [2023-08-24T08:26:14.907+00:00][INFO ][custom-branding-service] CustomBrandingService registering plugin: customBranding
2023-08-24 11:26:14 [2023-08-24T08:26:14.909+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 0909b472-d3d1-4ef9-85fb-a52f892b1b90
2023-08-24 11:26:14 [2023-08-24T08:26:14.952+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
2023-08-24 11:26:14 [2023-08-24T08:26:14.964+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
2023-08-24 11:26:14 [2023-08-24T08:26:14.969+00:00][INFO ][plugins.encryptedSavedObjects] Hashed 'xpack.encryptedSavedObjects.encryptionKey' for this instance: AzsZyxRIrXJpTS7qUJTSBc46aXvQHsPRSGXRyf7Cs7s=
2023-08-24 11:26:14 [2023-08-24T08:26:14.979+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
2023-08-24 11:26:15 [2023-08-24T08:26:15.044+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
2023-08-24 11:26:15 [2023-08-24T08:26:15.057+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
2023-08-24 11:26:15 [2023-08-24T08:26:15.200+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
2023-08-24 11:26:15 [2023-08-24T08:26:15.366+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
2023-08-24 11:26:15 [2023-08-24T08:26:15.397+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception
2023-08-24 11:26:15 Root causes:
2023-08-24 11:26:15 security_exception: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
2023-08-24 11:26:15 [2023-08-24T08:26:15.702+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
Thanks for your share,
I have one problem. Https not working correctly. SSL sertificate error.
I Try this code but not solve SSL error.
curl --cacert /tmp/ca.crt -u 'elastic:mypass!!!' https://localhost:9200
docker cp ako-elastic-full-paket-es01-1:/usr/share/elasticsearch/config/certs/ca/ca.crt /tmp/.
PS: I see the file in the tmp folder.
My Url:
https://10.10.50.69:9200 OS: Ubuntu
Hello!
I'm reading the article and everything has worked good until Metricbeat which I can't force to work.
When I start docker compose up -d
in Metricbeat logs I see the following:
user@ubuntu:~/elk$ docker logs elk-metricbeat01-1
Exiting: 9 errors: host parsing failed for elasticsearch-ccr: 'password' config for module elasticsearch is not a string; host parsing failed for elasticsearch-enrich: 'password' config for module elasticsearch is not a string; host parsing failed for elasticsearch-cluster_stats: 'password' config for module elasticsearch is not a string; host parsing failed for elasticsearch-index: 'password' config for module elasticsearch is not a string; host parsing failed for elasticsearch-index_recovery: 'password' config for module elasticsearch is not a string; host parsing failed for elasticsearch-index_summary: 'password' config for module elasticsearch is not a string; host parsing failed for elasticsearch-ml_job: 'password' config for module elasticsearch is not a string; host parsing failed for elasticsearch-node_stats: 'password' config for module elasticsearch is not a string; host parsing failed for elasticsearch-shard: 'password' config for module elasticsearch is not a string
But why?
This is my project folder (*.tar.gz are docker images archives which I've moved from another docker host):
user@ubuntu:~/elk$ ll
total 1113664
drwxrwxr-x 8 user user 4096 Jun 8 18:04 ./
drwxr-x--- 7 user user 4096 Jun 8 13:02 ../
drwxr-x--- 5 root root 4096 Jun 8 13:03 certs/
-rw-rw-r-- 1 user user 7139 Jun 8 18:00 docker-compose.yml
-rw-rw-r-- 1 user user 623698342 Jun 8 12:39 elasticsearch.tar.gz
-rw-rw-r-- 1 user user 877 Jun 8 12:39 .env
drwxr-xr-x 5 user user 4096 Jun 8 18:03 esdata01/
drwxrwxr-x 2 user user 4096 Jun 8 12:39 filebeatdata01/
drwxrwxr-x 8 user user 4096 Jun 8 18:01 kibanadata/
-rw-rw-r-- 1 user user 391381853 Jun 8 12:39 kibana.tar.gz
drwxrwxr-x 2 user user 4096 Jun 8 12:39 logstashdata01/
drwxrwxr-x 2 user user 4096 Jun 8 18:01 metricbeatdata01/
-rw-rw-r-- 1 user user 125247211 Jun 8 12:39 metricbeat.tar.gz
-rw-rw-r-- 1 user user 1174 Jun 8 17:57 metricbeat.yml
What I modified in comparison with original configs:
Elasticsearch and kibana work as expected:
user@ubuntu:~/elk$ docker compose ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
elk-es01-1 elasticsearch:8.14.0 "/bin/tini -- /usr/l…" es01 26 minutes ago Up 25 minutes (healthy) 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp
elk-kibana-1 kibana:8.14.0 "/bin/tini -- /usr/l…" kibana 25 minutes ago Up 25 minutes (healthy) 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp
Kibana's web interface can be opened:
In docker-compose.yml
, environment variables of filebeat01
are defined like this:
environment:
- ELASTIC_USER=elastic
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- ELASTIC_HOSTS=https://es01:9200
- KIBANA_HOSTS=http://kibana:5601
- LOGSTASH_HOSTS=http://logstash01:9600
I think the http://
and the port number might be incorrect in the LOGSTASH_HOSTS
variable. The variable isn't used in filebeat.yml
but it sure looks like it could be! During my tests using the logstash
output of Filebeat with SSL/TLS, keeping the protocol (but using port 5044 since this seems to be the convention for Beats, 9600 being the web API endpoint) produced this error:
Failed to connect to backoff(async(tcp://[https://logstash01:5044]:5044)): lookup https://logstash01:5044: no such host
And the link from Filebeat to Logstash doesn't work. The value between square brackets looks strange too?... According to the Filebeat configuration docs and how these variables will be used in filebeat.yml
:
setup.kibana.host
allows only one URL
or host:port
; It might be better to rename KIBANA_HOSTS
to KIBANA_HOST
since the settings expect only one? https://www.elastic.co/guide/en/beats/filebeat/current/setup-kibana-endpoint.htmloutput.elasticsearch.hosts
allows one or several URL
or host:port
https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#hosts-optionoutput.logstash.hosts
seem to allow one or several host:port
strings. Docs aren't very clear about this though: https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#hostsAfter using only host and port, some (expected) error message are seen but the Filebeat-Logstash link eventually works:
Failed to connect to backoff(async(tcp://logstash01:5044)): lookup logstash01 on 127.0.0.11:53: no such host
Failed to connect to backoff(async(tcp://logstash01:5044)): dial tcp 172.27.0.2:5044: connect: connection refused
Finally, I haven't tried putting several hosts in the variables but I wonder if it would work? Should they be comma-separated or something?
I did my best to explain this clearly but I'm still a little bit confused about all this.
TL;DR
LOGSTASH_HOSTS
makes Filebeat fail when using the Logstash outputKIBANA_HOSTS
to KIBANA_HOST
filebeat01-1 | /usr/local/bin/docker-entrypoint: line 25: exec: strict.perms=false: not found
filebeat01-1 exited with code 127
logstash01-1 | 2023/12/27 09:07:31 Setting 'xpack.monitoring.enabled' from environment.
logstash01-1 | /usr/local/bin/docker-entrypoint: line 30: exec: strict.perms=false: not found
metricbeat01-1 | /usr/local/bin/docker-entrypoint: line 25: exec: strict.perms=false: not found
logstash01-1 exited with code 127
metricbeat01-1 exited with code 127
I don't know if this is a Windows-specific error but when running this, the filebeat container fails with the following error:
Exiting: error loading config file: config file ("filebeat.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /usr/share/filebeat/filebeat.yml')
The same thing applies to the metricbeat container
Exiting: error loading config file: config file ("metricbeat.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /usr/share/metricbeat/metricbeat.yml')
A workaround for this is to add the command: --strict.perms=false -e
line in the docker-compose.yml
file for each one of the "problematic" containers,
Would you be interested in a PR with such fix?
This repo is great for our use case, thanks a lot first of all.
I wanted to integrate new Microsoft SQL connector client, using windows 11, docker with wsl.
I am able to setup connector in kibana ui.
After that changed config.yml of connector like below
But unfortunately im getting ssl certification error while running connector client container
[FMWK][23:12:00][ERROR] Could not connect to the server
[FMWK][23:12:00][ERROR] Cannot connect to host host.docker.internal:9200 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')]
SSL should be working even tough its POC. What I am missing here, any helps would be great.
Thanks
After starting docker container setup service exited
The logs of setup service:
Creating CA
Archive: config/certs/ca.zip
creating: config/certs/ca/
inflating: config/certs/ca/ca.crt
inflating: config/certs/ca/ca.key
Creating certs
Archive: config/certs/certs.zip
creating: config/certs/es01/
inflating: config/certs/es01/es01.crt
inflating: config/certs/es01/es01.key
creating: config/certs/kibana/
inflating: config/certs/kibana/kibana.crt
inflating: config/certs/kibana/kibana.key
Setting file permissions
Waiting for Elasticsearch availability
Setting kibana_system password
�
!
What can be reason of that?
Hi
I believe my request is far beyond the original scope of your project, but I consider this GitHub repo as a great start for elastic developers.
Adding a fleet server to the docker compose would be awesome
logstash Error log
[2023-11-22T20:13:59,330][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Temporary failure in name resolution"} [2023-11-22T20:14:28,842][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"} [2023-11-22T20:14:29,345][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch: Temporary failure in name resolution", :exception=>Manticore::ResolutionFailure, :cause=>#<Java::JavaNet::UnknownHostException: elasticsearch: Temporary failure in name resolution>} [2023-11-22T20:14:29,345][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Temporary failure in name resolution"}
logstash.conf
input {
file {
#https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html
#default is TAIL which assumes more data will come into the file.
#change to mode => "read" if the file is a compelte file. by default, the file will be removed once reading is complete -- backup your files if you need them.
mode => "tail"
path => "/usr/share/logstash/ingest_data/*"
}
}
filter {
}
output {
elasticsearch {
index => "logstash-%{+YYYY.MM.dd}"
hosts=> "${ELASTIC_HOSTS}"
user=> "${ELASTIC_USER}"
password=> "${ELASTIC_PASSWORD}"
cacert=> "certs/ca/ca.crt"
}
}
Hello.
Most of all, thank you for your post https://www.elastic.co/kr/blog/getting-started-with-the-elastic-stack-and-docker-compose
It was very helpful.
However, I've got error as bellow.
filebeat
elastic-stack-docker-part-one-filebeat01-1 | Exiting: error loading config file: config file ("filebeat.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /usr/share/filebeat/filebeat.yml') elastic-stack-docker-part-one-filebeat01-1 exited with code 1
metricbeat
2023-06-23 14:26:48 Exiting: error loading config file: config file ("metricbeat.yml") can only be writable by the owner but the permissions are "-rwxrwxrwx" (to fix the permissions use: 'chmod go-w /usr/share/metricbeat/metricbeat.yml')
It gave me suffer because i'm not pro user for docker.
Anyway, I figured out to solve it with giving command like bellow.
filebeat01:
depends_on:
es01:
condition: service_healthy
image: docker.elastic.co/beats/filebeat:${STACK_VERSION}
user: root
volumes:
- certs:/usr/share/filebeat/certs
- filebeatdata01:/usr/share/filebeat/data
- "./filebeat_ingest_data/:/usr/share/filebeat/ingest_data/"
- "./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro"
- "/var/lib/docker/containers:/var/lib/docker/containers:ro"
- "/var/run/docker.sock:/var/run/docker.sock:ro"
environment:
- ELASTIC_USER=elastic
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- ELASTIC_HOSTS=https://es01:9200
- KIBANA_HOSTS=http://kibana:5601
- LOGSTASH_HOSTS=http://logstash01:9600
command:
-strict.perms=false
metricbeat01:
depends_on:
es01:
condition: service_healthy
kibana:
condition: service_healthy
image: docker.elastic.co/beats/metricbeat:${STACK_VERSION}
user: root
volumes:
- certs:/usr/share/metricbeat/certs
- metricbeatdata01:/usr/share/metricbeat/data
- "./metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro"
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
- "/proc:/hostfs/proc:ro"
- "/:/hostfs:ro"
environment:
- ELASTIC_USER=elastic
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- ELASTIC_HOSTS=https://es01:9200
- KIBANA_HOSTS=http://kibana:5601
- LOGSTASH_HOSTS=http://logstash01:9600
command:
-strict.perms=false
I am issueing for anybody who suffers like me :)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.