It seems that the polyaxon-rabbitmq-ha-0 is not starting.
I used the following command to install the chart.
sudo polyaxon admin deploy -f config.yml
2020-03-08 02:19:38.666 [info] <0.8.0> Feature flags: list of feature flags found:
2020-03-08 02:19:38.666 [info] <0.8.0> Feature flags: [ ] drop_unroutable_metric
2020-03-08 02:19:38.666 [info] <0.8.0> Feature flags: [ ] empty_basic_get_metric
2020-03-08 02:19:38.666 [info] <0.8.0> Feature flags: [ ] implicit_default_bindings
2020-03-08 02:19:38.666 [info] <0.8.0> Feature flags: [ ] quorum_queue
2020-03-08 02:19:38.666 [info] <0.8.0> Feature flags: [ ] virtual_host_metadata
2020-03-08 02:19:38.666 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2020-03-08 02:19:38.698 [info] <0.266.0> ra: meta data store initialised. 0 record(s) recovered
2020-03-08 02:19:38.698 [info] <0.271.0> WAL: recovering ["/var/lib/rabbitmq/mnesia/rabbit@polyaxon-rabbitmq-ha-0.polyaxon-rabbitmq-ha-discovery.polyaxon.svc.cluster.local/quorum/rabbit@polyaxon-rabbitmq-ha-0.polyaxon-rabbitmq-ha-discovery.polyaxon.svc.cluster.local/00000003.wal"]
2020-03-08 02:19:38.699 [info] <0.275.0>
Starting RabbitMQ 3.8.0 on Erlang 22.1.5
Copyright (C) 2007-2019 Pivotal Software, Inc.
Licensed under the MPL. See https://www.rabbitmq.com/
## ## RabbitMQ 3.8.0
## ##
########## Copyright (C) 2007-2019 Pivotal Software, Inc.
###### ##
########## Licensed under the MPL. See https://www.rabbitmq.com/
Doc guides: https://rabbitmq.com/documentation.html
Support: https://rabbitmq.com/contact.html
Tutorials: https://rabbitmq.com/getstarted.html
Monitoring: https://rabbitmq.com/monitoring.html
Logs: <stdout>
Config file(s): /etc/rabbitmq/rabbitmq.conf
Starting broker...2020-03-08 02:19:38.699 [info] <0.275.0>
node : rabbit@polyaxon-rabbitmq-ha-0.polyaxon-rabbitmq-ha-discovery.polyaxon.svc.cluster.local
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.conf
cookie hash : z9Efk6foMzTWv7yMOCE7Sg==
log(s) : <stdout>
database dir : /var/lib/rabbitmq/mnesia/rabbit@polyaxon-rabbitmq-ha-0.polyaxon-rabbitmq-ha-discovery.polyaxon.svc.cluster.local
2020-03-08 02:19:38.712 [info] <0.275.0> Running boot step pre_boot defined by app rabbit
2020-03-08 02:19:38.712 [info] <0.275.0> Running boot step rabbit_core_metrics defined by app rabbit
2020-03-08 02:19:38.713 [info] <0.275.0> Running boot step rabbit_alarm defined by app rabbit
2020-03-08 02:19:38.715 [info] <0.281.0> Memory high watermark set to 244 MiB (256000000 bytes) of 64408 MiB (67536941056 bytes) total
2020-03-08 02:19:38.717 [info] <0.283.0> Enabling free disk space monitoring
2020-03-08 02:19:38.717 [info] <0.283.0> Disk free limit set to 50MB
2020-03-08 02:19:38.719 [info] <0.275.0> Running boot step code_server_cache defined by app rabbit
2020-03-08 02:19:38.719 [info] <0.275.0> Running boot step file_handle_cache defined by app rabbit
2020-03-08 02:19:38.719 [info] <0.286.0> Limiting to approx 65436 file handles (58890 sockets)
2020-03-08 02:19:38.720 [info] <0.287.0> FHC read buffering: OFF
2020-03-08 02:19:38.720 [info] <0.287.0> FHC write buffering: ON
2020-03-08 02:19:38.720 [info] <0.275.0> Running boot step worker_pool defined by app rabbit
2020-03-08 02:19:38.720 [info] <0.276.0> Will use 16 processes for default worker pool
2020-03-08 02:19:38.720 [info] <0.276.0> Starting worker pool 'worker_pool' with 16 processes in it
2020-03-08 02:19:38.720 [info] <0.275.0> Running boot step database defined by app rabbit
2020-03-08 02:19:38.720 [info] <0.275.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@polyaxon-rabbitmq-ha-0.polyaxon-rabbitmq-ha-discovery.polyaxon.svc.cluster.local is empty. Assuming we need to join an existing cluster or initialise from scratch...
2020-03-08 02:19:38.720 [info] <0.275.0> Configured peer discovery backend: rabbit_peer_discovery_k8s
2020-03-08 02:19:38.720 [info] <0.275.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2020-03-08 02:19:38.721 [info] <0.275.0> Peer discovery backend does not support locking, falling back to randomized delay
2020-03-08 02:19:38.721 [info] <0.275.0> Peer discovery backend rabbit_peer_discovery_k8s supports registration.
2020-03-08 02:19:38.721 [info] <0.275.0> Will wait for 1820 milliseconds before proceeding with registration...
2020-03-08 02:19:40.588 [info] <0.275.0> Failed to get nodes from k8s - {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}},
{inet,[inet],nxdomain}]}
2020-03-08 02:19:40.589 [error] <0.274.0> CRASH REPORT Process <0.274.0> with 0 neighbours exited with reason: no case clause matching {error,"{failed_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\n {inet,[inet],nxdomain}]}"} in rabbit_mnesia:init_from_config/0 line 140 in application_master:init/4 line 138
2020-03-08 02:19:40.589 [info] <0.43.0> Application rabbit exited with reason: no case clause matching {error,"{failed_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\n {inet,[inet],nxdomain}]}"} in rabbit_mnesia:init_from_config/0 line 140
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,\"{failed_connect,[{to_address,{\\"kubernetes.default.svc.cluster.local\\",443}},\n {inet,[inet],nxdomain}]}\"}},[{rabbit_mnesia,init_from_config,0,[{file,\"src/rabbit_mnesia.erl\"},{line,140}]},{rabbit_mnesia,init_with_lock,3,[{file,\"src/rabbit_mnesia.erl\"},{line,120}]},{rabbit_mnesia,init,0,[{file,\"src/rabbit_mnesia.erl\"},{line,87}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,55}]},{rabbit_boot_steps,run_step,2,[{file,\"src/rabbit_boot_steps.erl\"},{line,59}]},{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,28}]},{rabbit_boot_steps,run_boot_steps,1,[{file,\"src/rabbit_boot_steps.erl\"},{line,29}]},{rabbit,start,2,[{file,\"src/rabbit.erl\"},{line,975}]}]}}}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,"{failed_connect,[{to_address,{\"kubernetes.defau
Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.6", GitCommit:"72c30166b2105cd7d3350f2c28a219e6abcd79eb", GitTreeState:"clean", BuildDate:"2020-01-18T23:31:31Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.6", GitCommit:"72c30166b2105cd7d3350f2c28a219e6abcd79eb", GitTreeState:"clean", BuildDate:"2020-01-18T23:23:21Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Client: &version.Version{SemVer:"v2.15.1", GitCommit:"cf1de4f8ba70eded310918a8af3a96bfe8e7683b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.15.1", GitCommit:"cf1de4f8ba70eded310918a8af3a96bfe8e7683b", GitTreeState:"clean"}
---
Metadata-Version: 2.1
Name: polyaxon-cli
Version: 0.6.0
Summary: Command Line Interface (CLI) for Polyaxon.
Home-page: https://github.com/polyaxon/polyaxon-cli
Author: Mourad Mourafiq
Author-email: [email protected]
Installer: pip
License: MIT
Location: /usr/local/lib/python3.5/dist-packages
Requires: polyaxon-client, raven, pathlib, polyaxon-deploy, click, tabulate, click-completion, polyaxon-dockerizer
Classifiers:
Programming Language :: Python
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.5
Programming Language :: Python :: 3.6
Programming Language :: Python :: 3.7
Operating System :: OS Independent
Intended Audience :: Developers
Intended Audience :: Science/Research
Topic :: Scientific/Engineering :: Artificial Intelligence
Entry-points:
[console_scripts]
polyaxon = polyaxon_cli.main:cli
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent