Could there be an problem with the clusterchk.socket and systemd by filling up the system with so many file descriptors that the system can't no longer handle it?
xxxxxxxxxxxxx@pr-be-mdb-galera01-01:~$ sudo systemctl start reboot.target
Failed to start reboot.target: Argument list too long
See system logs and 'systemctl status reboot.target' for details.
xxxxxxxxxxxxxx@pr-be-mdb-galera01-01:~$ sudo systemctl status reboot.target
Failed to get properties: Unknown object '/org/freedesktop/systemd1/unit/reboot_2etarget'.
xxxxxxxxxxxxxx@pr-be-mdb-galera01-01:~$ sudo systemctl reboot
Failed to reboot system via logind: Invalid request descriptor
Failed to start reboot.target: Argument list too long
See system logs and 'systemctl status reboot.target' for details.
xxxxxxxxxxx@pr-be-mdb-galera01-01:~$ sudo journalctl --unit dbus
-- Journal begins at Tue 2023-11-21 22:04:31 CET, ends at Mon 2023-11-27 11:34:07 CET. --
Nov 27 07:31:47 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:32:12 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:32:37 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:33:02 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:33:27 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:33:52 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:34:17 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:34:42 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:35:07 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:35:32 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:35:57 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:36:22 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:36:47 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:37:12 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:37:37 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:38:02 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:38:27 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:38:52 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:39:17 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:39:42 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:40:07 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:40:32 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:40:57 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:41:22 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:41:47 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:42:12 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:42:37 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:43:02 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:43:27 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:43:52 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:44:17 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:44:42 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:45:07 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:45:32 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:45:57 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:46:22 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:46:47 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:47:12 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:47:37 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:48:02 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:48:27 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:48:52 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:49:17 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:49:42 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:50:07 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:50:32 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:50:57 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 07:51:19 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Successfully activated service 'org.freedesktop.systemd1'
Nov 27 08:15:03 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 08:15:28 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 08:15:53 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Nov 27 08:16:02 pr-be-mdb-galera01-01 dbus-daemon[477]: [system] Successfully activated service 'org.freedesktop.systemd1'
-- Boot 29a3a40650fe42d4a37bdce0bdf2b71d --
Nov 27 08:19:11 pr-be-mdb-galera01-01 systemd[1]: Started D-Bus System Message Bus.
Nov 27 08:19:28 pr-be-mdb-galera01-01 systemd[1]: Stopping D-Bus System Message Bus...
Nov 27 08:19:28 pr-be-mdb-galera01-01 systemd[1]: dbus.service: Succeeded.
Nov 27 08:19:28 pr-be-mdb-galera01-01 systemd[1]: Stopped D-Bus System Message Bus.
-- Boot 363a9b051c5d4b0289eec63faec54352 --
Nov 27 08:19:45 pr-be-mdb-galera01-01 systemd[1]: Started D-Bus System Message Bus.
xxxxxxxxxxxx@pr-be-mdb-galera01-01:~$ sudo journalctl --unit clusterchk.socket
-- Journal begins at Tue 2023-11-21 22:04:31 CET, ends at Mon 2023-11-27 11:35:11 CET. --
Nov 25 21:18:48 pr-be-mdb-galera01-01 systemd[1]: clusterchk.socket: Failed to queue service startup job (Maybe the service file is missing or not a template unit?): Argument list too long
Nov 25 21:18:48 pr-be-mdb-galera01-01 systemd[1]: clusterchk.socket: Failed with result 'resources'.
Nov 25 21:18:48 pr-be-mdb-galera01-01 systemd[1]: clusterchk.socket: Consumed 26min 57.357s CPU time.
Nov 27 07:51:19 pr-be-mdb-galera01-01 systemd[1]: Listening on Clusterchk socket.
Nov 27 07:51:32 pr-be-mdb-galera01-01 systemd[1]: clusterchk.socket: Failed to queue service startup job (Maybe the service file is missing or not a template unit?): Argument list too long
Nov 27 07:51:32 pr-be-mdb-galera01-01 systemd[1]: clusterchk.socket: Failed with result 'resources'.
-- Boot 29a3a40650fe42d4a37bdce0bdf2b71d --
Nov 27 08:19:11 pr-be-mdb-galera01-01 systemd[1]: Listening on Clusterchk socket.
Nov 27 08:19:33 pr-be-mdb-galera01-01 systemd[1]: clusterchk.socket: Succeeded.
Nov 27 08:19:33 pr-be-mdb-galera01-01 systemd[1]: Closed Clusterchk socket.
-- Boot 363a9b051c5d4b0289eec63faec54352 --
Nov 27 08:19:45 pr-be-mdb-galera01-01 systemd[1]: Listening on Clusterchk socket.
Nov 19 01:38:25 pr-be-mdb-galera01-01 systemd[1]: Started Check the status of Galera/MySQL (10.49.97.138:33982).
Nov 19 01:38:25 pr-be-mdb-galera01-01 systemd[1]: Started Check the status of Galera/MySQL (10.49.97.139:44600).
Nov 19 01:38:25 pr-be-mdb-galera01-01 clusterchk.sh[4010546]: /bin/echo: write error: Connection reset by peer
Nov 19 01:38:25 pr-be-mdb-galera01-01 clusterchk.sh[4010547]: /bin/echo: write error: Broken pipe
Nov 19 01:38:25 pr-be-mdb-galera01-01 clusterchk.sh[4010548]: /bin/echo: write error: Broken pipe
Nov 19 01:38:25 pr-be-mdb-galera01-01 clusterchk.sh[4010549]: /bin/echo: write error: Broken pipe
Nov 19 01:38:25 pr-be-mdb-galera01-01 systemd[1]: [email protected]:9999-10.49.97.138:33982.service: Main process exited, code=exited, status=1/FAILURE
Nov 19 01:38:25 pr-be-mdb-galera01-01 systemd[1]: [email protected]:9999-10.49.97.138:33982.service: Failed with result 'exit-code'.
Nov 19 01:38:25 pr-be-mdb-galera01-01 clusterchk.sh[4010555]: /bin/echo: write error: Connection reset by peer
Nov 19 01:38:25 pr-be-mdb-galera01-01 clusterchk.sh[4010556]: /bin/echo: write error: Broken pipe
Nov 19 01:38:25 pr-be-mdb-galera01-01 clusterchk.sh[4010557]: /bin/echo: write error: Broken pipe
Nov 19 01:38:25 pr-be-mdb-galera01-01 clusterchk.sh[4010558]: /bin/echo: write error: Broken pipe
Nov 19 01:38:25 pr-be-mdb-galera01-01 systemd[1]: [email protected]:9999-10.49.97.139:44600.service: Main process exited, code=exited, status=1/FAILURE
Nov 19 01:38:25 pr-be-mdb-galera01-01 systemd[1]: [email protected]:9999-10.49.97.139:44600.service: Failed with result 'exit-code'.
Nov 25 21:18:48 pr-be-mdb-galera01-01 systemd[1]: cannot add name, manager has too many units: Argument list too long
Nov 25 21:18:48 pr-be-mdb-galera01-01 systemd[1]: clusterchk.socket: Failed to queue service startup job (Maybe the service file is missing or not a template unit?): Argument list too long
Nov 25 21:18:48 pr-be-mdb-galera01-01 systemd[1]: clusterchk.socket: Failed with result 'resources'.
Nov 25 21:18:48 pr-be-mdb-galera01-01 systemd[1]: clusterchk.socket: Consumed 26min 57.357s CPU time.
Nov 26 00:00:08 pr-be-mdb-galera01-01 systemd[1]: cannot add name, manager has too many units: Argument list too long
Nov 26 00:00:08 pr-be-mdb-galera01-01 systemd[1]: cannot add name, manager has too many units: Argument list too long
Nov 26 00:00:08 pr-be-mdb-galera01-01 systemd[1]: cannot add name, manager has too many units: Argument list too long
It leads to an very slow system without the possibility to reboot (only hard reset at vm level possible).