Can't use RBDSR as heartbeat-sr-uuid about rbdsr HOT 9 CLOSED

SR_PROBE is present in caps

from rbdsr.

Now it supports HA but it's not tested.

from rbdsr.

Hello! Thanks for adding support for HA. I just cloned master branch and tried creating a HA in my Xenserver pool but unfortunately it did't work. It detected that the SR had support for HA but when completing the guide in XenCenter o got an error:
"Failed","Enabling HA on pool 'Xenserver 7 Pool' This operation cannot be performed because VDI OpaqueRef:0cf4946e-9ac9-f4ec-74c2-4ef1b9023943 could not be properly attached to the VM. Time: 00:00:04","Xenserver 7 Pool","Aug 14, 2016 12:42 AM"

The interesting part of SMlog is here.
Aug 14 00:42:36 arb-xen3 SM: [4409] vdi_create {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'DummyRef:|4f4bf0ae-5aeb-d922-a868-d33346bdc16a|VDI.create', 'vdi_type': 'ha_statefile', 'args': ['266240', 'Statefile for HA', 'Used for storage heartbeating', '', 'false', '19700101T00:00:00Z', '', 'false'], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:bda94161-4075-fb8d-ea57-041741125c00', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_create', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_sm_config': {'type': 'raw'}}
Aug 14 00:42:36 arb-xen3 SM: [4409] RBDVDI.create for 1a20d597-4cc2-4da8-8424-d0ae33df4134
Aug 14 00:42:36 arb-xen3 SM: [4409] ['rbd', 'create', 'VHD-1a20d597-4cc2-4da8-8424-d0ae33df4134', '--size', '0', '--object-size', '2097152', '--image-format', '2', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 00:42:36 arb-xen3 SM: [4409] pread SUCCESS
Aug 14 00:42:36 arb-xen3 SM: [4409] ['rbd', 'image-meta', 'set', 'VHD-1a20d597-4cc2-4da8-8424-d0ae33df4134', 'VDI_LABEL', 'Statefile for HA', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 00:42:37 arb-xen3 SM: [4409] pread SUCCESS
Aug 14 00:42:37 arb-xen3 SM: [4409] ['rbd', 'image-meta', 'set', 'VHD-1a20d597-4cc2-4da8-8424-d0ae33df4134', 'VDI_DESCRIPTION', 'Used for storage heartbeating', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 00:42:37 arb-xen3 SM: [4409] pread SUCCESS
Aug 14 00:42:37 arb-xen3 SM: [4530] ['uuidgen', '-r']
Aug 14 00:42:37 arb-xen3 SM: [4530] pread SUCCESS
Aug 14 00:42:37 arb-xen3 SM: [4530] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Aug 14 00:42:37 arb-xen3 SM: [4530] pread SUCCESS
Aug 14 00:42:37 arb-xen3 SM: [4530] vdi_create {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'DummyRef:|9a6b0752-ce82-048e-92e9-9a5ca9c2944b|VDI.create', 'vdi_type': 'redo_log', 'args': ['268435456', 'Metadata for HA', 'Used for master failover', '', 'false', '19700101T00:00:00Z', '', 'false'], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:c7ddd03e-f779-0aa3-dfd9-852fa730c34b', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_create', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_sm_config': {'type': 'raw'}}
Aug 14 00:42:37 arb-xen3 SM: [4530] RBDVDI.create for c981e941-b2f6-44b2-9369-9fcc166a202c
Aug 14 00:42:37 arb-xen3 SM: [4530] ['rbd', 'create', 'VHD-c981e941-b2f6-44b2-9369-9fcc166a202c', '--size', '256', '--object-size', '2097152', '--image-format', '2', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 00:42:37 arb-xen3 SM: [4530] pread SUCCESS
Aug 14 00:42:37 arb-xen3 SM: [4530] ['rbd', 'image-meta', 'set', 'VHD-c981e941-b2f6-44b2-9369-9fcc166a202c', 'VDI_LABEL', 'Metadata for HA', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 00:42:38 arb-xen3 SM: [4530] pread SUCCESS
Aug 14 00:42:38 arb-xen3 SM: [4530] ['rbd', 'image-meta', 'set', 'VHD-c981e941-b2f6-44b2-9369-9fcc166a202c', 'VDI_DESCRIPTION', 'Used for master failover', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 00:42:38 arb-xen3 SM: [4530] pread SUCCESS
Aug 14 00:42:38 arb-xen3 SM: [4672] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Aug 14 00:42:39 arb-xen3 SM: [4672] pread SUCCESS
Aug 14 00:42:39 arb-xen3 SM: [4672] vdi_generate_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:bcdc23a7-ee2f-b781-ed21-ea3c508c1068', 'vdi_ref': 'OpaqueRef:0cf4946e-9ac9-f4ec-74c2-4ef1b9023943', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '1a20d597-4cc2-4da8-8424-d0ae33df4134', 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:1ee8c8d6-1ce1-f449-9314-886f5ef2ca6f', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_generate_config', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_uuid': '1a20d597-4cc2-4da8-8424-d0ae33df4134'}
Aug 14 00:42:39 arb-xen3 SM: [4672] RBDVDI.generate_config
Aug 14 00:42:39 arb-xen3 SM: [4672] ***** generic exception: vdi_generate_config: EXCEPTION <type 'exceptions.TypeError'>, coercing to Unicode: need string or buffer, NoneType found
Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 110, in run
Aug 14 00:42:39 arb-xen3 SM: [4672] return self._run_locked(sr)
Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Aug 14 00:42:39 arb-xen3 SM: [4672] rv = self._run(sr, target)
Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 282, in _run
Aug 14 00:42:39 arb-xen3 SM: [4672] return target.generate_config(self.params['sr_uuid'], self.vdi_uuid)
Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/RBDSR", line 654, in generate_config
Aug 14 00:42:39 arb-xen3 SM: [4672] if not util.pathexists(self.path):
Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/util.py", line 400, in pathexists
Aug 14 00:42:39 arb-xen3 SM: [4672] os.stat(path)
Aug 14 00:42:39 arb-xen3 SM: [4672]
Aug 14 00:42:39 arb-xen3 SM: [4672] ***** RBD: EXCEPTION <type 'exceptions.TypeError'>, coercing to Unicode: need string or buffer, NoneType found
Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 352, in run
Aug 14 00:42:39 arb-xen3 SM: [4672] ret = cmd.run(sr)
Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 110, in run
Aug 14 00:42:39 arb-xen3 SM: [4672] return self._run_locked(sr)
Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Aug 14 00:42:39 arb-xen3 SM: [4672] rv = self._run(sr, target)
Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/SRCommand.py", line 282, in _run
Aug 14 00:42:39 arb-xen3 SM: [4672] return target.generate_config(self.params['sr_uuid'], self.vdi_uuid)
Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/RBDSR", line 654, in generate_config
Aug 14 00:42:39 arb-xen3 SM: [4672] if not util.pathexists(self.path):
Aug 14 00:42:39 arb-xen3 SM: [4672] File "/opt/xensource/sm/util.py", line 400, in pathexists
Aug 14 00:42:39 arb-xen3 SM: [4672] os.stat(path)
Aug 14 00:42:39 arb-xen3 SM: [4672]

If any more information is needed please just tell me what and i will try to extract it.

from rbdsr.

Hi,
Fixed. Please check.

from rbdsr.

Thank you for a really quick fix. Unfortunately the same kind of error in XenCenter when activating ha.

"Failed","Enabling HA on pool 'Xenserver 7 Pool' This operation cannot be performed because VDI OpaqueRef:524feed2-f5e4-ce90-dacb-84b807139caa could not be properly attached to the VM. Time: 00:00:05","Xenserver 7 Pool","Aug 14, 2016 10:29 AM"

This is the SMlog, a new type of error there.

Aug 14 10:29:23 arb-xen3 SM: [8581] sr_scan {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|6a2105c1-68f1-d82b-5e9d-22355fb36b35|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:f0ba5026-762e-d8d5-8bf2-c61297294664', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}
Aug 14 10:29:23 arb-xen3 SM: [8596] sr_update {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|69dc7dd0-5f3a-99a6-6ca4-6503d6098556|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:28c7cade-a113-9c3c-988a-51de34d47241', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}
Aug 14 10:29:33 arb-xen3 SM: [8648] ['uuidgen', '-r']
Aug 14 10:29:33 arb-xen3 SM: [8648] pread SUCCESS
Aug 14 10:29:33 arb-xen3 SM: [8648] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Aug 14 10:29:33 arb-xen3 SM: [8648] pread SUCCESS
Aug 14 10:29:33 arb-xen3 SM: [8648] vdi_create {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'DummyRef:|1de2660f-67d4-9511-36d8-91a1f7512429|VDI.create', 'vdi_type': 'ha_statefile', 'args': ['266240', 'Statefile for HA', 'Used for storage heartbeating', '', 'false', '19700101T00:00:00Z', '', 'false'], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:ebcd9c4d-6525-0b91-ffe7-54c11af79797', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_create', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_sm_config': {'type': 'raw'}}
Aug 14 10:29:33 arb-xen3 SM: [8648] RBDVDI.create for fc8df789-83c0-4ad7-b128-3e88e12b3c53
Aug 14 10:29:33 arb-xen3 SM: [8648] ['rbd', 'create', 'VHD-fc8df789-83c0-4ad7-b128-3e88e12b3c53', '--size', '0', '--object-size', '2097152', '--image-format', '2', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 10:29:34 arb-xen3 SM: [8648] pread SUCCESS
Aug 14 10:29:34 arb-xen3 SM: [8648] ['rbd', 'image-meta', 'set', 'VHD-fc8df789-83c0-4ad7-b128-3e88e12b3c53', 'VDI_LABEL', 'Statefile for HA', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 10:29:34 arb-xen3 SM: [8648] pread SUCCESS
Aug 14 10:29:34 arb-xen3 SM: [8648] ['rbd', 'image-meta', 'set', 'VHD-fc8df789-83c0-4ad7-b128-3e88e12b3c53', 'VDI_DESCRIPTION', 'Used for storage heartbeating', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 10:29:34 arb-xen3 SM: [8648] pread SUCCESS
Aug 14 10:29:34 arb-xen3 SM: [8783] ['uuidgen', '-r']
Aug 14 10:29:34 arb-xen3 SM: [8783] pread SUCCESS
Aug 14 10:29:34 arb-xen3 SM: [8783] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Aug 14 10:29:34 arb-xen3 SM: [8783] pread SUCCESS
Aug 14 10:29:34 arb-xen3 SM: [8783] vdi_create {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'DummyRef:|51afc66f-0528-a4be-be5b-220edddd3402|VDI.create', 'vdi_type': 'redo_log', 'args': ['268435456', 'Metadata for HA', 'Used for master failover', '', 'false', '19700101T00:00:00Z', '', 'false'], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:8e01113f-b68c-cfc8-ef7f-4d806e15b602', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_create', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_sm_config': {'type': 'raw'}}
Aug 14 10:29:34 arb-xen3 SM: [8783] RBDVDI.create for aa386559-0f70-474a-8e84-1e309f6f2156
Aug 14 10:29:34 arb-xen3 SM: [8783] ['rbd', 'create', 'VHD-aa386559-0f70-474a-8e84-1e309f6f2156', '--size', '256', '--object-size', '2097152', '--image-format', '2', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 10:29:35 arb-xen3 SM: [8783] pread SUCCESS
Aug 14 10:29:35 arb-xen3 SM: [8783] ['rbd', 'image-meta', 'set', 'VHD-aa386559-0f70-474a-8e84-1e309f6f2156', 'VDI_LABEL', 'Metadata for HA', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 10:29:35 arb-xen3 SM: [8783] pread SUCCESS
Aug 14 10:29:35 arb-xen3 SM: [8783] ['rbd', 'image-meta', 'set', 'VHD-aa386559-0f70-474a-8e84-1e309f6f2156', 'VDI_DESCRIPTION', 'Used for master failover', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Aug 14 10:29:35 arb-xen3 SM: [8783] pread SUCCESS
Aug 14 10:29:35 arb-xen3 SM: [8921] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Aug 14 10:29:36 arb-xen3 SM: [8921] pread SUCCESS
Aug 14 10:29:36 arb-xen3 SM: [8921] vdi_generate_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:27f868d0-e195-2f92-7dea-5294e6d86180', 'vdi_ref': 'OpaqueRef:524feed2-f5e4-ce90-dacb-84b807139caa', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': 'fc8df789-83c0-4ad7-b128-3e88e12b3c53', 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:31298192-e6d0-a3a1-9cca-985307626720', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_generate_config', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_uuid': 'fc8df789-83c0-4ad7-b128-3e88e12b3c53'}
Aug 14 10:29:36 arb-xen3 SM: [8921] RBDVDI.generate_config
Aug 14 10:29:36 arb-xen3 SM: [8952] Failed to parse commandline; exception = 'device_config' argv = ['/opt/xensource/sm/RBDSR', "\n\nvdi_attach_from_config\n\n\n\n\nsr_uuid\n4ceb0f8a-1539-40a4-bee2-450a025b04e1\n\n\ncommand\nvdi_attach_from_config\n\n\nvdi_uuid\nfc8df789-83c0-4ad7-b128-3e88e12b3c53\n\n\n\n\n"]
Aug 14 10:29:36 arb-xen3 SM: [8952] Raising exception [143, Failed to parse the request]
Aug 14 10:29:36 arb-xen3 SM: [8952] **** RBD: EXCEPTION <class 'SR.SROSError'>, Failed to parse the request
Aug 14 10:29:36 arb-xen3 SM: [8952] File "/opt/xensource/sm/SRCommand.py", line 348, in run
Aug 14 10:29:36 arb-xen3 SM: [8952] cmd.parse()
Aug 14 10:29:36 arb-xen3 SM: [8952] File "/opt/xensource/sm/SRCommand.py", line 101, in parse
Aug 14 10:29:36 arb-xen3 SM: [8952] raise xs_errors.XenError('BadRequest')
Aug 14 10:29:36 arb-xen3 SM: [8952] File "/opt/xensource/sm/xs_errors.py", line 52, in init
Aug 14 10:29:36 arb-xen3 SM: [8952] raise SR.SROSError(errorcode, errormessage)
Aug 14 10:29:36 arb-xen3 SM: [8952]
Aug 14 10:29:36 arb-xen3 SM: [8956] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Aug 14 10:29:37 arb-xen3 SM: [8956] pread SUCCESS
Aug 14 10:29:37 arb-xen3 SM: [8956] vdi_detach {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:677b70f6-73f8-8912-53f3-e78f7a3351a9', 'vdi_ref': 'OpaqueRef:524feed2-f5e4-ce90-dacb-84b807139caa', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': 'fc8df789-83c0-4ad7-b128-3e88e12b3c53', 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:025bba04-b2d0-4ad0-80d8-16eb29cbb35b', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_detach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:ca45453a-eeb5-1da1-837c-607133a54114', 'vdi_uuid': 'fc8df789-83c0-4ad7-b128-3e88e12b3c53'}
Aug 14 10:29:37 arb-xen3 SM: [8956] lock: opening lock file /var/lock/sm/fc8df789-83c0-4ad7-b128-3e88e12b3c53/vdi
Aug 14 10:29:37 arb-xen3 SM: [8956] lock: closed /var/lock/sm/fc8df789-83c0-4ad7-b128-3e88e12b3c53/vdi
Aug 14 10:29:53 arb-xen3 SM: [9061] sr_scan {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|fbe92242-3794-8377-e674-0648bc28fdd8|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:7999bf48-7189-6fe9-64ec-8b7f810c4ff9', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}
Aug 14 10:29:53 arb-xen3 SM: [9076] sr_update {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|797fa15d-6698-ebb8-2e10-752a08dbb368|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:e36b391c-0a8b-5758-d424-8a5be0f149fe', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}
Aug 14 10:30:23 arb-xen3 SM: [9212] sr_scan {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|3e27d6ef-36cc-32cf-afbd-74acd87b8298|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:b455db94-1f5a-de2e-bc02-05a899fd6b41', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}
Aug 14 10:30:23 arb-xen3 SM: [9227] sr_update {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|2d5999c7-2a1f-382e-d4d0-e01d37defbc4|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:e5da20c4-47da-7e6d-8555-81a93b3c9f4d', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}
Aug 14 10:30:53 arb-xen3 SM: [9357] sr_scan {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|c7d32ae8-fda3-5cea-0f78-8f475ab802a2|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:a916e1d9-3130-3f33-e1a7-65a5760eeff5', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}
Aug 14 10:30:53 arb-xen3 SM: [9372] sr_update {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|6a828226-379b-90a3-1e6b-0c9e2c919f1a|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:a3157a1e-1302-176d-fa8b-2d0f0db77095', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}
Aug 14 10:31:23 arb-xen3 SM: [9502] sr_scan {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|7e4fcf49-bfba-b6e2-d2dd-92863aec0dad|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:9dcee133-6bd6-8116-cf9e-fa9b76ea7646', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}
Aug 14 10:31:23 arb-xen3 SM: [9518] sr_update {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|73277515-dec7-0ed0-8fdb-4b2b391b97ab|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:1085dfe4-7d8c-659b-7bf9-4fd812f3da05', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}
Aug 14 10:31:24 arb-xen3 SMGC: [6342] GC active, quiet period ended
Aug 14 10:31:24 arb-xen3 SM: [6342] lock: tried lock /var/lock/sm/17d872f0-20f8-2e87-de98-5110566873b8/sr, acquired: True (exists: True)
Aug 14 10:31:24 arb-xen3 SM: [6342] LVMCache: refreshing
Aug 14 10:31:24 arb-xen3 SM: [6342] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-17d872f0-20f8-2e87-de98-5110566873b8']
Aug 14 10:31:24 arb-xen3 SM: [6342] pread SUCCESS
Aug 14 10:31:24 arb-xen3 SM: [6342] ['/usr/bin/vhd-util', 'scan', '-f', '-c', '-m', 'VHD-', '-l', 'VG_XenStorage-17d872f0-20f8-2e87-de98-5110566873b8']
Aug 14 10:31:24 arb-xen3 SM: [6342] pread SUCCESS
Aug 14 10:31:24 arb-xen3 SMGC: [6342] SR 17d8 ('local storage') (1 VDIs in 1 VHD trees):
Aug 14 10:31:24 arb-xen3 SMGC: [6342] cb45e34cVHD
Aug 14 10:31:24 arb-xen3 SMGC: [6342]
Aug 14 10:31:24 arb-xen3 SM: [6342] lock: released /var/lock/sm/17d872f0-20f8-2e87-de98-5110566873b8/sr
Aug 14 10:31:24 arb-xen3 SM: [6342] ['/sbin/vgs', '--noheadings', '--nosuffix', '--units', 'b', 'VG_XenStorage-17d872f0-20f8-2e87-de98-5110566873b8']
Aug 14 10:31:24 arb-xen3 SM: [6342] pread SUCCESS
Aug 14 10:31:24 arb-xen3 SM: [6342] ['/sbin/vgs', '--noheadings', '--nosuffix', '--units', 'b', 'VG_XenStorage-17d872f0-20f8-2e87-de98-5110566873b8']
Aug 14 10:31:24 arb-xen3 SM: [6342] pread SUCCESS
Aug 14 10:31:24 arb-xen3 SMGC: [6342] No work, exiting
Aug 14 10:31:24 arb-xen3 SMGC: [6342] GC process exiting, no work left
Aug 14 10:31:24 arb-xen3 SM: [6342] lock: released /var/lock/sm/17d872f0-20f8-2e87-de98-5110566873b8/gc_active
Aug 14 10:31:24 arb-xen3 SMGC: [6342] SR 17d8 ('local storage') (1 VDIs in 1 VHD trees): no changes
Aug 14 10:31:53 arb-xen3 SM: [9656] sr_scan {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|15d87e62-fcd6-5eb1-f35f-7186c6a92035|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:2fe38590-9266-29ee-d6b3-98130bb34cd8', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}
Aug 14 10:31:53 arb-xen3 SM: [9675] sr_update {'sr_uuid': '88c59ccb-b00a-9440-4b2c-bec91e664cfb', 'subtask_of': 'DummyRef:|4643c98d-4a1a-90f4-c4dd-738a84732036|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:3d549b1c-8571-3d40-9a35-aa60c1da6129', 'session_ref': 'OpaqueRef:13fff3d7-85c4-ee68-c34e-cdeebe276237', 'device_config': {'username': 'root', 'cifspassword_secret': '3f7b5af7-df1c-636d-9956-6f9d39e453c7', 'iso_path': '/iso', 'location': '//192.168.9.65/ceph1', 'type': 'cifs', 'SRmaster': 'true'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:c94900cc-c9d5-e0a0-d631-8e2fbe628905'}

from rbdsr.

One more fix

from rbdsr.

Hello again!

Great! Now the guide to enable HA completes without any errors. Also if any protected VM is shutdown it's restarted automatically. So far so good.
What i'm not able to test is is to kill one Xenserver and make sure protected VM:s on the killed Xenserver is booted on another one in the pool. This is because i only have one server in my Xenserver 7 Pool and i'm on holidays this week. I will test this as soon as i get back so i can install another server.

from rbdsr.

Hi there!

Having problems with HA via ceph too. 3 nodes, no VMs, trying to enable HA. Got this in SMlog:

Sep 19 14:51:11 alahd-xen2 SM: [22201] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Sep 19 14:51:12 alahd-xen2 SM: [22201] pread SUCCESS
Sep 19 14:51:12 alahd-xen2 SM: [22201] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Sep 19 14:51:12 alahd-xen2 SM: [22201] pread SUCCESS
Sep 19 14:51:12 alahd-xen2 SM: [22201] vdi_generate_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:f29e6641-87fa-d985-9268-107822105265', 'vdi_ref': 'OpaqueRef:26516ae1-cae0-4319-ccb1-aaff2a2af419', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '78edbe81-3530-4cc7-8834-dca1c385f99f', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:d0e98f1a-3e25-f983-af2d-9f5e907b3234', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_generate_config', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '78edbe81-3530-4cc7-8834-dca1c385f99f'}
Sep 19 14:51:12 alahd-xen2 SM: [22201] RBDVDI.generate_config
Sep 19 14:51:12 alahd-xen2 SM: [22254] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Sep 19 14:51:12 alahd-xen2 SM: [22254] pread SUCCESS
Sep 19 14:51:12 alahd-xen2 SM: [22254] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Sep 19 14:51:12 alahd-xen2 SM: [22254] pread SUCCESS
Sep 19 14:51:12 alahd-xen2 SM: [22254] vdi_attach_from_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_attach_from_config', 'vdi_uuid': '78edbe81-3530-4cc7-8834-dca1c385f99f'}
Sep 19 14:51:12 alahd-xen2 SM: [22254] RBDVDI.attach_from_config
Sep 19 14:51:12 alahd-xen2 SM: [22254] RBDSR.attach for 4ceb0f8a-1539-40a4-bee2-450a025b04e1
Sep 19 14:51:12 alahd-xen2 SM: [22254] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Sep 19 14:51:13 alahd-xen2 SM: [22254] pread SUCCESS
Sep 19 14:51:13 alahd-xen2 SM: [22254] ['mkdir', '-p', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1']
Sep 19 14:51:13 alahd-xen2 SM: [22254] pread SUCCESS
Sep 19 14:51:13 alahd-xen2 SM: [22254] ['rbd-nbd', '--nbds_max', '64', 'map', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-78edbe81-3530-4cc7-8834-dca1c385f99f', '--name', 'client.admin']
Sep 19 14:51:13 alahd-xen2 SM: [22254] pread SUCCESS
Sep 19 14:51:13 alahd-xen2 SM: [22254] ['ln', '-s', '/dev/nbd0', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-78edbe81-3530-4cc7-8834-dca1c385f99f']
Sep 19 14:51:13 alahd-xen2 SM: [22254] pread SUCCESS
Sep 19 14:51:13 alahd-xen2 SM: [22379] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Sep 19 14:51:14 alahd-xen2 SM: [22379] pread SUCCESS
Sep 19 14:51:14 alahd-xen2 SM: [22379] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Sep 19 14:51:14 alahd-xen2 SM: [22379] pread SUCCESS
Sep 19 14:51:14 alahd-xen2 SM: [22379] vdi_generate_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:eb2dcbc5-786f-c551-b8d8-b24f7d12c684', 'vdi_ref': 'OpaqueRef:3f3a5442-2870-755e-df3f-be4db94df5a5', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:c6d7a2e4-6d1c-a156-299e-52c363f93cdc', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_generate_config', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'}
Sep 19 14:51:14 alahd-xen2 SM: [22379] RBDVDI.generate_config
Sep 19 14:51:14 alahd-xen2 SM: [22432] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Sep 19 14:51:14 alahd-xen2 SM: [22432] pread SUCCESS
Sep 19 14:51:14 alahd-xen2 SM: [22432] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Sep 19 14:51:14 alahd-xen2 SM: [22432] pread SUCCESS
Sep 19 14:51:14 alahd-xen2 SM: [22432] vdi_attach_from_config {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_attach_from_config', 'vdi_uuid': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'}
Sep 19 14:51:14 alahd-xen2 SM: [22432] RBDVDI.attach_from_config
Sep 19 14:51:14 alahd-xen2 SM: [22432] RBDSR.attach for 4ceb0f8a-1539-40a4-bee2-450a025b04e1
Sep 19 14:51:14 alahd-xen2 SM: [22432] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Sep 19 14:51:15 alahd-xen2 SM: [22432] pread SUCCESS
Sep 19 14:51:15 alahd-xen2 SM: [22432] ['mkdir', '-p', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1']
Sep 19 14:51:15 alahd-xen2 SM: [22432] pread SUCCESS
Sep 19 14:51:15 alahd-xen2 SM: [22432] ['rbd-nbd', '--nbds_max', '64', 'map', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1', '--name', 'client.admin']
Sep 19 14:51:15 alahd-xen2 SM: [22432] pread SUCCESS
Sep 19 14:51:15 alahd-xen2 SM: [22432] ['ln', '-s', '/dev/nbd1', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1']
Sep 19 14:51:15 alahd-xen2 SM: [22432] pread SUCCESS

Then freeze for ~2 minutes, in xensource.log got:

Sep 19 14:53:19 alahd-xen2 xapi: [error|alahd-xen2|579 INET :::80|host.ha_join_liveset R:194ce241ab08|xapi] HA enable failed with BOOTJOIN_TIMEOUT

And then in SMlog:

Sep 19 14:53:20 alahd-xen2 SM: [23049] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Sep 19 14:53:20 alahd-xen2 SM: [23049] pread SUCCESS
Sep 19 14:53:20 alahd-xen2 SM: [23049] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Sep 19 14:53:20 alahd-xen2 SM: [23049] pread SUCCESS
Sep 19 14:53:20 alahd-xen2 SM: [23049] vdi_deactivate {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:e3037904-79a3-c69a-e038-da1b5dd16e71', 'vdi_ref': 'OpaqueRef:26516ae1-cae0-4319-ccb1-aaff2a2af419', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '78edbe81-3530-4cc7-8834-dca1c385f99f', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:eb43896d-b93f-8308-8d17-2d98d3147028', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_deactivate', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '78edbe81-3530-4cc7-8834-dca1c385f99f'}
Sep 19 14:53:20 alahd-xen2 SM: [23049] lock: opening lock file /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi
Sep 19 14:53:20 alahd-xen2 SM: [23049] blktap2.deactivate
Sep 19 14:53:20 alahd-xen2 SM: [23049] lock: acquired /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi
Sep 19 14:53:20 alahd-xen2 SM: [23049] Backend path /dev/sm/backend/4ceb0f8a-1539-40a4-bee2-450a025b04e1/78edbe81-3530-4cc7-8834-dca1c385f99f does not exist
Sep 19 14:53:20 alahd-xen2 SM: [23049] Calling _unmap_VHD
Sep 19 14:53:20 alahd-xen2 SM: [23049] Calling ceph_plugin
Sep 19 14:53:20 alahd-xen2 SM: [23049] Calling rbd/nbd map on localhost OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01
Sep 19 14:53:20 alahd-xen2 SM: [23114] ['realpath', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-78edbe81-3530-4cc7-8834-dca1c385f99f']
Sep 19 14:53:20 alahd-xen2 SM: [23114] pread SUCCESS
Sep 19 14:53:20 alahd-xen2 SM: [23114] ['unlink', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-78edbe81-3530-4cc7-8834-dca1c385f99f']
Sep 19 14:53:20 alahd-xen2 SM: [23114] pread SUCCESS
Sep 19 14:53:20 alahd-xen2 SM: [23114] ['rbd-nbd', 'unmap', '/dev/nbd0', '--name', 'client.admin']
Sep 19 14:53:20 alahd-xen2 SM: [23114] pread SUCCESS
Sep 19 14:53:20 alahd-xen2 SM: [23049] WARNING: host key host_OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01 not found!
Sep 19 14:53:20 alahd-xen2 SM: [23049] lock: released /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi
Sep 19 14:53:20 alahd-xen2 SM: [23049] lock: closed /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi
Sep 19 14:53:20 alahd-xen2 SM: [23126] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Sep 19 14:53:21 alahd-xen2 SM: [23126] pread SUCCESS
Sep 19 14:53:21 alahd-xen2 SM: [23126] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Sep 19 14:53:21 alahd-xen2 SM: [23126] pread SUCCESS
Sep 19 14:53:21 alahd-xen2 SM: [23126] vdi_detach {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:e3037904-79a3-c69a-e038-da1b5dd16e71', 'vdi_ref': 'OpaqueRef:26516ae1-cae0-4319-ccb1-aaff2a2af419', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '78edbe81-3530-4cc7-8834-dca1c385f99f', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:1001a030-6a52-fcb6-95af-fbafb348d29f', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_detach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '78edbe81-3530-4cc7-8834-dca1c385f99f'}
Sep 19 14:53:21 alahd-xen2 SM: [23126] lock: opening lock file /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi
Sep 19 14:53:21 alahd-xen2 SM: [23126] lock: closed /var/lock/sm/78edbe81-3530-4cc7-8834-dca1c385f99f/vdi
Sep 19 14:53:21 alahd-xen2 SM: [23182] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Sep 19 14:53:21 alahd-xen2 SM: [23182] pread SUCCESS
Sep 19 14:53:21 alahd-xen2 SM: [23182] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Sep 19 14:53:21 alahd-xen2 SM: [23182] pread SUCCESS
Sep 19 14:53:21 alahd-xen2 SM: [23182] vdi_deactivate {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:e3037904-79a3-c69a-e038-da1b5dd16e71', 'vdi_ref': 'OpaqueRef:3f3a5442-2870-755e-df3f-be4db94df5a5', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:8690e36e-1b69-5633-7fa3-cfb87b3f3e07', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_deactivate', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'}
Sep 19 14:53:21 alahd-xen2 SM: [23182] lock: opening lock file /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi
Sep 19 14:53:21 alahd-xen2 SM: [23182] blktap2.deactivate
Sep 19 14:53:21 alahd-xen2 SM: [23182] lock: acquired /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi
Sep 19 14:53:21 alahd-xen2 SM: [23182] Backend path /dev/sm/backend/4ceb0f8a-1539-40a4-bee2-450a025b04e1/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1 does not exist
Sep 19 14:53:21 alahd-xen2 SM: [23182] Calling _unmap_VHD
Sep 19 14:53:21 alahd-xen2 SM: [23182] Calling ceph_plugin
Sep 19 14:53:21 alahd-xen2 SM: [23182] Calling rbd/nbd map on localhost OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01
Sep 19 14:53:22 alahd-xen2 SM: [23240] ['realpath', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1']
Sep 19 14:53:22 alahd-xen2 SM: [23240] pread SUCCESS
Sep 19 14:53:22 alahd-xen2 SM: [23240] ['unlink', '/dev/nbd/RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1/VHD-3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1']
Sep 19 14:53:22 alahd-xen2 SM: [23240] pread SUCCESS
Sep 19 14:53:22 alahd-xen2 SM: [23240] ['rbd-nbd', 'unmap', '/dev/nbd1', '--name', 'client.admin']
Sep 19 14:53:22 alahd-xen2 SM: [23240] pread SUCCESS
Sep 19 14:53:22 alahd-xen2 SM: [23182] WARNING: host key host_OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01 not found!
Sep 19 14:53:22 alahd-xen2 SM: [23182] lock: released /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi
Sep 19 14:53:22 alahd-xen2 SM: [23182] lock: closed /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi
Sep 19 14:53:22 alahd-xen2 SM: [23253] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Sep 19 14:53:22 alahd-xen2 SM: [23253] pread SUCCESS
Sep 19 14:53:22 alahd-xen2 SM: [23253] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-4ceb0f8a-1539-40a4-bee2-450a025b04e1', '--name', 'client.admin']
Sep 19 14:53:22 alahd-xen2 SM: [23253] pread SUCCESS
Sep 19 14:53:22 alahd-xen2 SM: [23253] vdi_detach {'sr_uuid': '4ceb0f8a-1539-40a4-bee2-450a025b04e1', 'subtask_of': 'OpaqueRef:e3037904-79a3-c69a-e038-da1b5dd16e71', 'vdi_ref': 'OpaqueRef:3f3a5442-2870-755e-df3f-be4db94df5a5', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1', 'host_ref': 'OpaqueRef:9b0bda3f-7a14-38c5-3f50-21aa5faa2a01', 'session_ref': 'OpaqueRef:411c2241-37e0-d09d-f14d-472f51b5fe18', 'device_config': {'SRmaster': 'false'}, 'command': 'vdi_detach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:224a0edd-4c3e-4e0d-4091-d3fd8b9c6ad9', 'vdi_uuid': '3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1'}
Sep 19 14:53:22 alahd-xen2 SM: [23253] lock: opening lock file /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi
Sep 19 14:53:22 alahd-xen2 SM: [23253] lock: closed /var/lock/sm/3ae2cb9f-36ef-42ba-81a8-4d5c3ecc96d1/vdi

In dmesg got:

[ 9163.277373] block nbd0: NBD_DISCONNECT
[ 9163.277484] block nbd0: Receive control failed (result -32)
[ 9163.277534] block nbd0: queue cleared
[ 9164.664371] block nbd1: NBD_DISCONNECT
[ 9164.664455] block nbd1: Receive control failed (result -32)
[ 9164.664504] block nbd1: queue cleared

Also tried to use kernel and fuse modes, but failed with different errors.
Tried to deploy few VMs with ceph - all works well, including live migration.

from rbdsr.

Hi,
By default in Jewel release and higher the librbd caching is enabled by default. This is was the reason why HA daemon couldn't start.
The problem can be addressed by disabling librbd caching by adding to ceph.conf

[client]
rbd cache = false

But it's not good idea to disable caching completely. The disabling librbd caching makes sense only for shared VDIs or for HA VDIs
So in the last fix the modified ceph.conf is made and rbd-nbd is invoked with this modified ceph.conf only for shared and HA VDIs. For all other VDIs rbd-nbd is invoked as usual.

from rbdsr.