A Python3-only functional test framework for Charms
License: Apache License 2.0
A Python3-only functional test framework for OpenStack Charms.
Source code: https://github.com/openstack-charmers/zaza
Bug reports: https://github.com/openstack-charmers/zaza/issues
Documentation: https://zaza.readthedocs.io/en/latest/
tox -e py3
tox -e pep8
tox -e docs
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
The tests.yaml allow bespoke workload status statuses and messages to be set. These are mostly useful during a deployment when some intervention is needed to ready the service. Once that action has been run the applications then have standard workload status settings but zaza is still looking for the old ones when assessing readyness.
An example of this is vault which comes up in a state of 'blocked' but once an action has been run goes into a state of 'active'. Any subsequent call to zaza to check application readyness hangs because vault is 'active' not 'blocked'.
Traceback (most recent call last):
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/charm_tests/keystone/tests.py", line 34, in test_901_pause_resume
self.pause_resume(['apache2'])
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/charm_tests/test_utils.py", line 146, in pause_resume
model_name=self.model_name)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/__init__.py", line 30, in _wrapper
return run(_run_it())
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/__init__.py", line 18, in run
return task.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/__init__.py", line 29, in _run_it
return await f(*args, **kwargs)
TypeError: async_run_action() got multiple values for argument 'model_name'
Hi,
In the case of the jenkins charm, it's status is 'Jenkins is running'. zaza will continue to wait because it doesn't see any of the approved message prefixes:
2019-06-20 16:30:15 [INFO] Deploy of bundle completed.
2019-06-20 16:30:16 [INFO] Waiting for environment to settle
2019-06-20 16:30:20 [INFO] Waiting for a unit to appear
2019-06-20 16:30:20 [INFO] Waiting for all units to be idle
2019-06-20 16:37:51 [INFO] Checking workload status of jenkins/0
2019-06-20 16:37:51 [INFO] Checking workload status message of jenkins/0
jenkins/0* active idle 0 10.55.32.25 8080/tcp,48484/tcp Jenkins is running
The cause seems to be a re-ordering of arguments to dependency async_run_on_unit
Traceback (most recent call last):
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/charm_tests/keystone/tests.py", line 34, in test_901_pause_resume
self.pause_resume(['keystone'])
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/charm_tests/test_utils.py", line 138, in pause_resume
model_name=self.model_name)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/__init__.py", line 30, in _wrapper
return run(_run_it())
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/__init__.py", line 18, in run
return task.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/__init__.py", line 29, in _run_it
return await f(*args, **kwargs)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/model.py", line 708, in async_block_until_service_status
await async_block_until(_check_service, timeout=timeout)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/model.py", line 776, in async_block_until
await asyncio.wait_for(_block(), timeout, loop=loop)
File "/usr/lib/python3.5/asyncio/tasks.py", line 392, in wait_for
return fut.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/model.py", line 770, in _block
result = await c()
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/model.py", line 700, in _check_service
timeout=timeout)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/model.py", line 220, in async_run_on_unit
async with run_in_model(model_name) as model:
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/async_generator/_util.py", line 34, in __aenter__
return await self._agen.asend(None)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/async_generator/_impl.py", line 366, in step
return await ANextIter(self._it, start_fn, *args)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/async_generator/_impl.py", line 197, in __next__
return self._invoke(first_fn, *first_args)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/async_generator/_impl.py", line 209, in _invoke
result = fn(*args)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/model.py", line 117, in run_in_model
await model.connect_model(model_name)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/juju/model.py", line 552, in connect_model
return await self.connect(model_name=model_name)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/juju/model.py", line 509, in connect
await self._connector.connect_model(model_name, **kwargs)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/juju/client/connector.py", line 126, in connect_model
raise JujuConnectionError('Model not found: {}'.format(model_name))
juju.errors.JujuConnectionError: Model not found: admin/pidof -x keystone
When running tests in several bundles, a new model is autocreated per bundle.
With the --keep-model flag models should not be destroyed after running tests. However when running multiple bundles only the last one is preserved
When trying to use get_unit_time like this:
mtime = zaza.model.get_unit_time(unit_name)I get the following stack trace:
Traceback (most recent call last):
File "/home/ubuntu/ceph-osd/.tox/func-smoke/lib/python3.6/site-packages/zaza/charm_tests/ceph/tests.py", line 278, in test_ceph_encryption
mtime = zaza_model.get_unit_time(unit_name)
File "/home/ubuntu/ceph-osd/.tox/func-smoke/lib/python3.6/site-packages/zaza/__init__.py", line 44, in _wrapper
return run(_run_it())
File "/home/ubuntu/ceph-osd/.tox/func-smoke/lib/python3.6/site-packages/zaza/__init__.py", line 32, in run
return task.result()
File "/home/ubuntu/ceph-osd/.tox/func-smoke/lib/python3.6/site-packages/zaza/__init__.py", line 43, in _run_it
return await f(*args, **kwargs)
File "/home/ubuntu/ceph-osd/.tox/func-smoke/lib/python3.6/site-packages/zaza/model.py", line 299, in async_get_unit_time
timeout=timeout)
File "/home/ubuntu/ceph-osd/.tox/func-smoke/lib/python3.6/site-packages/zaza/model.py", line 246, in async_run_on_unit
async with run_in_model(model_name) as model:
File "/home/ubuntu/ceph-osd/.tox/func-smoke/lib/python3.6/site-packages/async_generator/_util.py", line 34, in __aenter__
return await self._agen.asend(None)
File "/home/ubuntu/ceph-osd/.tox/func-smoke/lib/python3.6/site-packages/async_generator/_impl.py", line 366, in step
return await ANextIter(self._it, start_fn, *args)
File "/home/ubuntu/ceph-osd/.tox/func-smoke/lib/python3.6/site-packages/async_generator/_impl.py", line 197, in __next__
return self._invoke(first_fn, *first_args)
File "/home/ubuntu/ceph-osd/.tox/func-smoke/lib/python3.6/site-packages/async_generator/_impl.py", line 209, in _invoke
result = fn(*args)
File "/home/ubuntu/ceph-osd/.tox/func-smoke/lib/python3.6/site-packages/zaza/model.py", line 143, in run_in_model
await model.connect_model(model_name)
File "/home/ubuntu/ceph-osd/.tox/func-smoke/lib/python3.6/site-packages/juju/model.py", line 552, in connect_model
return await self.connect(model_name=model_name)
File "/home/ubuntu/ceph-osd/.tox/func-smoke/lib/python3.6/site-packages/juju/model.py", line 509, in connect
await self._connector.connect_model(model_name, **kwargs)
File "/home/ubuntu/ceph-osd/.tox/func-smoke/lib/python3.6/site-packages/juju/client/connector.py", line 126, in connect_model
raise JujuConnectionError('Model not found: {}'.format(model_name))
juju.errors.JujuConnectionError: Model not found: admin/date +'%s'
Lines 290 to 294 in 8b98b23
model_name, unit_name and command where it should have been:Line 227 in 8b98b23
As far as I see, Zaza creates a model with the following model-config and doesn't have a way to override it.
zaza/zaza/charm_lifecycle/prepare.py
Lines 28 to 40 in d66f93a
Those values are sane to me, but I was hit by a corner case that a daily image didn't properly come up on a specific provider and wanted to override it to use a known-to-work image with "released" stream instead. It might be a good idea to allow overriding model-configs in some places like tests/tests.yaml.
While calling async_block_until_service_status (through block_until_service_status) with a unit_name that doesn't exist ends up raising an Exception in get_unit_from_name.
As it can be observed in the following trace:
Traceback (most recent call last):
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/zaza/charm_tests/ceph/tests.py", line 96, in test_services
target_status='running'
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/zaza/__init__.py", line 44, in _wrapper
return run(_run_it())
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/zaza/__init__.py", line 32, in run
return task.result()
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/zaza/__init__.py", line 43, in _run_it
return await f(*args, **kwargs)
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/zaza/model.py", line 723, in async_block_until_service_status
await async_block_until(_check_service, timeout=timeout)
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/zaza/model.py", line 791, in async_block_until
await asyncio.wait_for(_block(), timeout, loop=loop)
File "/usr/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
return fut.result()
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/zaza/model.py", line 785, in _block
result = await c()
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/zaza/model.py", line 715, in _check_service
timeout=timeout)
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/zaza/model.py", line 236, in async_run_on_unit
unit = get_unit_from_name(unit_name, model)
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/zaza/model.py", line 111, in get_unit_from_name
raise Exception
Exception
The main problem being that there is no message in the Exception that is raised by get_unit_from_name. It it thus difficult to troubleshoot the issue when it happens.
Several solutions (ordered by preference, last is best):
ExceptionUnitNotFound exception or something similar.After a successfull run of functest-run-suite zaza spews the bellow tracebacks. It may not be closing the event loop properly or it is trying to close it to often.
ok
test_pause_resume (zaza.charm_tests.keystone.tests.CharmOperationTest)
Run pause and resume tests. ... ok
Ran 2 tests in 149.459s
OK
ERROR:asyncio:Fatal write error on socket transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7efe44bf9080>
transport: <_SelectorSocketTransport fd=9>
Traceback (most recent call last):
File "/usr/lib/python3.6/asyncio/selector_events.py", line 765, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
ERROR:asyncio:Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7efe44bf9080>
transport: <_SelectorSocketTransport closing fd=9>
Traceback (most recent call last):
File "/usr/lib/python3.6/asyncio/selector_events.py", line 765, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/asyncio/sslproto.py", line 650, in _process_write_backlog
self._transport.write(chunk)
File "/usr/lib/python3.6/asyncio/selector_events.py", line 769, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File "/usr/lib/python3.6/asyncio/selector_events.py", line 645, in _fatal_error
self._force_close(exc)
File "/usr/lib/python3.6/asyncio/selector_events.py", line 657, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib/python3.6/asyncio/base_events.py", line 575, in call_soon
self._check_closed()
File "/usr/lib/python3.6/asyncio/base_events.py", line 358, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<WebSocketCommonProtocol.transfer_data() running at /home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/websockets/protocol.py:528> w
ait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7efe3ebb5948>()]> cb=[<TaskWakeupMethWrapper object at 0x7efe44c01048>(), _wait.._on_completion()
at /usr/lib/python3.6/asyncio/tasks.py:380]>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<WebSocketCommonProtocol.close_connection() running at /home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/websockets/protocol.py:805
wait_for=<Task pending coro=<WebSocketCommonProtocol.transfer_data() running at /home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/websockets/protocol.py:5
28> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7efe3ebb5948>()]> cb=[<TaskWakeupMethWrapper object at 0x7efe44c01048>(), _wait.._on_complet
ion() at /usr/lib/python3.6/asyncio/tasks.py:380]>>
Exception ignored in: <generator object WebSocketCommonProtocol.close_connection at 0x7efe44bfb360>
Traceback (most recent call last):
File "/home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/websockets/protocol.py", line 855, in close_connection
if (yield from self.wait_for_connection_lost()):
File "/home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/websockets/protocol.py", line 880, in wait_for_connection_lost
self.timeout, loop=self.loop)
File "/usr/lib/python3.6/asyncio/tasks.py", line 342, in wait_for
timeout_handle = loop.call_later(timeout, _release_waiter, waiter)
File "/usr/lib/python3.6/asyncio/base_events.py", line 544, in call_later
timer = self.call_at(self.time() + delay, callback, *args)
File "/usr/lib/python3.6/asyncio/base_events.py", line 554, in call_at
self._check_closed()
File "/usr/lib/python3.6/asyncio/base_events.py", line 358, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<_Task.start..run() running at /home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/juju/client/connection.py:565> wait_for=<F
uture pending cb=[<TaskWakeupMethWrapper object at 0x7efe3fa48a98>()]>>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<_Task.start..run() running at /home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/juju/client/connection.py:565> wait_for=<F
uture pending cb=[<TaskWakeupMethWrapper object at 0x7efe3faafb88>()]>>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<Model._watch.._all_watcher() running at /home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/juju/model.py:808> wait_for=<Fut
ure pending cb=[<TaskWakeupMethWrapper object at 0x7efe3fa14af8>()]>>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<Event.wait() running at /usr/lib/python3.6/asyncio/locks.py:283> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7efe3face9a8>
()]> cb=[_wait.._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<Connection._pinger.._do_ping() done, defined at /home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/juju/client/connection.p
y:248> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7efe3faaf498>()]> cb=[_wait.._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<Event.wait() done, defined at /usr/lib/python3.6/asyncio/locks.py:269> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7efe3fa
ce7c8>()]> cb=[_wait.._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<WebSocketCommonProtocol.recv() done, defined at /home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/websockets/protocol.py:299> wait
_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7efe3ebb5c18>()]> cb=[_wait.._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
Exception ignored in: <generator object Queue.get at 0x7efe3ecae990>
Traceback (most recent call last):
File "/usr/lib/python3.6/asyncio/queues.py", line 169, in get
getter.cancel() # Just in case getter is not done yet.
File "/usr/lib/python3.6/asyncio/base_events.py", line 575, in call_soon
self._check_closed()
File "/usr/lib/python3.6/asyncio/base_events.py", line 358, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<Queue.get() done, defined at /usr/lib/python3.6/asyncio/queues.py:155> wait_for= cb=[_wait.._on_completion() at /usr
/lib/python3.6/asyncio/tasks.py:380]>
Exception ignored in: <coroutine object AllWatcherFacade.Next at 0x7efe3ee6b6d0>
Traceback (most recent call last):
File "/home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/juju/client/facade.py", line 412, in wrapper
reply = await f(*args, **kwargs)
File "/home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/juju/client/_client1.py", line 59, in Next
reply = await self.rpc(msg)
File "/home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/juju/client/overrides.py", line 105, in rpc
result = await self.connection.rpc(msg, encoder=TypeEncoder)
File "/home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/juju/client/connection.py", line 308, in rpc
result = await self._recv(msg['request-id'])
File "/home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/juju/client/connection.py", line 210, in _recv
return await self.messages.get(request_id)
File "/home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/juju/utils.py", line 61, in get
value = await self._queues[id].get()
File "/usr/lib/python3.6/asyncio/queues.py", line 169, in get
getter.cancel() # Just in case getter is not done yet.
File "/usr/lib/python3.6/asyncio/base_events.py", line 575, in call_soon
self._check_closed()
File "/usr/lib/python3.6/asyncio/base_events.py", line 358, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<AllWatcherFacade.Next() done, defined at /home/thedac/dev/run/.tox/clients/lib/python3.6/site-packages/juju/client/facade.py:409> wait_for= cb=[_wait.._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<Event.wait() done, defined at /usr/lib/python3.6/asyncio/locks.py:269> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7efe3fb
639d8>()]> cb=[_wait.._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
The functional test appear to be hit by this upstream Ceph bug: https://tracker.ceph.com/issues/23516
A fix has been merged @ ceph/ceph#22370 but it does not appear to have made it into any released version of Luminous.
I guess we would need a tactical fix in our functional tests for Ceph versions prior to Mimic.
16:00:08 Validate that a volume created through Cinder is mirrored. ... INFO:root:Using keystone API V2 for overcloud auth
16:00:23 INFO:root:creating
16:00:25 INFO:root:creating
16:00:33 INFO:root:downloading
16:00:42 INFO:root:downloading
16:00:58 INFO:root:downloading
16:01:34 INFO:root:available
16:02:43 INFO:root:entries_behind_master=16480
16:02:48 INFO:root:entries_behind_master=16480
16:02:53 INFO:root:entries_behind_master=11006
16:02:57 INFO:root:entries_behind_master=11006
16:03:02 INFO:root:entries_behind_master=11006
16:03:05 INFO:root:entries_behind_master=11006
16:03:09 INFO:root:entries_behind_master=11006
16:03:13 INFO:root:entries_behind_master=11006
16:03:16 INFO:root:entries_behind_master=11006
16:03:19 INFO:root:entries_behind_master=5502
16:03:23 INFO:root:entries_behind_master=5502
16:03:26 INFO:root:entries_behind_master=5502
16:03:30 INFO:root:entries_behind_master=5502
16:03:33 INFO:root:entries_behind_master=5502
16:03:36 INFO:root:entries_behind_master=5502
16:03:39 INFO:root:entries_behind_master=5502
16:03:48 INFO:root:entries_behind_master=14
16:03:52 INFO:root:entries_behind_master=14
[ goes on forever ]
In my local environment I may have Zaza performing tests on one juju controller (i.e. localhost-localhost) and at the same time using my juju environmet to interface with a local or remote maas controller.
If I have a model on a different controller in focus Zaza will bomb out.
Example Traceback:
Deploy of bundle completed.
INFO:root:Waiting for environment to settle
INFO:root:Waiting for a unit to appear
INFO:root:Waiting for all units to be idle
INFO:root:Checking workload status of cinder-ceph/0
INFO:root:Checking workload status message of cinder-ceph/0
INFO:root:Checking workload status of ceph-mon/0
INFO:root:Checking workload status message of ceph-mon/0
INFO:root:Checking workload status of ceph-mon/1
INFO:root:Checking workload status message of ceph-mon/1
INFO:root:Checking workload status of ceph-mon/2
INFO:root:Checking workload status message of ceph-mon/2
INFO:root:Checking workload status of ceph-mon-b/0
INFO:root:Checking workload status message of ceph-mon-b/0
INFO:root:Checking workload status of ceph-mon-b/1
INFO:root:Checking workload status message of ceph-mon-b/1
INFO:root:Checking workload status of ceph-mon-b/2
INFO:root:Checking workload status message of ceph-mon-b/2
INFO:root:Checking workload status of ceph-osd/0
INFO:root:Checking workload status message of ceph-osd/0
INFO:root:Checking workload status of ceph-osd/1
INFO:root:Checking workload status message of ceph-osd/1
INFO:root:Checking workload status of ceph-osd/2
INFO:root:Checking workload status message of ceph-osd/2
INFO:root:Checking workload status of ceph-osd-b/0
INFO:root:Checking workload status message of ceph-osd-b/0
INFO:root:Checking workload status of ceph-osd-b/1
INFO:root:Checking workload status message of ceph-osd-b/1
INFO:root:Checking workload status of ceph-osd-b/2
INFO:root:Checking workload status message of ceph-osd-b/2
INFO:root:Checking workload status of ceph-rbd-mirror/0
INFO:root:Checking workload status message of ceph-rbd-mirror/0
INFO:root:Checking workload status of ceph-rbd-mirror-b/0
INFO:root:Checking workload status message of ceph-rbd-mirror-b/0
INFO:root:Checking workload status of cinder/0
INFO:root:Checking workload status message of cinder/0
INFO:root:Checking workload status of glance/0
INFO:root:Checking workload status message of glance/0
INFO:root:Checking workload status of keystone/0
INFO:root:Checking workload status message of keystone/0
INFO:root:Checking workload status of mysql/0
INFO:root:Checking workload status message of mysql/0
INFO:root:Checking workload status of rabbitmq-server/0
INFO:root:Checking workload status message of rabbitmq-server/0
Traceback (most recent call last):
File "/tmp/tox/func/bin/functest-run-suite", line 10, in <module>
sys.exit(main())
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 130, in main
bundle=args.bundle)
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 62, in func_test_runner
configure.configure(model_name, test_config['configure'])
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/configure.py", line 45, in configure
run_configure_list(functions)
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/configure.py", line 35, in run_configure_list
utils.get_class(func)()
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/charm_tests/glance/setup.py", line 97, in add_lts_image
image_name=image_name)
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/charm_tests/glance/setup.py", line 46, in add_image
keystone_session = openstack_utils.get_overcloud_keystone_session()
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/utilities/openstack.py", line 335, in get_overcloud_keystone_session
return get_keystone_session(get_overcloud_auth(),
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/utilities/openstack.py", line 1401, in get_overcloud_auth
remote_interface_name='certificates')
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/__init__.py", line 44, in _wrapper
return run(_run_it())
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/__init__.py", line 32, in run
return task.result()
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/__init__.py", line 43, in _run_it
return await f(*args, **kwargs)
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/model.py", line 1188, in async_get_relation_id
async with run_in_model(model_name) as model:
File "/tmp/tox/func/lib/python3.6/site-packages/async_generator/_util.py", line 34, in __aenter__
return await self._agen.asend(None)
File "/tmp/tox/func/lib/python3.6/site-packages/async_generator/_impl.py", line 366, in step
return await ANextIter(self._it, start_fn, *args)
File "/tmp/tox/func/lib/python3.6/site-packages/async_generator/_impl.py", line 197, in __next__
return self._invoke(first_fn, *first_args)
File "/tmp/tox/func/lib/python3.6/site-packages/async_generator/_impl.py", line 209, in _invoke
result = fn(*args)
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/model.py", line 153, in run_in_model
await model.connect_model(model_name)
File "/tmp/tox/func/lib/python3.6/site-packages/juju/model.py", line 554, in connect_model
return await self.connect(model_name=model_name)
File "/tmp/tox/func/lib/python3.6/site-packages/juju/model.py", line 511, in connect
await self._connector.connect_model(model_name, **kwargs)
File "/tmp/tox/func/lib/python3.6/site-packages/juju/client/connector.py", line 126, in connect_model
raise JujuConnectionError('Model not found: {}'.format(model_name))
juju.errors.JujuConnectionError: Model not found: admin/zaza-f91698b3b7c3
ERROR: InvocationError: '/tmp/tox/func/bin/functest-run-suite --keep-model'
___________________________________ summary ____________________________________
ERROR: func: commands failed
When doing a full functional test run for a charm it can fail with an unknown model error such as:
https://review.openstack.org/#/c/578955
This appears to be due to zaza caching the model name and not updating the name when the next functional test runs.
I'm trying to run Zaza on AWS provider. And the default VPC will add both private IP and public IP to an instance.
When I run the functest-* commands from inside the VPC which means from the same network space and subnet with deployed applications, Zaza "configure" and "test" tasks try to connect to the applications using a public IP. Unless I explicitly modify the default security group or expose the application with Juju, those tasks will fail because of blocked traffic by security groups created by Juju.
It would be nice to have some flexibility to specify which IP address to be used as private or public, or specifying a network space.
Lines 435 to 446 in d66f93a
At this moment, full connectivity to public IP address is assumed.
zaza-openstack-tests has a similar assumption, but it's a different project.
https://github.com/openstack-charmers/zaza-openstack-tests/blob/ce9db8434bdf8cd03b04c7788ebde3d62bd0d291/zaza/openstack/utilities/openstack.py#L1364-L1374
Hi,
My functional test is failing, and it seems to be zaza overriding the location to the local charm. The charm bundle contains:
jenkins-slave:
charm: /tmp/charm-builds/jenkins-slave
When running 'functest-run-suite --keep-model', it fails with:
2019-06-20 14:54:34 [INFO] panic: charm or bundle URL has invalid form: "../../../jenkins-slave"
2019-06-20 14:54:34 [INFO] goroutine 1 [running]:
It looks to be related to get_charm_config_context(). Any ideas how to work around this? Perhaps if just the charm name is given, then charm_location set to '../../../{charm_name}' but if a full path is given, use that instead.
It would be great if one could write tests that only are run on a subset of the gate/smoke/dev bundles, for example, I might want to validate that a service works correctly with loosing a member in an HA configuration, but killing the single unit of a service will break it.
I could imagine it looking something like:
charm_name: api-example
tests:
- zaza.openstack.charm_tests.api.api_test
- zaza.openstack.charm_tests.api.ha_test:
only: [ ha ]
- zaza.openstack.charm_tests.api.single_unit_lost:
not: [ ha ]
configure: []
gate_bundles:
- ha
- basic
smoke_bundles:
- basicWhile this would, marginally, complicate the parsing of tests, I think that adding optional configuration to the test runs could end up being valuable in a number of scenarios.
For reference, the above YAML gets parsed into a Dict that looks like:
{
'charm_name': 'api-example',
'configure': [],
'gate_bundles': ['ha', 'basic'],
'smoke_bundles': ['basic'],
'tests': ['zaza.openstack.charm_tests.api.api_test',
{'zaza.openstack.charm_tests.api.ha_test': {'only': ['ha']}},
{'zaza.openstack.charm_tests.api.single_unit_lost': {'not': ['ha']}}]}zaza currently ships with all the tests. It would be great if zaza could import the tests from other branches. This would allow other projects to use zaza without having to manage there tests in zaza
Erm, yes, this quite a big one. zaza deploy fails to detect hook failures or unhappy workload statuses.
https://pastebin.canonical.com/p/XBHg2X3D6P/
setting the series to eoan (rather than trying to get it) gets me quite far past this issue
Running functional tests on a multi-cloud controllers fails as below.
It would be useful if Zaza could accept a parameter specifying a cloud to be used when add-model is invoked.
Controller Model User Access Cloud/Region Models Nodes HA Version
canonistack-manual* zaza admin superuser canonistack-manual 3 1 - 2.7-beta1
ERROR this controller manages more than one cloud.
Please specify which cloud/region to use:
juju add-model [options] <model-name> cloud[/region]
The clouds/regions supported by this controller are:
Cloud Regions
canonistack canonistack
canonistack-manual
Traceback (most recent call last):
File "/home/ubuntu/charm-keystone-saml-mellon/src/.tox/func/bin/functest-run-suite", line 10, in <module>
sys.exit(main())
File "/home/ubuntu/charm-keystone-saml-mellon/src/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 140, in main
bundle=args.bundle)
File "/home/ubuntu/charm-keystone-saml-mellon/src/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 59, in func_test_runner
prepare.prepare(model_name)
File "/home/ubuntu/charm-keystone-saml-mellon/src/.tox/func/lib/python3.6/site-packages/zaza/utilities/run_report.py", line 144, in wrapper
result = f(*args, **kwds)
File "/home/ubuntu/charm-keystone-saml-mellon/src/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/prepare.py", line 38, in prepare
config=deployment_env.get_model_settings())
File "/home/ubuntu/charm-keystone-saml-mellon/src/.tox/func/lib/python3.6/site-packages/zaza/__init__.py", line 48, in _wrapper
return run(_run_it())
File "/home/ubuntu/charm-keystone-saml-mellon/src/.tox/func/lib/python3.6/site-packages/zaza/__init__.py", line 36, in run
return task.result()
File "/home/ubuntu/charm-keystone-saml-mellon/src/.tox/func/lib/python3.6/site-packages/zaza/__init__.py", line 47, in _run_it
return await f(*args, **kwargs)
File "/home/ubuntu/charm-keystone-saml-mellon/src/.tox/func/lib/python3.6/site-packages/zaza/controller.py", line 43, in async_add_model
subprocess.check_call(model_cmd)
File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['juju', 'add-model', '--no-switch', '--config', '/tmp/tmp4l4nsdp0.yaml', 'zaza-e6fe7d94fe5c']' returned non-zero exit status 1.
ERROR: InvocationError: '/home/ubuntu/charm-keystone-saml-mellon/src/.tox/func/bin/functest-run-suite --keep-model'
______________________________________________________________________________ summary _______________________________________________________________________________
ERROR: func: commands failed
For this Cosmic-Rocky deployment of nova-cell-controller, the nova-cell-controller unit is Bionic. That is a pretty critical problem:
Model Controller Cloud/Region Version SLA Timestamp
zaza-56c81868dc35 auto-osci-sv06 serverstack/serverstack 2.5-beta1 unsupported 16:28:01Z
App Version Status Scale Charm Store Rev OS Notes
glance 17.0.0 active 1 glance jujucharms 340 ubuntu
keystone 14.0.0 active 1 keystone jujucharms 392 ubuntu
mysql 5.7.20-29.24 active 1 percona-cluster jujucharms 317 ubuntu
mysql-cell2 5.7.20-29.24 active 1 percona-cluster jujucharms 317 ubuntu
neutron-api 13.0.1 active 1 neutron-api jujucharms 368 ubuntu
neutron-gateway 13.0.1 active 1 neutron-gateway jujucharms 344 ubuntu
neutron-openvswitch 13.0.1 active 1 neutron-openvswitch jujucharms 328 ubuntu
nova-cell-controller-cell2 17.0.6 active 1 nova-cell-controller local 0 ubuntu
nova-cloud-controller 18.0.1 active 1 nova-cloud-controller jujucharms 384 ubuntu
nova-compute-cell2 18.0.1 active 1 nova-compute jujucharms 404 ubuntu
rabbitmq-server-neutron 3.6.10 active 1 rabbitmq-server jujucharms 321 ubuntu
rabbitmq-server-nova 3.6.10 active 1 rabbitmq-server jujucharms 321 ubuntu
rabbitmq-server-nova-cell2 3.6.10 active 1 rabbitmq-server jujucharms 321 ubuntu
Unit Workload Agent Machine Public address Ports Message
glance/0* active idle 0 172.17.106.5 9292/tcp Unit is ready
keystone/0* active idle 1 172.17.106.6 5000/tcp Unit is ready
mysql-cell2/0* active idle 3 172.17.106.12 3306/tcp Unit is ready
mysql/0* active idle 2 172.17.106.4 3306/tcp Unit is ready
neutron-api/0* active idle 4 172.17.106.16 9696/tcp Unit is ready
neutron-gateway/0* active idle 5 172.17.106.13 Unit is ready
nova-cell-controller-cell2/0* active idle 6 172.17.106.18 Unit is ready
nova-cloud-controller/0* active idle 7 172.17.106.19 8774/tcp,8775/tcp,8778/tcp Unit is ready
nova-compute-cell2/0* active idle 8 172.17.106.17 Unit is ready
neutron-openvswitch/0* active idle 172.17.106.17 Unit is ready
rabbitmq-server-neutron/0* active idle 9 172.17.106.9 5672/tcp Unit is ready
rabbitmq-server-nova-cell2/0* active idle 11 172.17.106.14 5672/tcp Unit is ready
rabbitmq-server-nova/0* active idle 10 172.17.106.8 5672/tcp Unit is ready
Machine State DNS Inst id Series AZ Message
0 started 172.17.106.5 ed670113-7949-4198-a696-c4b70d40a20f cosmic nova ACTIVE
1 started 172.17.106.6 c1b0a695-445e-4ce5-99a4-35a37540d407 cosmic nova ACTIVE
2 started 172.17.106.4 082e2864-4a02-4e73-a6ae-665e2f347d2f cosmic nova ACTIVE
3 started 172.17.106.12 2ab07015-8fd6-40b7-8b7f-5c4d66d5a812 cosmic nova ACTIVE
4 started 172.17.106.16 19044221-1d10-4240-b754-6369dddef473 cosmic nova ACTIVE
5 started 172.17.106.13 04cfa4b1-10d9-4c8b-977f-04e374027436 cosmic nova ACTIVE
6 started 172.17.106.18 8e1600e6-0d57-448e-8b23-d5801ab113b6 bionic nova ACTIVE
7 started 172.17.106.19 99290dc0-338f-47ca-a7b6-23713826adfe cosmic nova ACTIVE
8 started 172.17.106.17 216ee435-fab6-4809-886d-fdfc1c806bb6 cosmic nova ACTIVE
9 started 172.17.106.9 9ace9975-fe50-4017-853e-d5f21a09b8dc cosmic nova ACTIVE
10 started 172.17.106.8 62a539ee-a0d2-445d-be6d-84b47791ed13 cosmic nova ACTIVE
11 started 172.17.106.14 62405f42-732d-4f02-a75b-781def88e59e cosmic nova ACTIVE
Additional context and artifacts:
The race fixed in #114 introduces a new problem. A part of the fix was to wait for agent to go into executing state before we move on to wait for it to complete and return to a idle state.
Telling Juju to change a charm config to a value that is already set is a noop and returns immediately without any effect on the deployment.
We should either fix consumers of zaza.charm_tests.test_utils.config_change to check before calling or do the check in function and determine whether we should expect and wait for executing state or not.
When running the functional tests for xenial-queens in ceph-osd I received the following errors. The functional tests actually pass.
2019-11-20 01:27:12 [ERROR] Fatal write error on socket transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7ff12160fac8>
transport: <_SelectorSocketTransport fd=10>
Traceback (most recent call last):
File "/usr/lib/python3.6/asyncio/selector_events.py", line 752, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
2019-11-20 01:27:13 [ERROR] Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7ff12160fac8>
transport: <_SelectorSocketTransport closing fd=10>
Traceback (most recent call last):
File "/usr/lib/python3.6/asyncio/selector_events.py", line 752, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/asyncio/sslproto.py", line 650, in _process_write_backlog
self._transport.write(chunk)
File "/usr/lib/python3.6/asyncio/selector_events.py", line 756, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File "/usr/lib/python3.6/asyncio/selector_events.py", line 634, in _fatal_error
self._force_close(exc)
File "/usr/lib/python3.6/asyncio/selector_events.py", line 646, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib/python3.6/asyncio/base_events.py", line 591, in call_soon
self._check_closed()
File "/usr/lib/python3.6/asyncio/base_events.py", line 377, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
2019-11-20 01:27:13 [ERROR] Task was destroyed but it is pending!
task: <Task pending coro=<WebSocketCommonProtocol.transfer_data() running at /home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/websockets/protocol.py:674> wait_for=<Fut
ure pending cb=[<TaskWakeupMethWrapper object at 0x7ff11ae6d378>()]> cb=[<TaskWakeupMethWrapper object at 0x7ff1216183a8>(), _wait.._on_completion() at /usr/lib/python3.6
/asyncio/tasks.py:380]>
2019-11-20 01:27:13 [ERROR] Task was destroyed but it is pending!
task: <Task pending coro=<WebSocketCommonProtocol.keepalive_ping() running at /home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/websockets/protocol.py:977> wait_for=<Fu
ture pending cb=[<TaskWakeupMethWrapper object at 0x7ff11ae6ddf8>()]>>
2019-11-20 01:27:13 [ERROR] Task was destroyed but it is pending!
task: <Task pending coro=<WebSocketCommonProtocol.close_connection() running at /home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/websockets/protocol.py:1019> wait_for=
<Task pending coro=<WebSocketCommonProtocol.transfer_data() running at /home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/websockets/protocol.py:674> wait_for=<Future pe
nding cb=[<TaskWakeupMethWrapper object at 0x7ff11ae6d378>()]> cb=[<TaskWakeupMethWrapper object at 0x7ff1216183a8>(), _wait.._on_completion() at /usr/lib/python3.6/async
io/tasks.py:380]>>
Exception ignored in: <generator object WebSocketCommonProtocol.close_connection at 0x7ff1216193b8>
Traceback (most recent call last):
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/websockets/protocol.py", line 1056, in close_connection
if (yield from self.wait_for_connection_lost()):
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/websockets/protocol.py", line 1080, in wait_for_connection_lost
loop=self.loop,
File "/usr/lib/python3.6/asyncio/tasks.py", line 342, in wait_for
timeout_handle = loop.call_later(timeout, _release_waiter, waiter)
File "/usr/lib/python3.6/asyncio/base_events.py", line 560, in call_later
timer = self.call_at(self.time() + delay, callback, *args)
File "/usr/lib/python3.6/asyncio/base_events.py", line 570, in call_at
self._check_closed()
File "/usr/lib/python3.6/asyncio/base_events.py", line 377, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
2019-11-20 01:27:13 [ERROR] Task was destroyed but it is pending!
task: <Task pending coro=<_Task.start..run() running at /home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/juju/client/connection.py:756> wait_for=<Future pendin
g cb=[<TaskWakeupMethWrapper object at 0x7ff11b8abbb8>()]>>
2019-11-20 01:27:13 [ERROR] Task was destroyed but it is pending!
task: <Task pending coro=<_Task.start..run() running at /home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/juju/client/connection.py:756> wait_for=<Future pendin
g cb=[<TaskWakeupMethWrapper object at 0x7ff11acc3f78>()]>>
2019-11-20 01:27:13 [ERROR] Task was destroyed but it is pending!
task: <Task pending coro=<Model._watch.._all_watcher() running at /home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/juju/model.py:841> wait_for=<Future pending
cb=[<TaskWakeupMethWrapper object at 0x7ff11b9cd888>()]>>
2019-11-20 01:27:13 [ERROR] Task was destroyed but it is pending!
task: <Task pending coro=<Connection._pinger.._do_ping() running at /home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/juju/client/connection.py:390> wait_for=<F
uture pending cb=[<TaskWakeupMethWrapper object at 0x7ff11b04e558>()]> cb=[_wait..on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
2019-11-20 01:27:13 [ERROR] Task was destroyed but it is pending!
task: <Task pending coro=<Event.wait() running at /usr/lib/python3.6/asyncio/locks.py:283> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7ff11b98f0a8>()]> cb=[
wait.._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
2019-11-20 01:27:13 [ERROR] Task was destroyed but it is pending!
task: <Task pending coro=<WebSocketCommonProtocol.recv() done, defined at /home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/websockets/protocol.py:369> wait_for=<Future
pending cb=[<TaskWakeupMethWrapper object at 0x7ff11b98f9a8>()]> cb=[_wait.._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
2019-11-20 01:27:13 [ERROR] Task was destroyed but it is pending!
task: <Task pending coro=<Event.wait() done, defined at /usr/lib/python3.6/asyncio/locks.py:269> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7ff11b9cd918>()]>
cb=[_wait.._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
Exception ignored in: <coroutine object AllWatcherFacade.Next at 0x7ff120b9af68>
Traceback (most recent call last):
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/juju/client/facade.py", line 472, in wrapper
reply = await f(*args, **kwargs)
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/juju/client/_client1.py", line 168, in Next
reply = await self.rpc(msg)
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/juju/client/overrides.py", line 105, in rpc
result = await self.connection.rpc(msg, encoder=TypeEncoder)
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/juju/client/connection.py", line 447, in rpc
result = await self._recv(msg['request-id'])
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/juju/client/connection.py", line 349, in _recv
return await self.messages.get(request_id)
File "/home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/juju/utils.py", line 64, in get
value = await self._queues[id].get()
File "/usr/lib/python3.6/asyncio/queues.py", line 169, in get
getter.cancel() # Just in case getter is not done yet.
File "/usr/lib/python3.6/asyncio/base_events.py", line 591, in call_soon
self._check_closed()
File "/usr/lib/python3.6/asyncio/base_events.py", line 377, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
2019-11-20 01:27:13 [ERROR] Task was destroyed but it is pending!
task: <Task pending coro=<AllWatcherFacade.Next() done, defined at /home/ubuntu/ceph-osd/.tox/func/lib/python3.6/site-packages/juju/client/facade.py:469> wait_for= cb=[_wait.._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
2019-11-20 01:27:13 [ERROR] Task was destroyed but it is pending!
task: <Task pending coro=<Event.wait() done, defined at /usr/lib/python3.6/asyncio/locks.py:269> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7ff11b926c48>()]>
cb=[_wait.._on_completion() at `/usr/lib/python3.6/asyncio/tasks.py:380]>
With debug output added in zaza.model:
diff --git a/zaza/model.py b/zaza/model.py
index 6a89720..0f4854a 100644
--- a/zaza/model.py
+++ b/zaza/model.py
@@ -23,7 +23,9 @@ import asyncio
from async_generator import async_generator, yield_, asynccontextmanager
import logging
import os
+import pprint
import subprocess
+import sys
import tempfile
import yaml
from oslo_config import cfg
@@ -645,6 +647,11 @@ async def async_wait_for_application_states(model_name=None, states=None,
logging.info("Waiting for all units to be idle")
await model.block_until(
lambda: model.all_units_idle(), timeout=timeout)
+ print('MODEL "{}"'.format(model_name))
+ for unit in model.applications['keystone'].units:
+ pprint.pprint(unit.data)
+ output = subprocess.check_output(['juju', 'status', '--color'])
+ sys.stdout.buffer.write(output)
for application in model.applications:
check_info = states.get(application, {})
for unit in model.applications[application].units:
We get this:
INFO:root:Waiting for all units to be idle
MODEL "zaza-58e3bcb7e32b"
{'agent-status': {'current': 'idle',
'message': '',
'since': '2018-08-16T06:52:14.711028126Z',
'version': ''},
'application': 'keystone',
'charm-url': 'local:xenial/keystone-0',
'machine-id': '1',
'model-uuid': '6532f315-e836-4103-840f-d0eca5e7f9dd',
'name': 'keystone/0',
'port-ranges': [{'from-port': 5000, 'protocol': 'tcp', 'to-port': 5000}],
'ports': [{'number': 5000, 'protocol': 'tcp'}],
'private-address': '10.5.0.25',
'public-address': '10.5.0.25',
'series': 'xenial',
'subordinate': False,
'workload-status': {'current': 'active',
'message': 'Unit is ready',
'since': '2018-08-16T05:20:58.047788626Z',
'version': ''}}
{'agent-status': {'current': 'idle',
'message': '',
'since': '2018-08-16T06:52:18.128244918Z',
'version': ''},
'application': 'keystone',
'charm-url': 'local:xenial/keystone-0',
'machine-id': '2',
'model-uuid': '6532f315-e836-4103-840f-d0eca5e7f9dd',
'name': 'keystone/1',
'port-ranges': [{'from-port': 5000, 'protocol': 'tcp', 'to-port': 5000}],
'ports': [{'number': 5000, 'protocol': 'tcp'}],
'private-address': '10.5.0.26',
'public-address': '10.5.0.26',
'series': 'xenial',
'subordinate': False,
'workload-status': {'current': 'active',
'message': 'Unit is ready',
'since': '2018-08-16T05:13:40.637224242Z',
'version': ''}}
{'agent-status': {'current': 'idle',
'message': '',
'since': '2018-08-16T06:52:16.482962212Z',
'version': ''},
'application': 'keystone',
'charm-url': 'local:xenial/keystone-0',
'machine-id': '3',
'model-uuid': '6532f315-e836-4103-840f-d0eca5e7f9dd',
'name': 'keystone/2',
'port-ranges': [{'from-port': 5000, 'protocol': 'tcp', 'to-port': 5000}],
'ports': [{'number': 5000, 'protocol': 'tcp'}],
'private-address': '10.5.0.13',
'public-address': '10.5.0.13',
'series': 'xenial',
'subordinate': False,
'workload-status': {'current': 'active',
'message': 'Unit is ready',
'since': '2018-08-16T05:19:29.000967167Z',
'version': ''}}
Model Controller Cloud/Region Version SLA Timestamp
zaza-58e3bcb7e32b fnordahl-serverstack serverstack/serverstack 2.5-beta1 unsupported 06:52:27Z
App Version Status Scale Charm Store Rev OS Charm version Notes
glance 15.0.1 active 1 glance jujucharms 322 ubuntu
keystone 12.0.0 active 3 keystone local 0 ubuntu
mysql 5.6.37-26.21 active 1 percona-cluster jujucharms 306 ubuntu
Unit Workload Agent Machine Public address Ports Message
glance/0* active idle 0 10.5.0.16 9292/tcp Unit is ready
keystone/0 active executing 1 10.5.0.25 5000/tcp (config-changed) Unit is ready
keystone/1* active executing 2 10.5.0.26 5000/tcp (config-changed) Unit is ready
keystone/2 active executing 3 10.5.0.13 5000/tcp (config-changed) Unit is ready
mysql/0* active idle 4 10.5.0.24 3306/tcp Unit is ready
Machine State DNS Inst id Series AZ Message
0 started 10.5.0.16 903792b7-b024-4f5c-9f29-d423d67baf9d xenial nova ACTIVE
1 started 10.5.0.25 348b31c1-e412-4edb-9ab2-0ffe89b4a402 xenial nova ACTIVE
2 started 10.5.0.26 d1da32fc-1244-4229-891c-4cac1fcc89bb xenial nova ACTIVE
3 started 10.5.0.13 3d892c0f-45bb-4e4a-9b25-24b62883474a xenial nova ACTIVE
4 started 10.5.0.24 79965736-f02a-465d-a1f5-eadb4307549b xenial nova ACTIVE
functest-run-suite terminates with an AssertionError exception and traceback when my test fails, and the model is not torn down. --keep-model is not being used.
2019-09-03 17:44:28 [INFO] Ran 1 test in 3.517s
2019-09-03 17:44:28 [INFO] FAILED
2019-09-03 17:44:28 [INFO] (errors=1)
Traceback (most recent call last):
File "/home/stub/charms/postgresql/.tox/func/bin/functest-run-suite", line 10, in <module>
sys.exit(main())
File "/home/stub/charms/postgresql/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 140, in main
bundle=args.bundle)
File "/home/stub/charms/postgresql/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 76, in func_test_runner
test_steps.get(model_alias, []))
File "/home/stub/charms/postgresql/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/test.py", line 68, in test
run_test_list(tests)
File "/home/stub/charms/postgresql/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/test.py", line 62, in run_test_list
assert test_result.wasSuccessful(), "Test run failed"
AssertionError: Test run failed
ERROR: InvocationError: '/home/stub/charms/postgresql/.tox/func/bin/functest-run-suite --smoke'
_______________________________________________________________________________ summary ________________________________________________________________________________
ERROR: func: commands failed
security.tests.FileOwnershipTest only runs tests if the test model is the current model (i.e. you switched to it with juju switch zaza-xyz)
This is because the generated tests are comparing the configured vs. currently deployed applications, and getting the latter is buggy
Models which are created by Zaza should enable test mode, to prevent artificially ticking metrics in the charm store for charm usage.
The Zaza models should also disable retrying failed hooks, as we expect all charms to never enter a hook error state. The retry failed hooks feature is useful for users, but we do want to bail and fail if a charm ever enters an error state in a hook.
test-mode: true
transmit-vendor-metrics: false
automatically-retry-hooks: false
These, and other useful tuning from OSCI are here for example:
We should probably bomb out immediately if a unit appears in error state?
Sahid found the following bug. When running the keystone setup.add_demo_user. Zaza assumes the charm under test is keystone. However when this is called from a different charm (nova-compute) the following error occurs:
Note: Charm: nova-compute Config: {'preferred-api-version': '3'}
Traceback (most recent call last):
File "/home/ubuntu/dev/func-smoke/bin/functest-configure", line 11, in
sys.exit(main())
File "/home/ubuntu/dev/func-smoke/lib/python3.6/site-packages/zaza/charm_lifecycle/configure.py", line 81, in main
configure(args.model_name, funcs)
File "/home/ubuntu/dev/func-smoke/lib/python3.6/site-packages/zaza/charm_lifecycle/configure.py", line 45, in configure
run_configure_list(functions)
File "/home/ubuntu/dev/func-smoke/lib/python3.6/site-packages/zaza/charm_lifecycle/configure.py", line 35, in run_configure_list
utils.get_class(func)()
File "/home/ubuntu/dev/func-smoke/lib/python3.6/site-packages/zaza/charm_tests/keystone/setup.py", line 34, in change_v3
{'preferred-api-version': '3'}):
File "/usr/lib/python3.6/contextlib.py", line 81, in enter
return next(self.gen)
File "/home/ubuntu/dev/func-smoke/lib/python3.6/site-packages/zaza/charm_tests/test_utils.py", line 102, in config_change
model_name=self.model_name)
File "/home/ubuntu/dev/func-smoke/lib/python3.6/site-packages/zaza/init.py", line 44, in _wrapper
return run(_run_it())
File "/home/ubuntu/dev/func-smoke/lib/python3.6/site-packages/zaza/init.py", line 32, in run
return task.result()
File "/home/ubuntu/dev/func-smoke/lib/python3.6/site-packages/zaza/init.py", line 43, in _run_it
return await f(*args, **kwargs)
File "/home/ubuntu/dev/func-smoke/lib/python3.6/site-packages/zaza/model.py", line 477, in async_set_application_config
.set_config(configuration))
File "/home/ubuntu/dev/func-smoke/lib/python3.6/site-packages/juju/application.py", line 308, in set_config
return await app_facade.Set(self.name, config)
File "/home/ubuntu/dev/func-smoke/lib/python3.6/site-packages/juju/client/facade.py", line 412, in wrapper
reply = await f(*args, **kwargs)
File "/home/ubuntu/dev/func-smoke/lib/python3.6/site-packages/juju/client/_client5.py", line 655, in Set
reply = await self.rpc(msg)
File "/home/ubuntu/dev/func-smoke/lib/python3.6/site-packages/juju/client/facade.py", line 537, in rpc
result = await self.connection.rpc(msg, encoder=TypeEncoder)
File "/home/ubuntu/dev/func-smoke/lib/python3.6/site-packages/juju/client/connection.py", line 316, in rpc
raise errors.JujuAPIError(result)
juju.errors.JujuAPIError: unknown option "preferred-api-version"
Traceback (most recent call last):
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/utilities/openstack.py", line 1271, in get_os_release
index = OPENSTACK_RELEASES_PAIRS.index(release_pair)
ValueError: 'bionic_stein' is not in list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/charm_tests/ceph/tests.py", line 279, in test_ceph_encryption
current_release = zaza_openstack.get_os_release()
File "/tmp/tox/func/lib/python3.6/site-packages/zaza/utilities/openstack.py", line 1277, in get_os_release
raise exceptions.ReleasePairNotFound(msg)
zaza.utilities.exceptions.ReleasePairNotFound: Release pair: bionic_stein not found in ['trusty_icehouse', 'trusty_kilo', 'trusty_liberty', 'trusty_mitaka', 'xenial_mitaka', 'xenial_newton', 'yakkety_newton', 'xenial_ocata', 'zesty_ocata', 'xenial_pike', 'artful_pike', 'xenial_queens', 'bionic_queens', 'bionic_rocky', 'cosmic_rocky', 'bionic-stein', 'disco-stein']
Notice the hyphen (-) vs underscore (_) in the newest additions to the list.
libjuju juju status no longer has units for subordinate charms, block_until_unit_wl_status will fail when the unit is a subordinate unit. See the code:
https://github.com/openstack-charmers/zaza/blob/master/zaza/model.py#L1423
v = model_status.applications[app]['units'][unit_name][ 'workload-status']['status']
2019-12-20 10:19:30 [INFO] ----------------------------------------------------------------------
2019-12-20 10:19:30 [INFO] Traceback (most recent call last):
2019-12-20 10:19:30 [INFO] File "/home/ubuntu/dev/hacluster/.tox/func-smoke/lib/python3.6/site-packages/zaza/openstack/charm_tests/hacluster/tests.py", line 55, in test
2019-12-20 10:19:30 [INFO] zaza.model.block_until_unit_wl_status(subordinate, "maintenance")
2019-12-20 10:19:30 [INFO] File "/home/ubuntu/dev/hacluster/.tox/func-smoke/lib/python3.6/site-packages/zaza/init.py", line 48, in _wrapper
2019-12-20 10:19:30 [INFO] return run(_run_it())
2019-12-20 10:19:30 [INFO] File "/home/ubuntu/dev/hacluster/.tox/func-smoke/lib/python3.6/site-packages/zaza/init.py", line 36, in run
2019-12-20 10:19:30 [INFO] return task.result()
2019-12-20 10:19:30 [INFO] File "/home/ubuntu/dev/hacluster/.tox/func-smoke/lib/python3.6/site-packages/zaza/init.py", line 47, in _run_it
2019-12-20 10:19:30 [INFO] return await f(*args, **kwargs)
2019-12-20 10:19:30 [INFO] File "/home/ubuntu/dev/hacluster/.tox/func-smoke/lib/python3.6/site-packages/zaza/model.py", line 1431, in async_block_until_unit_wl_status
2019-12-20 10:19:30 [INFO] await async_block_until(_unit_status, timeout=timeout)
2019-12-20 10:19:30 [INFO] File "/home/ubuntu/dev/hacluster/.tox/func-smoke/lib/python3.6/site-packages/zaza/model.py", line 1137, in async_block_until
2019-12-20 10:19:30 [INFO] await asyncio.wait_for(_block(), timeout, loop=loop)
2019-12-20 10:19:30 [INFO] File "/usr/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
2019-12-20 10:19:30 [INFO] return fut.result()
2019-12-20 10:19:30 [INFO] File "/home/ubuntu/dev/hacluster/.tox/func-smoke/lib/python3.6/site-packages/zaza/model.py", line 1131, in _block
2019-12-20 10:19:30 [INFO] result = await c()
2019-12-20 10:19:30 [INFO] File "/home/ubuntu/dev/hacluster/.tox/func-smoke/lib/python3.6/site-packages/zaza/model.py", line 1423, in _unit_status
2019-12-20 10:19:30 [INFO] v = model_status.applications[app]['units'][unit_name][
2019-12-20 10:19:30 [INFO] TypeError: 'NoneType' object is not subscriptable
2019-12-20 10:19:30 [INFO] ----------------------------------------------------------------------
Test case is as follow:
`
def test(self):
status = zaza.model.get_status().applications[self.application_name]
if status.get("units") is None and status.get("subordinate-to"):
primary_status = juju_utils.get_application_status(
status.get("subordinate-to")[0])
primary_leader_unit = None
for unit in primary_status["units"]:
if primary_status["units"][unit].get("leader"):
primary_leader_unit = unit
for subordinate in primary_status["units"][primary_leader_unit]["subordinates"]:
zaza.model.run_action(subordinate, "pause")
zaza.model.block_until_unit_wl_status(primary_leader_unit, "blocked")
zaza.model.block_until_unit_wl_status(subordinate, "maintenance")`
Suggest change:
if units are None and 'subordinate-to' has a value, check the primary application status to get the subordinate unit instead.
When using individual CLI tools in a script it would be useful to have Zaza auto-generate model name instead of re-inventing this functionality in the calling script.
For example you could do something like:
set -x
model=$(functest-prepare| awk '/^model_name:/ { print $2; }')
time functest-deploy -m $model -b tests/bundles/bionic-queens.yaml
functest-destroy -m $model
Traceback (most recent call last):
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/charm_tests/keystone/tests.py", line 34, in test_901_pause_resume
self.pause_resume(['apache2'])
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/charm_tests/test_utils.py", line 145, in pause_resume
model_name=self.model_name)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/__init__.py", line 30, in _wrapper
return run(_run_it())
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/__init__.py", line 18, in run
return task.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/__init__.py", line 29, in _run_it
return await f(*args, **kwargs)
File "/home/ubuntu/src/keystone/.tox/func-smoke/lib/python3.5/site-packages/zaza/model.py", line 453, in async_run_action
action_obj = await unit.run_action(action_name, **action_params)
TypeError: run_action() argument after ** must be a mapping, not NoneType
Given the following bundle.yaml and tests.yaml example, we could use the variables defined in the tests.yaml to build our test matrix, reducing the repetition in our charms.
bundle.yaml:
series: *series
applications:
ceph-osd:
charm: ceph-osd
options:
source: *source
ceph-mon:
charm: cs:ceph-mon
options:
source: *source
relations:
- - ceph-mon
- ceph-osdtests.yaml:
charm_name: ceph-osd
base_bundle:
- bundle
# gate_bundles:
# --snip--
# smoke_bundles:
# --snip--
# dev_bundles:
# --snip--
configure: []
tests:
- zaza.openstack.charm_tests.ceph.tests.CephTest
# The below section is entirely optional and will be used as a transform
# on the {gate,smoke,dev}_bundles sections at the beginning. The matrix below
# would transform into, basically:
# gate_bundles:
# - bionic-stein
# - bionic-rocky
# - bionic-queens
# - xenial-queens
# - xenial-pike
# - xenial-ocata
# - xenial-mitaka
# - trusty-mitaka
# smoke_bundles:
# - bionic-queens
# dev_bundles:
# - cosmic-rocky
# - disco-stein
matrix:
series:
- disco:
- source|openstack-origin:
- stein:
- dev: true
- gate: false
- cosmic:
- source|openstack-origin:
- rocky:
- dev: true
- gate: false
- bionic:
- source|openstack-origin:
- stein
- rocky
- queens:
- smoke: true
- xenial:
- source|openstack-origin:
- queens
- pike
- ocata
- mitaka
- trusty:
- source|openstack-origin:
- mitakaAn alternate method for mapping in the matrix section to the *-bundle section could be:
base-bundle:
bundle
# The mode_map key would be used as a replace map to identify
# the pattern to transform the matrix variables into the bundle
# mappings for dev/smoke/gate
mode_map: "*series-*openstack-origin"
gate_bundles:
- bionic-stein
- bionic-rocky
- bionic-queens
smoke_bundles:
- bionic-queens
dev_bundles:
- cosmic-rocky
- disco-stein
matrix:
series:
- disco:
- source|openstack-origin:
- stein
- cosmic:
- source|openstack-origin:
- rocky
- bionic:
- source|openstack-origin:
- stein
- rocky
- queensIn addition to mapping the bundles to be run, the above would additionally add the following section to the bundle.yaml (bundle.yaml in the above config):
variables:
series: &series disco
source: &source stein
openstack-origin: &openstack-origin steinWhere the [disco / stein] combination is an entry generated by the matrix values in tests.yaml.
While this method would increase the length of the tests.yaml (potentially considerably), It would almost entirely remove the repetition mapping each U-OS combination to a new bundle.
It is also possible that the base-bundle section could be extended to allow for mapping different base bundles depending on the variables in the matrix section. I think that this would be valuable when considering bundles that change deployed applications per release, or for restricting some applications to a subset of releases.
After combining these variables in their "base" bundle, we could render the base bundle in addition to overlays as the primary bundle to deploy.
I'm using Zaza for testing kubernetes charms, and I realized that if a k8s pod is in an error state, the units of the juju applications are active, but the application isn't:
$ kubectl -n testmodel get pods
NAME READY STATUS RESTARTS AGE
zookeeper-k8s-0 0/1 CrashLoopBackOff 2 60s
zookeeper-k8s-1 0/1 CrashLoopBackOff 2 60s
zookeeper-k8s-operator-0 1/1 Running 0 84s
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
testmodel k8s-cloud microk8s/localhost 2.6.5 unsupported 10:53:53Z
App Version Status Scale Charm Store Rev OS Address Notes
zookeeper-k8s waiting 1 zookeeper-k8s local 0 kubernetes 10.152.183.57
Unit Workload Agent Address Ports Message
zookeeper-k8s/0* active idle 10.1.1.63 2181/TCP,2888/TCP,3888/TCP ready
zookeeper-k8s/1* active idle 10.1.1.64 2181/TCP,2888/TCP,3888/TCP ready
In the zaza.model, is it checked the workload_status_message of the units, but not the status of the application.
Receiving a KeyError: 'parameters' when raise_on_failure=True for zaza.model.run_action(), where the action itself fails.
2019-12-10 17:06:01 [INFO] Traceback (most recent call last):
2019-12-10 17:06:01 [INFO] File "./tests/functional/utils.py", line 27, in wrapped_f
2019-12-10 17:06:01 [INFO] f(*args)
2019-12-10 17:06:01 [INFO] File "./tests/functional/test_duplicity.py", line 193, in test_sftp_full_do_backup
2019-12-10 17:06:01 [INFO] zaza.model.run_action(self.duplicity_unit.name, self.action, raise_on_failure=True)
2019-12-10 17:06:01 [INFO] File "/home/user/Projects/Charms/charm-duplicity/.tox/func-noop/lib/python3.7/site-packages/zaza/__init__.py", line 48, in _wrapper
2019-12-10 17:06:01 [INFO] return run(_run_it())
2019-12-10 17:06:01 [INFO] File "/home/user/Projects/Charms/charm-duplicity/.tox/func-noop/lib/python3.7/site-packages/zaza/__init__.py", line 36, in run
2019-12-10 17:06:01 [INFO] return task.result()
2019-12-10 17:06:01 [INFO] File "/home/user/Projects/Charms/charm-duplicity/.tox/func-noop/lib/python3.7/site-packages/zaza/__init__.py", line 47, in _run_it
2019-12-10 17:06:01 [INFO] return await f(*args, **kwargs)
2019-12-10 17:06:01 [INFO] File "/home/zackz/Projects/Charms/charm-duplicity/.tox/func-noop/lib/python3.7/site-packages/zaza/model.py", line 589, in async_run_action
2019-12-10 17:06:01 [INFO] raise ActionFailed(action_obj)
2019-12-10 17:06:01 [INFO] File "/home/zackz/Projects/Charms/charm-duplicity/.tox/func-noop/lib/python3.7/site-packages/zaza/model.py", line 557, in __init__
2019-12-10 17:06:01 [INFO] .format(action.name, action.parameters, action.receiver,
2019-12-10 17:06:01 [INFO] File "/home/zackz/Projects/Charms/charm-duplicity/.tox/func-noop/lib/python3.7/site-packages/juju/model.py", line 266, in __getattr__
2019-12-10 17:06:01 [INFO] return self.safe_data[name]
2019-12-10 17:06:01 [INFO] KeyError: 'parameters'
2019-12-10 17:06:01 [INFO] ----------------------------------------------------------------------
It looks like the code as an issue when retrieving action.parameters where parameters does not exist in action dict:
message = ('Run of action "{}" with parameters "{}" on "{}" failed '
'with "{}" (id={} status={} enqueued={} started={} '
'completed={})'
.format(action.name, action.parameters, action.receiver,
action.message, action.id, action.status,
action.enqueued, action.started, action.completed))
Example: clone the percona-cluster charm, try to run zaza func tests with tox -e func and witness:
(func-noop) 1 ubuntu@beisner-bastion:~/git/charm-percona-cluster$ functest-run-suite --keep-model --smoke
Traceback (most recent call last):
File "/home/ubuntu/git/charm-percona-cluster/.tox/func-noop/bin/functest-run-suite", line 10, in <module>
sys.exit(main())
File "/home/ubuntu/git/charm-percona-cluster/.tox/func-noop/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 162, in main
bundle=args.bundle)
File "/home/ubuntu/git/charm-percona-cluster/.tox/func-noop/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 100, in func_test_runner
environment_deploys = utils.get_environment_deploys(bundle_key)
File "/home/ubuntu/git/charm-percona-cluster/.tox/func-noop/lib/python3.6/site-packages/zaza/charm_lifecycle/utils.py", line 211, in get_environment_deploys
environment_deploys.append(get_environment_deploy(bundle_mapping))
File "/home/ubuntu/git/charm-percona-cluster/.tox/func-noop/lib/python3.6/site-packages/zaza/charm_lifecycle/utils.py", line 124, in get_environment_deploy
return env_deploy_f[get_deployment_type(deployment_directive)](
KeyError: None
It would be useful if zaza looked for a local config file and loaded config from it rather than relying on environment variables. The config file could also store secrets that should not be exposed in logs. My current thinking is something like:
~/.zaza.yaml
provider_secrets:
OS_AUTH_URL: http://100.10.10.10:5000/v3
OS_USERNAME: bob
OS_PASSWORD: password
OS_USER_DOMAIN_NAME: user
OS_PROJECT_NAME: bobproj
OS_PROJECT_ID: a5cb900
OS_PROJECT_DOMAIN_NAME: user
OS_AUTH_VERSION: 3
OS_IDENTITY_API_VERSION: 3
OS_REGION_NAME: serverstack
runtime_config:
OS_VAR: 10.100.12.3
http_proxy: http://squid
secrets:
NAS_PASSWORD: hardpassword
Hello,
I have only hit this once, so I do not expect it to hit often, but do we need to catch errors like these and silently retry?
The controller in question happily lives on and is serving other models with Zaza right now.
Traceback (most recent call last):
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/bin/functest-run-suite", line 8, in <module>
sys.exit(main())
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 173, in main
bundle=args.bundle)
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 118, in func_test_runner
run_env_deployment(env_deployment, keep_model=preserve_model)
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 68, in run_env_deployment
config_steps.get(deployment.model_alias, []))
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/zaza/charm_lifecycle/configure.py", line 48, in configure
run_configure_list(functions)
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/zaza/charm_lifecycle/configure.py", line 37, in run_configure_list
utils.get_class(func)()
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/zaza/openstack/charm_tests/nova/setup.py", line 53, in manage_ssh_key
keystone_session = openstack_utils.get_overcloud_keystone_session()
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/zaza/openstack/utilities/openstack.py", line 384, in get_overcloud_keystone_session
get_overcloud_auth(model_name=model_name),
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/zaza/openstack/utilities/openstack.py", line 1634, in get_overcloud_auth
model_name=model_name)
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/zaza/openstack/utilities/openstack.py", line 1505, in get_application_config_option
model_name=model_name)
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/zaza/__init__.py", line 48, in _wrapper
return run(_run_it())
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/zaza/__init__.py", line 36, in run
return task.result()
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/zaza/__init__.py", line 47, in _run_it
return await f(*args, **kwargs)
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/zaza/model.py", line 513, in async_get_application_config
return await model.applications[application_name].get_config()
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/juju/application.py", line 244, in get_config
return (await app_facade.Get(application=self.name)).config
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/juju/client/facade.py", line 471, in wrapper
reply = await f(*args, **kwargs)
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/juju/client/_client8.py", line 957, in Get
reply = await self.rpc(msg)
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/juju/client/facade.py", line 607, in rpc
result = await self.connection.rpc(msg, encoder=TypeEncoder)
File "/home/ubuntu/src/charm-octavia/build/builds/octavia/.tox/func-smoke/lib/python3.6/site-packages/juju/client/connection.py", line 456, in rpc
raise errors.JujuAPIError(result)
juju.errors.JujuAPIError: getting state: getting storage provider registry: authentication failed.: authentication failed
caused by: requesting token: failed executing the request http://10.245.161.156:5000/v3/auth/tokens
caused by: Post http://10.245.161.156:5000/v3/auth/tokens: EOF
This may be a python-libjuju issue, but adding it as a issue here for tracking.
2019-08-28 08:22:11 [INFO] Using keystone API V3 (or later) for overcloud auth
Traceback (most recent call last):
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/bin/functest-run-suite", line 10, in <module>
sys.exit(main())
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 140, in main
bundle=args.bundle)
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 71, in func_test_runner
config_steps.get(model_alias, []))
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/configure.py", line 48, in configure
run_configure_list(functions)
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/zaza/charm_lifecycle/configure.py", line 37, in run_configure_list
utils.get_class(func)()
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/zaza/openstack/charm_tests/dragent/configure.py", line 88, in setup
network.setup_sdn(network_config, keystone_session=keystone_session)
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/zaza/openstack/configure/network.py", line 114, in setup_sdn
if openstack_utils.get_keystone_api_version() > 2:
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/zaza/openstack/utilities/openstack.py", line 1404, in get_keystone_api_version
'preferred-api-version')
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/zaza/openstack/utilities/openstack.py", line 1310, in get_application_config_option
application_config = model.get_application_config(application)
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/zaza/__init__.py", line 48, in _wrapper
return run(_run_it())
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/zaza/__init__.py", line 36, in run
return task.result()
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/zaza/__init__.py", line 47, in _run_it
return await f(*args, **kwargs)
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/zaza/model.py", line 469, in async_get_application_config
async with run_in_model(model_name) as model:
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/async_generator/_util.py", line 34, in __aenter__
return await self._agen.asend(None)
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/async_generator/_impl.py", line 366, in step
return await ANextIter(self._it, start_fn, *args)
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/async_generator/_impl.py", line 197, in __next__
return self._invoke(first_fn, *first_args)
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/async_generator/_impl.py", line 209, in _invoke
result = fn(*args)
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/zaza/model.py", line 155, in run_in_model
await model.connect_model(model_name)
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/juju/model.py", line 573, in connect_model
return await self.connect(model_name=model_name)
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/juju/model.py", line 530, in connect
await self._connector.connect_model(model_name, **kwargs)
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/juju/client/connector.py", line 143, in connect_model
bakery_client=self.bakery_client_for_controller(controller_name),
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/juju/client/connector.py", line 158, in bakery_client_for_controller
controller_name)
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/juju/client/jujudata.py", line 218, in cookies_for_controller
jar.load()
File "/usr/lib/python3.6/http/cookiejar.py", line 1784, in load
self._really_load(f, filename, ignore_discard, ignore_expires)
File "/home/ubuntu/src/charm-neutron-dynamic-routing/build/builds/neutron-dynamic-routing/.tox/func/lib/python3.6/site-packages/juju/client/gocookies.py", line 16, in _really_load
data = json.load(f) or []
File "/usr/lib/python3.6/json/__init__.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Tests will fail when these environment variables aren't present but Zaza will happily allow undefined variables in bundle templates
Long model names break OpenStack's DNS.
With the generated model name, zaza-neutron-dynamic-routingxenial-queens-functional20180419131500, DNS names were getting shortened which cased duplicate records to exist:
$ hostname
juju-4b888b-zaza-neutron-dynamic-routingxenial-queens-functiona
$ dig juju-4b888b-zaza-neutron-dynamic-routingxenial-queens-functiona.serverstack.
; <<>> DiG 9.10.3-P4-Ubuntu <<>> juju-4b888b-zaza-neutron-dynamic-routingxenial-queens-functiona.serverstack.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44713
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;juju-4b888b-zaza-neutron-dynamic-routingxenial-queens-functiona.serverstack. IN A
;; ANSWER SECTION:
juju-4b888b-zaza-neutron-dynamic-routingxenial-queens-functiona.serverstack. 0 IN A 10.5.0.22
juju-4b888b-zaza-neutron-dynamic-routingxenial-queens-functiona.serverstack. 0 IN A 10.5.0.17
juju-4b888b-zaza-neutron-dynamic-routingxenial-queens-functiona.serverstack. 0 IN A 10.5.0.29
juju-4b888b-zaza-neutron-dynamic-routingxenial-queens-functiona.serverstack. 0 IN A 10.5.0.9
juju-4b888b-zaza-neutron-dynamic-routingxenial-queens-functiona.serverstack. 0 IN A 10.5.0.28
juju-4b888b-zaza-neutron-dynamic-routingxenial-queens-functiona.serverstack. 0 IN A 10.5.0.23
zaza.charm_lifecycle.func_test_runner.generate_model_name generates unnecessarily long model names with zaza-, the charm name, the bundle name and the timestamp. These could be abbreviated to zaza-$TIMESTAMP which would remain unique.
2019-10-02 06:30:19 [WARNING] RPC: Connection closed, reconnecting
2019-10-02 06:30:19 [WARNING] Receiver: Connection closed, reconnecting
2019-10-02 06:30:29 [ERROR] Task exception was never retrieved
future: <Task finished coro=<Connection.reconnect() done, defined at /tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/juju/client/connection.py:563> exception=OSError(113, "Connect call failed ('172.17.112.25', 17070)")>
Traceback (most recent call last):
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/juju/client/connection.py", line 571, in reconnect
await self._connect_with_login([(self.endpoint, self.cacert)])
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/juju/client/connection.py", line 631, in _connect_with_login
await self._connect(endpoints)
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/juju/client/connection.py", line 591, in _connect
result = await task
File "/usr/lib/python3.5/asyncio/tasks.py", line 492, in _wait_for_one
return f.result() # May raise f.exception().
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/juju/client/connection.py", line 580, in _try_endpoint
return await self._open(endpoint, cacert)
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/juju/client/connection.py", line 334, in _open
max_size=self.max_frame_size,
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/websockets/py35/client.py", line 12, in __await_impl__
transport, protocol = await self._creating_connection
File "/usr/lib/python3.5/asyncio/base_events.py", line 695, in create_connection
raise exceptions[0]
File "/usr/lib/python3.5/asyncio/base_events.py", line 682, in create_connection
yield from self.sock_connect(sock, address)
File "/usr/lib/python3.5/asyncio/selector_events.py", line 402, in sock_connect
return (yield from fut)
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/selector_events.py", line 432, in _sock_connect_cb
raise OSError(err, 'Connect call failed %s' % (address,))
OSError: [Errno 113] Connect call failed ('172.17.112.25', 17070)
2019-10-02 06:30:29 [ERROR] RPC: Automatic reconnect failed
Traceback (most recent call last):
File "/tmp/tmp.Afq1xXiq7A/func/bin/functest-run-suite", line 10, in <module>
sys.exit(main())
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 162, in main
bundle=args.bundle)
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 107, in func_test_runner
run_env_deployment(env_deployment, keep_model=preserve_model)
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 53, in run_env_deployment
model_ctxt=model_aliases)
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/zaza/charm_lifecycle/deploy.py", line 294, in deploy
test_config.get('target_deploy_status', {}))
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/zaza/__init__.py", line 48, in _wrapper
return run(_run_it())
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/zaza/__init__.py", line 36, in run
return task.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/zaza/__init__.py", line 47, in _run_it
return await f(*args, **kwargs)
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/zaza/model.py", line 855, in async_wait_for_application_states
timeout=timeout)
File "/tmp/tmp.Afq1xXiq7A/func/lib/python3.5/site-packages/juju/model.py", line 713, in block_until
raise websockets.ConnectionClosed(1006, 'no reason')
websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), reason = no reason
Full CI artifact set: https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-api/684348/5/3934/index.html
The test_all_clients_authenticated test in vault/tests.py uses a hard-coded sleep time, wrapped by a for loop to retry.
It would be better to use tenacity retries to be consistent with the established approach to similar needs in other tests.
https://github.com/openstack-charmers/zaza-openstack-tests/blob/2244f131a058159a8329385b00d0157655328769/zaza/openstack/charm_tests/vault/tests.py#L131-L139
I feel the same about the is_initialized method in vault/utils.py:
https://github.com/openstack-charmers/zaza-openstack-tests/blob/2244f131a058159a8329385b00d0157655328769/zaza/openstack/charm_tests/vault/utils.py#L129-L141
And, same for mysql/tests.py:
https://github.com/openstack-charmers/zaza-openstack-tests/blob/2244f131a058159a8329385b00d0157655328769/zaza/openstack/charm_tests/mysql/tests.py#L380-L392
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.