We're using the logging module in a few places, and in general it would be helpful to have more debug logging for some of our more intricate tests. We should add log handlers to pick up those log messages and put them somewhere useful.

There are some good snippets about this in this stacko post and replies, though some tweaks would be needed for our environment:
http://stackoverflow.com/questions/4673373/logging-within-py-test-tests

Many data files, like PXE Provisioning kickstart files and customization templates, have a standard format that could be saved in a .template file but also include links to live servers, passwords and other sensitive data.

It's becoming increasingly useful to have a utility that could use .template files and incorporate entries from cfme_data.yaml to build real, usable data files.

Further discussions about the idea were captured here (closed PR):
#101

As a test creator, I want to put data into cfme_data.yaml, but be able to mark certain fields as "random". When looked up, it should return a random value of a given type (string, int, uuid, int with bounds, etc).

For distributed testing, we'll need to be able to change an appliance's region. This is traditionally done via the "black console", which wraps a rake task:
"rake evm:db:region -- --region (region_number)". Additionally, appliances will soon start shipping with databases disabled; enabling the DB may also be a rake task.

We've already got SSH functionality via paramiko, so this can probably a pretty thin wrapper around that, targeted at running rake tasks.

This would be particularly useful for distributed test cases (multi-region, multi-appliance per zone, etc), but could also be used anywhere else a stunt VM is needed in a test case. Ideally, the fixture would return a list of objects that can be used to generate base_urls for mozwebqa, ssh clients, etc, for use in related tests.

I'm currently targetting provisioning using the SOAP interface, but we may have to fall back on going through the UI if we can't find good documentation on the SOAP provisioning methods.

Things that we need to make distributed testing happen
(work in progress, add things as needed til it's done)

• #64: provisioning fixture
• #66: dynamic selenium targeting
• #67: be able to run rake tasks

Currently, we can aim selenium at one appliance and run all of our tests against it. This is useful for most of our test cases, but when we're setting up and testing distributed appliance cases we'll need to be able to interact with multiple appliances in a single test run (e.g. configuring an appliances to replicate its database for a multi-region setup, configuring specific server roles to ensure failover is working, etc).

As a test developer, I want the db fixture to support join tables (or any other table for that matter) that do not have a primary key.

We need a doc that goes into a little more detail on the project layout, coding/testing standards, etc. Preliminary work on this has already been done, but it needs to be fleshed out and dropped into the github wiki.

This test was deleted from CFME_Pages, by the commit below, because it is a destructive test. It should go in CFME_Tests instead.

tehsmyers/cfme_pages@0b1bbf6

With the infra/cloud split and addition of the RHOS provider, there are a few things that we should do to normalize the exposed provider interfaces.

• Create a RHOSSystem in cfme_pages/common/mgmt_system.py
• Inform provider_factory of the new provider type
• Change the mgmt_sys_api_clients fixture to use provider_factory

It would seem #74 broke support for specifying the cfme_data.yaml file on the command line

(default)[dajohnso@localhost cfme_tests]$ py.test tests/ui/services/test_vm_power_control.py -k on --pdb --collectonly --credentials=/home/dajohnso/workspace_cfme/cfme-qe-yamls/default/credentials.yaml --untrusted --cfmedata=/home/dajohnso/workspace_cfme/cfme-qe-yamls/default/cfme_data.yaml
No cfme_db_url defined
================================================================================== test session starts ==================================================================================
platform linux2 -- Python 2.7.3 -- pytest-2.3.4
plugins: mozwebqa
collected 0 items / 2 errors

======================================================================================== ERRORS =========================================================================================
______________________________________________________________ ERROR collecting tests/ui/services/test_vm_power_control.py ______________________________________________________________
tests/ui/services/test_vm_power_control.py:35: in pytest_generate_tests

  data = load_cfme_data()

utils/cfme_data.py:39: in load_cfme_data
raise Exception(msg)
E Exception: Usable to load cfme_data file at /home/dajohnso/workspace_cfme/cfme_tests/cfme_data.yaml
______________________________________________________________ ERROR collecting tests/ui/services/test_vm_power_control.py ______________________________________________________________
tests/ui/services/test_vm_power_control.py:35: in pytest_generate_tests
data = load_cfme_data()
utils/cfme_data.py:39: in load_cfme_data
raise Exception(msg)
E Exception: Usable to load cfme_data file at /home/dajohnso/workspace_cfme/cfme_tests/cfme_data.yaml

Default roles are defined in multiple places at the moment:

• https://github.com/RedHatQE/cfme_tests/blob/master/fixtures/server_roles.py#L8
• https://github.com/RedHatQE/cfme_tests/blob/master/cfme_data.yaml.template#L111

Furthermore, the default roles in the cfme_data template aren't actually the default roles on a freshly made appliance, as the ems_metrics (cap&u) roles aren't on by default.

I propose replacing the default roles in the cfme_data template with the ones listed in fixtures.server_roles.

If necessary, we can populate the default_roles list in fixtures.server roles from the cfme_data template to keep it backward compatible.

The "setup_infrastructure_providers" fixture succeeds but sometimes edit_host_and_save is called before the host is actually discovered.

This test was deleted from CFME_Pages, by the commit below, because it is a destructive test. It should go in CFME_Tests instead.

tehsmyers/cfme_pages@3631bbd

Some information in the docstring for server_roles fixture is out of date (or just incorrect).

As a test developer, I want to use the setup_infrastructure_providers fixture, but I want it to be implemented via the db magic.

pages tests can benefit from the navigation fixtures, and navigation from page to page itself seems like the sort of thing that should be abstracted in the pages repo.

@bcrochet Is already doing this, but I thought I'd be nice and make an issue for it.

When powered on, power on should not be listed, when powered off, power off or shutdown guest should not be listed

Getting cloud provisioning fixtures working will probably go a long way to closing this guy: https://github.com/RedHatQE/cfme_pages/issues/114

#71 probably has some good meat in it for testing C&U collection, but the mechanism for getting at the DB data is a little too rigid for our current needs. We should revisit it at some point.

The ssh_client fixture supports running a rake command. Some of these commands presuppose a local file exists on the appliance, such as when importing an Automate custom namespace. Extending ssh_client would be ideal.

Here's my os.system hack:

import os
...
os.system("sshpass -p '%s' scp %s@%s:/root/" % \
            (mozwebqa.credentials['ssh']['password'],      
             mozwebqa.credentials['ssh']['username'], 
             soap_client.evm_server_hostname))

In tests/ui/infrastructure/test_infrastructure_providers.py, TestInfrastructureProviders.test_provider_add_with_bad_credentials[rhevm32] fails in this way:

first = u'401 Unauthorized', second = 'Cannot complete login due to an incorrect user name or password.', msg = 'Flash message did not match expected value'

    @classmethod
    def equal(self, first, second, msg=None):
        """
            Asserts that 2 elements are the same

            :Args:
             - First object to be tested
             - Second object to be tested
             - Message that will be printed if it fails
            """
>       assert first == second, msg
E       AssertionError: Flash message did not match expected value

The expected message of '401 Unauthorized' seems wrong, so the test just needs a quick update to look for the new (and correct) flash message.

I think there was a BZ opened about the '401 Unauthorized' message, so some spelunking in ought to be done to find and verify that.

This test was deleted from CFME_Pages, by the commit below, because it is a destructive test. It should go in CFME_Tests instead.

tehsmyers/cfme_pages@4583c1d

This test was deleted from CFME_Pages, by the commit below, because it is a destructive test. It should go in CFME_Tests instead.

tehsmyers/cfme_pages@62f11f6

This also had the following issues opened in CFME_Pages:

https://github.com/RedHatQE/cfme_pages/issues/150
https://github.com/RedHatQE/cfme_pages/issues/149

This test was deleted from CFME_Pages, by the commit below, because it is a destructive test. It should go in CFME_Tests instead.

tehsmyers/cfme_pages@d2de438

Is there any benefit to requiring unittestzero as a dependency?
unittestzero syntax: Assert.true(ms_pg.is_the_current_page)
native python syntax: assert ms_pg.is_the_current_page

This test was deleted from CFME_Pages, by the commit below, because it is a destructive test. It should go in CFME_Tests instead.

tehsmyers/cfme_pages@a9f75db

py.test adds test directories seemingly inconsistently based on invocation, e.g. py.test -k test will result in a different sys.path than py.test tests/. This is causing the 'from common.mgmt_system' import in tests/conftest.py to fail. To ensure this import works every time, the cleanest solution is probably to move mgmt_system related fixtures into the fixtures dir, where they'll be autoloaded.

More detail can be found in the guts of this bug report:
https://bitbucket.org/hpk42/pytest/issue/82/syspath-is-extended-with-every-dir