usmqe / usmqe-tests Goto Github PK

https://github.com/Tendrl/usmqe-tests/blob/master/usmqe/api/tendrlapi/cephapi.py#L21
should be return super().import_cluster(nodes, "ceph", asserts_in)

The goal is to create test for basic API features described for example in documentation mentioned in this pull request Tendrl/api#65
Especially https://github.com/Tendrl/documentation/blob/master/api/overview.adoc

We are reusing pytest config module for usmqe test configuration, which creates few problems:

• it's not possible to load configuration in fixture parametrization code
• the configuration structure is flat
• current integration is error prone (eg. the defaults in pytest ini doesn't work)

So for example, there is no reasonable way I could configure list of tendrl packages or configuration of system repositories.

Proposal and tasks

Create new usmqe pytest plugin for configuration, which:

• doesn't use pytest config as the current one
• properly separated
• is usable in fixtures, markers and parametrization code
• can push values to other pytest config or command line options (so that value can be passed into pytest-ansible module, avoiding specification of ansible inventory file twice)
• takes command line options for:
  • file path of usmqe config file (usmqe ini)
  • ad hoc override of particular config value
• single conf. file uses yaml syntax
• mrg log demo works without reconfiguration (when setting up the qe server using the docs)

Update function flows after Tendrl/api#351 is resolved.

Goal is to implement importing of Ceph cluster from UI. Infrastructure required for this test is shared with #9

Monitoring data related pull request Tendrl/specifications#95

Documentation is not fully up to date with latest changes, we need to fix that:

• qe_server_setup.rst
• test_configuration.rst
• test_development.rst
• test_execution.rst

It would be good, if our RPM validation tests will support locally created repo.

usm_rpm_baseurl = file:/home/usmqe/tendrl/

Above mentioned url is completely valid as baseurl for yum repository, but our tests fails with following error:

_____________ ERROR at setup of test_rpmlint[gstatus] _________________________

    @pytest.fixture(scope="module")
    def rpm_repo():
        """
        Check if we can connect to the repo. If not, this issue will be immediately
        reported during setup so that the test case will end up in ERROR state
        (instead of FAILED if we were checking this during test itself).
        """
        baseurl = pytest.config.getini("usm_rpm_baseurl")
>       reg = requests.get(baseurl)

usmqe_tests/rpm/conftest.py:111: 
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../.local/lib/python3.5/site-packages/requests/api.py:70: in get
    return request('get', url, params=params, **kwargs)
../.local/lib/python3.5/site-packages/requests/api.py:56: in request
    return session.request(method=method, url=url, **kwargs)
../.local/lib/python3.5/site-packages/requests/sessions.py:488: in request
    resp = self.send(prep, **send_kwargs)
../.local/lib/python3.5/site-packages/requests/sessions.py:603: in send
    adapter = self.get_adapter(url=request.url)
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <requests.sessions.Session object at 0x7fa70049a7f0>, url = 'file:/home/usmqe/tendrl/'

    def get_adapter(self, url):
        """
            Returns the appropriate connection adapter for the given URL.
    
            :rtype: requests.adapters.BaseAdapter
            """
        for (prefix, adapter) in self.adapters.items():
    
            if url.lower().startswith(prefix):
                return adapter
    
        # Nothing matches :-/
>       raise InvalidSchema("No connection adapters were found for '%s'" % url)
E       requests.exceptions.InvalidSchema: No connection adapters were found for 'file:/home/usmqe/tendrl/'

../.local/lib/python3.5/site-packages/requests/sessions.py:685: InvalidSchema

see Tendrl/specifications#124

TEN-229
As a Tendrl user, I want to create 1 or more Ceph pools and be able to specify the PGs and journal configuration for it.

Assumptions:

Ceph cluster exists.

Acceptance Criteria:

• 1 or more pool(s) is created with desired PG and desired journal configuration in cluster user specifies.
• Pool(s) created outside Tendrl do not interfere or fail to be discovered by Tendrl
• Space usage calculations are updated immediately following the operation
• A task will be generated when user submits the task to create Ceph pool(s).
• All events and transactions (including user actions) will be logged by the System.
• Pool(s) is immediately accessible upon successful pool creation.

Tendrl/specifications#49

During review of @ltrilety's updates to pytest mrglog integration, I come up with a simple set of dummy test cases to check how the logging setup behaves. Would it be a good idea to include something like that into usmqe tests so that we can check this next time we tweak pytest/logging setup?

The dummy tests I used were very basic, we should consider adding few more scenarios if we decide to include such examples into the repo:

$ cat usmqe_tests/examples/test_foo.py
# -*- coding: utf8 -*-

import pytest


@pytest.fixture
def fixture_error():
    raise Exception


@pytest.fixture(params=[1,2])
def parametrized_fixture(request):
    return [request.param]


def test_pass():
    pytest.check(True)


@pytest.mark.parametrize("x", [1,2,3])
@pytest.mark.parametrize("y", ["a","b"])
def test_pass_parametrized(x, y):
    pytest.check(len(x*y) == x)
    pytest.check(y in x*y)


def test_pass_parametrized_fixture(parametrized_fixture):
    pytest.check(len(parametrized_fixture) > 0)


def test_fail_check():
    pytest.check(False)


def test_fail_exception():
    raise Exception


def test_error(fixture_error):
    pytest.check(True)

And the test run report like this:

collected 12 items 

usmqe_tests/examples/test_foo.py::test_pass PASSED
usmqe_tests/examples/test_foo.py::test_pass_parametrized[a-1] PASSED
usmqe_tests/examples/test_foo.py::test_pass_parametrized[a-2] PASSED
usmqe_tests/examples/test_foo.py::test_pass_parametrized[a-3] PASSED
usmqe_tests/examples/test_foo.py::test_pass_parametrized[b-1] PASSED
usmqe_tests/examples/test_foo.py::test_pass_parametrized[b-2] PASSED
usmqe_tests/examples/test_foo.py::test_pass_parametrized[b-3] PASSED
usmqe_tests/examples/test_foo.py::test_pass_parametrized_fixture[1] PASSED
usmqe_tests/examples/test_foo.py::test_pass_parametrized_fixture[2] PASSED
usmqe_tests/examples/test_foo.py::test_fail_check FAILED
usmqe_tests/examples/test_foo.py::test_fail_exception FAILED
usmqe_tests/examples/test_foo.py::test_error ERROR

Goals:

1. create infrastructure to support this test
2. create ceph cluster import test

remove workaround for Tendrl/api#251 after resolved.

Audit the code for hidden workarounds, which doesn't have it's own "remove workaround" issue.

see Tendrl/specifications#93

remove workaround for Tendrl/api#106 after resolved.

see Tendrl/specifications#83

Workaround introduced in #112

Issue #89 needs to be fixed first.

There's a usm_brick_path option in usm.ini, which says where the brick is located on all gluster nodes. In other words there has to be the same path for a brick on all those nodes. Moreover as such we could use just one brick on a node.
We should use some different configuration which says more than one brick for a node and it will be specific for the node, path doesn't have to be the same on all nodes.

Introduce new fixtures:

valid_new_normal_user
valid_normal_user_data
valid_new_admin_user
valid_admin_user_data

see #124

Description of the problem

In pytest framework, test is not executed and ends up in error state when it's fixture fails (eg. exception is raised during execution of fixture). The problem is that mrglog has no idea about this (or most other pytest concepts), can't express this state and for this reason, test log generated by mrglog is misleading.

Example and reproducer

When test_create_volume_valid (or any other test case that uses valid_cluster_id() fixture) is executed while no cluster is imported, it ends up in error state as explained above. But in test summary reported by mrglog there are 3 passes and 0 errors. These passes are from api call to GetClusterList (checking response format, etc). Thanks to this the test is marked as passed in Jenkins which shows mrglog output. There should be 1 error in test summary and test should be marked as Error. It would be nice to see that test failed during calling a fixture.

$ py.test -svvvrx usmqe_tests/api/tendrl/ -k test_create_volume_valid

[10:12:32,225] Test run ID   : main_test
[10:12:32,225] Package       : unknown
[10:12:32,225] Test started  : 10:12:32,225729
[10:12:32,233] Hostname      : localhost.localdomain
[10:12:32,233] Architecture  : x86_64
[10:12:32,233] Distro:       : CentOS Linux release 7.2.1511 (Core)
[10:12:32,233] :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
============================================================================================= test session starts ==============================================================================================
platform linux -- Python 3.5.1, pytest-3.0.4, py-1.4.31, pluggy-0.4.0 -- /opt/rh/rh-python35/root/usr/bin/python3
cachedir: .cache
rootdir: /home/usmqe/usmqe-tests, inifile: pytest.ini
plugins: interactive-0.1.1
collected 22 items 

usmqe_tests/api/tendrl/gluster/test_volume.py::test_create_volume_valid [10:12:32,408] :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
[10:12:32,408] Test create volume valid started
[10:12:32,569] [ DEBUG   ] api_base:: request.url:  http://fbalak-usm1-server.usmqe.lab.eng.brq.redhat.com:9292/1.0/GetClusterList
[10:12:32,569] [ DEBUG   ] api_base:: request.method:  GET
[10:12:32,570] [ DEBUG   ] api_base:: request.body:  None
[10:12:32,570] [ DEBUG   ] api_base:: request.headers:  {'Accept': '*/*', 'User-Agent': 'python-requests/2.12.1', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive'}
[10:12:32,570] [ DEBUG   ] api_base:: response.cookies: <RequestsCookieJar[]>
[10:12:32,570] [ DEBUG   ] api_base:: response.headers: {'Access-Control-Allow-Methods': 'POST,GET,OPTIONS,PUT,DELETE', 'Content-Length': '15', 'Content-Type': 'application/json', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'Origin,Content-Type,Accept,Authorization,X-Requested-With', 'X-Content-Type-Options': 'nosniff'}
[10:12:32,570] [ DEBUG   ] api_base:: response.ok:      True
[10:12:32,570] [ DEBUG   ] api_base:: response.reason:  OK
[10:12:32,571] [ DEBUG   ] api_base:: response.status:  200
[10:12:32,595] [ DEBUG   ] api_base:: response.text:    {"clusters":[]}
[10:12:32,664] [ PASS    ] main_test:: usmqe/api/base.py:66: AssumptionPassed
	There should be ok == True
[10:12:32,669] [ PASS    ] main_test:: usmqe/api/base.py:68: AssumptionPassed
	Status code should equal to 200
[10:12:32,674] [ PASS    ] main_test:: usmqe/api/base.py:70: AssumptionPassed
	Reason should equal to OK
ERROR[10:12:32,690] Test create volume valid duration  : 0s
[10:12:32,690] Test create volume valid assertions: 3 good, 0 bad
[10:12:32,690] Test create volume valid result    : PASS
[10:12:32,690] :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
[10:12:32,691] :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
[10:12:32,691] =======================================================================================
[10:12:32,691] =======================================================================================
[10:12:32,691] Test-case[s] Summary: (1 found)
[10:12:32,691] ---------------------------------------------------------------------------------------
[10:12:32,692] PASS create volume valid                     :  #tests:3     #fails:0     desc.:""
[10:12:32,692] =======================================================================================
[10:12:32,692] List of known issues:
[10:12:32,692] ---------------------------------------------------------------------------------------
[10:12:32,692] =======================================================================================
[10:12:32,692] =======================================================================================
[10:12:32,692] Test-Cases Summary   #TOTAL: 1     #PASSED: 1     #FAILED: 0     #ERRORS: 0
[10:12:32,693] Test Summary : ERROR #TOTAl: 3     #PASSED: 3     #FAILED: 0     #ERRORS: 0   #WAIVES: 0
[10:12:32,693] Test name    : /home/usmqe/usmqe-tests
[10:12:32,693] Duration     : 0:00:00.465425
[10:12:32,693] Test on      : CentOS Linux release 7.2.1511 (Core) x86_64
[10:12:32,693] =======================================================================================
[10:12:32,694] =======================================================================================
[10:12:32,694] Test finished: 10:12:32,691154


==================================================================================================== ERRORS ====================================================================================================
__________________________________________________________________________________ ERROR at setup of test_create_volume_valid __________________________________________________________________________________

    @pytest.fixture
    def valid_cluster_id():
        """
        Generate valid id of imported cluster.
        """
        # TODO change
        api = glusterapi.TendrlApiGluster()
>       return api.get_cluster_list()[0]["cluster_id"]
E       IndexError: list index out of range

usmqe_tests/api/tendrl/gluster/conftest.py:14: IndexError
============================================================================================= 21 tests deselected ==============================================================================================
==================================================================================== 21 deselected, 1 error in 0.44 seconds ====================================================================================

Task status has to be checked in API

We need to update our rpm validation tests based on this upstream change in copr, as Tim wrote on tendrl-devel:

All the tendrl dependency packages are now moved from "tendrl" copr repo
to "dependencies" repo.
url: https://copr.fedorainfracloud.org/coprs/tendrl/dependencies/

Please use the following url to enable the repository:
https://copr.fedorainfracloud.org/coprs/tendrl/dependencies/repo/epel-7/tendrl-dependencies-epel-7.repo

When Tendrl/api#159 is fixed remove sleep after the cluster is finished.
Related commit 4bf87b8

We need to check that packages from Tendrl dependencies repo doesn't override packages from any other system or storage repo we depend on.

1. Prepare webstr framework
2. Create environment to be able to use webstr
3. Install tendrl GUI
4. Create log in test

This function is based on legacy functionality that is not supported in current version.

Since ping api calls no longer requires authentication, we need to change test cases for api authentication accordingly.

Based on information from anivargi:

[4/19/17 09:55] <mbukatov> is authentication required for ping api call?
[4/19/17 10:57] <anivargi> @mbukatov it should not be needed
[4/19/17 10:57] <anivargi> it is used to see if the server is up
[4/19/17 10:59] <mbukatov> anivargi: ok, thanks for checking

TEN-214
As a Tendrl user, I want to view the status of the actions/tasks I perform within the System to know if my actions/tasks completed successfully, failed, or require additional action on my part.

AR:
Adhere's to design

• Should have a status pane with recent jobs unique id, description, and their status which allows a user to go and look in logs for detailed status information by filtering on the unique ID
• The statuses should be initiated, in progress, completed, failed
• The jobs should be ordered with most recent jobs at the top of the status pane
• Jobs would include provisioning tasks, cluster creation tasks, and any other data or object management task that console is capable of performing

Related spec task-details-view.adoc:
Tendrl/specifications#125

see Tendrl/specifications#105

see Tendrl/specifications#75 and Tendrl/specifications#108

Reconfigure sphinx project in docs directory to include docstring based documentation of usmqe module. This may require an update of unit test/tox configuration.

Go through the docs and change test related terminology to conform istqb standards, add a glossary with links to the definition.

Right now, we use an ad-hoc review process, but it would be nice to have it described in the documentation, along with mandatory reviewers of particular subsets of the code.

There should be test for API chnages mentioned in Tendrl/specifications#73

This issue was created to make sure that temporary change #7 will be properly discussed later on.

Note that verbose_lvl is option of mrglog, and 0 means that the log will contain everything.

The log levels are documented in mrglog_demo.py:

• verbose not set or 0: prints everything
• verbose set to 1: prints only something interesting
• verbose set to 2: prints only log messages

Now wait_for function waits till job is in "finished" of "failed" state. There is timeout fo about 1 hour.

I think that it should be changed to have timeout since last task change. Last update attribute can be checked or changes in list of messages.

Remove usage of "tendrl.org" domain from test data, especially when email alerting is enabled.

We should use example.com by default.

see Tendrl/specifications#77

Create test for alerts, see Tendrl/specifications#94

see Tendrl/specifications#82

see Tendrl/specifications#85

Unmanage test sometimes fails with following error:

[04:39:16,687] [ FAIL     ] pytests_test:: usmqe_tests/api/gluster/test_gluster_cluster.py:218: AssumptionFailure
	graphite health of cluster 46d2e4c3-2a9f-4e04-8d21-560668337579: []
        There should be related data.

for example job: tendrl-3-ci-usm2-test-gluster-distrep-api-import/97

Goal is to create test for importing Gluster cluster from UI. Infrastructure is shared with API test #6

Based on review of #95:

Test cases in usmqe_tests/api/gluster/test_volume.py are incomplete, we should also check that the deleted volume is no longer present in the gui.

Class names for etcdapi, tendrlapi and graphiteapi are called ApiCommon. This should be changed to improve readability and prevent import errors.

TEN-231
As an Tendrl admin I want users to be able to authenticate either "locally" via securely hashed or encrypted local credentials inherent to the Tendrl system

Acceptance Criteria:

• Users must be able to authenticate via the pre-established credentials setup by an admin
• Users must be able to change their password via UI and an underlying API call
• If possible the mechanism should leverage an existing password storage scheme that offers password strength policies
• If LDAP authentication has been configured and explicitly chosen it will take precedence over local accounts, therefore the authentication service must have a notion of precedence via config

Related tendrl issue:
Tendrl/specifications#128

Test test_cluster_unmanage_valid is failing with following error:

[11:21:02,573] [ ERROR    ] main_test:: UnboundLocalError: local variable 'unmanaged_cluster' referenced before assignment

The problem seems to be in file test_gluster_cluster.py around line 228:

228     for cluster in tendrl_api.get_cluster_list():
229         if cluster["cluster_id"] == cluster_id:
230             unmanaged_cluster = cluster
231             break
232     pytest.check(
233         unmanaged_cluster["is_managed"] == "no",
234         "is_managed: {}\nThere should be ``no``.".format(unmanaged_cluster["is_managed"]))

Unmanaged cluster seems to dispensary for some time and tendrl_api.get_cluster_list() returns empty list, so unmanaged_cluster variable is not defined.

TEN-214
As a Tendrl user, I want to view the status of the actions/tasks I perform within the System to know if my actions/tasks completed successfully, failed, or require additional action on my part.

AR:
Adhere's to design

Should have a status pane with recent jobs unique id, description, and their status which allows a user to go and look in logs for detailed status information by filtering on the unique ID
The statuses should be initiated, in progress, completed, failed
The jobs should be ordered with most recent jobs at the top of the status pane
Jobs would include provisioning tasks, cluster creation tasks, and any other data or object management task that console is capable of performing

Related spec task-details-view.adoc:
see Tendrl/specifications#125