testproject-io / python-opensdk Goto Github PK

Using a Generic driver: from src.testproject.sdk.drivers.webdriver import Generic
and setting the test func to: driver.report().test(name="Test", passed=True, message="Check")

The test result will be "Skipped" in the platform instead of "passed"

For passed=False the test result is "Failed" as expected

Expected Behavior

• The step description should be Type sometext in #my_id
• The value should not be array

Current Behavior

The step description is

And it's value is

Possible Solution

If I change the line from

driver.find_element(By.CSS_SELECTOR, "#my_id").send_keys("sometext")

To

driver.find_element(By.CSS_SELECTOR, "#my_id").send_keys("some text")

Then the value is written in the description:

For some reason, it must be 2 words for it to work.

Steps to Reproduce

Try to run the code:

from selenium.webdriver.common.by import By
from src.testproject.sdk.drivers.webdriver import Chrome


def test_driver_auto_command_report_find_element_strategies():
    driver = Chrome(projectname="python sdk project",
                    jobname="driver_auto_command_report_find_element_strategies",
                    disable_reports=False)

    def driver_draw_html_code(html_code):
        driver.execute_script("document.write('" + html_code.replace("'", r"\'").replace("\n", "") + "')")

    driver_draw_html_code("<input class='my_class' id='my_id'>")

    # CSS Selector
    driver.find_element(By.CSS_SELECTOR, "#my_id").send_keys("sometext")

    driver.quit()

    print("Done")

Context (Environment)

Windows 10
Agent 0.63.1
Latest SDK version

Detailed Description

The first issue is that the value must be 2 words in order that the step will contain the value in the description.
The second issue is that for some reason the value is an array.

Does the OpenSDK provide a way to test RESTful API (e.g. by utilizing the RESTful API Client Addon)?

The examples in the README do not address this kind of tests.

The following test:

@pytest.fixture
def driver():
    driver = webdriver.Chrome()
    yield driver
    driver.quit()


def test_update_profile_expect_success_message_to_be_displayed(driver):

    LoginPage(driver).open().login_as("John Smith", "12345")
    ProfilePage(driver).update_profile(
        country="Australia", address="Main Street 123", email="[email protected]", phone="+1987654321",
    )
    assert ProfilePage(driver).saved_message_is_displayed()

fails with version 0.63.19 of the SDK, because the capabilities sent to the Agent will be empty.

My code:

features/environment.py:

from src.testproject.sdk.drivers import webdriver
from src.testproject.decorator.behave_reporter import behave_reporter


@behave_reporter()
def before_scenario(context, scenario):
    context.driver = webdriver.Chrome(project_name="Python")


@behave_reporter(screenshot=True)
def after_step(context, step):
    pass


@behave_reporter()
def after_scenario(context, scenario):
    context.driver.quit()

features/steps/step_definitions.py:

from behave import *


@given(u'I navigate to TestProject')
def step_imp(context):
    context.driver.get("http://example.testproject.io/")


@when(u'I should perform a login')
def step_impl(context):
    context.driver.report().step("Step Report (description): " + "hello",
                                 "Step Report (message): " + "message", True,True)


@then(u'I should see a logout button')
def step_impl(context):
    context.driver.report().step("Step Report (description): " + "hello2",
                                 "Step Report (message): " + "message", True, True)

features/web_test.feature:

Feature: TestProject Behave Test
  Scenario: Test1
    Given I navigate to TestProject
    When I should perform a login
    Then I should see a logout button

  Scenario: Test2
    Given I navigate to TestProject
    When I should perform a login
    Then I should see a logout button

Issue
If I run the following code, I get this issue in report:

Expected Result
It should take the job name from here

Workaround
In features/environment.py add job_name=context.feature.name. For example:

@behave_reporter()
def before_scenario(context, scenario):
    context.driver = webdriver.Chrome(project_name="Python",
                                      job_name=context.feature.name  # <-- Added
                                      )

After the workaround fix the job name will be TestProject Behave Test as expected.

When running tests with testing frameworks and working with assertions, passed assertions aren't included in the test reports, instead, the tests are marked as skipped.

I think it would be a great feature to have the passed assertions reported as well, as passed steps in the reports.

For example, while running the following tests:

import pytest
import requests
import json
from src.testproject.decorator import report_assertion_errors
from src.testproject.sdk.drivers import webdriver

@pytest.fixture(scope="session", autouse=True)
def driver():
    driver = webdriver.Chrome(projectname = "PyTest Reports", token = "TOKEN")
    yield driver
    driver.quit()

def get_response_status_code():
    response = requests.get("https://www.metaweather.com/api/location/search/?query=San Francisco")
    return response.status_code

def get_response_body():
    response = requests.get("https://www.metaweather.com/api/location/search/?query=San Francisco")
    return json.dumps(response.json())

@report_assertion_errors
def test_response_status_code():
    # Should be equal to 200.
    assert get_response_status_code() == 200

@report_assertion_errors
def test_response_location_type():
    # Should contain 'City' in response.
    assert ('City' in get_response_body())

As the assertions pass, the only information in the report is both test names marked as skipped:

I'm working on WSL2 with the TestProject agent running on Windows so that it can connect to devices attached via USB. I configure requests to the agent to go to the virtual network interface that Windows uses for WSL2 by setting TP_AGENT_URL to e.g. http://172.19.160.1:8585 and then forward them to the agent on the Windows network (with matching firewall rules).

However, since the agent is running on 127.0.0.1, the response object returned by _request_session_from_agent includes the localhost address, e.g. 'http://127.0.0.1:56426/wd/hub' and this is used for constructing the client's agent session, which breaks the connection.

I can get the driver to run tests correctly by editing the address used to construct the SessionResponse object in _request_session_from_agent to use the value for TP_AGENT_URL (without the port) for the server_address parameter.

I haven't looked at the SDK code in detail so wanted to check how this is expected to work before creating a PR that e.g. uses TP_AGENT_URL, if set, to construct the SessionResponse object.

Given that you have the following setup:

features/web_test.feature:

Feature: TestProject Behave Test
  Scenario: Test1
    Given I navigate to TestProject
    When I should perform a login
    Then I should see a logout button

  Scenario: Test2
    Given I navigate to TestProject
    When I should perform a login
    Then I should see a logout button

features/environment.py:

from src.testproject.sdk.drivers import webdriver
from src.testproject.decorator.behave_reporter import behave_reporter


@behave_reporter()
def before_scenario(context, scenario):
    context.driver = webdriver.Chrome()


@behave_reporter(screenshot=False)
def after_step(context, step):
    pass


@behave_reporter()
def after_scenario(context, scenario):
    context.driver.quit()

features/steps/step_definitions.py

from behave import *


@given(u'I navigate to TestProject')
def step_imp(context):
    context.driver.get("http://example.testproject.io/")


@when(u'I should perform a login')
def step_impl(context):
    context.driver.report().step("Step Report (description): " + "hello",
                                 "Step Report (message): " + "message", True, True)


@then(u'I should see a logout button')
def step_impl(context):
    context.driver.report().step("Step Report (description): " + "hello2",
                                 "Step Report (message): " + "message", True, True)

Issue
When running this setup, I get one job with one test in reports.

Changing in features/environment.py:

@behave_reporter()
def before_scenario(context, scenario):
    context.driver = webdriver.Chrome()

To:

@behave_reporter()
def before_scenario(context, scenario):
    context.driver = webdriver.Chrome(job_name="Custom Job Name")

Will fix the issue

Given that you have this code:

 features/steps/step_definitions.py

from behave import *


@given(u'Given Header')
def step_imp(context):
    context.driver.get("http://example.testproject.io/")
    context.driver.report().step("GIVEN STEP", True, True)


@step(u'But Header')
def step_imp(context):
    context.driver.report().step("BUT STEP", True, True)


@step(u'And Header')
def step_imp(context):
    context.driver.report().step("AND STEP", True, True)


@when(u'When Header')
def step_impl(context):
    context.driver.report().step("WHEN STEP", True, True)


@then(u'Then Header')
def step_impl(context):
    context.driver.report().step("THEN STEP", True, True)

 
features/web_test.feature

Feature: TestProject Behave Test
  Scenario: Test1
    Given Given Header
    And And Header
    But But Header
    When When Header
    Then Then Header {code}

environment.py

from src.testproject.sdk.drivers import webdriver
from src.testproject.decorator.behave_reporter import behave_reporter


@behave_reporter()
def before_scenario(context, scenario):
    context.driver = webdriver.Chrome(
                                      job_name=context.feature.name  
                                      )


@behave_reporter(screenshot=False)
def after_step(context, step):
    pass


@behave_reporter()
def after_scenario(context, scenario):
    context.driver.quit()

The report you get is this:

The expected result is this:

• Given Given Header
• GIVEN STEP
• And And Header
• AND STEP
• But But Header
• BUT STEP
• When When header
• WHEN STEP
• Then Then Header
• THEN STEP

The issue here is that in the current result, the step is always before the header.

Running an addon that doesn't return anything in the result's 'fields' will cause a TypeError.

return AddonExecutionResponse(execution_result_type=(ExecutionResultType.Passed
                                                             if operation_result.data["resultType"] == "Passed"
                                                             else ExecutionResultType.Failed),
                                      message=operation_result.data["message"],
>                                     fields=[ResultField(**field) for field in operation_result.data['fields']])

E TypeError: 'NoneType' object is not iterablevenv\lib\site-packages\src\testproject\sdk\internal\agent\agent_client.py:432: TypeError

Consider this code:

from addons.visible_elements_operations import VisibleElementsOperations
from selenium.webdriver.common.by import By
from src.testproject.classes import DriverStepSettings, StepSettings
from src.testproject.decorator import report_assertion_errors
from src.testproject.enums import SleepTimingType
from src.testproject.sdk.drivers import webdriver
import pytest

@pytest.fixture()
def driver():
    driver = webdriver.Chrome(token="",
                              project_name="My first Project",
                              job_name="isContainsTextIfVisible")
    step_settings = StepSettings(timeout=15000,
                                 sleep_time=500,
                                 sleep_timing_type=SleepTimingType.Before)
    with DriverStepSettings(driver, step_settings):
        yield driver
    driver.quit()
@report_assertion_errors
def test_main(driver):
    # Test Parameters
    # Auto generated application URL parameter
    ApplicationURL = "https://google.com"
    # 1. Navigate to '{ApplicationURL}'
    # Navigates the specified URL (Auto-generated)
    driver.get(f'{ApplicationURL}')
    # 2. Click 'q'
    q = driver.find_element(By.CSS_SELECTOR,
                            "[name='q']")
    q.click()
    # 3. Type 'Hello' in 'q'
    q = driver.find_element(By.CSS_SELECTOR,
                            "[name='q']")
    q.send_keys("Hello")
    # 4. 'q' contains text '[NONE]'?
    q = (By.XPATH, "//div[2]/input")
    driver.addons().execute(
        VisibleElementsOperations.containstextifvisibleweb(
            text="Hello",
            timeout="10"), *q)

Fails at

    q = (By.XPATH, "//div[2]/input")
    driver.addons().execute(
        VisibleElementsOperations.containstextifvisibleweb(
            text="Hello",
            timeout="10"), *q)

With the following error:

Changing q = (By.XPATH, "//div[2]/input") to work with CSS Selector instead of XPATH works fine.

Expectation:
Based on the SDK documentation: Disable all reports I expected that by using these API calls:

browser.report().disable_reports(True)
<some browser action here>   
browser.report().disable_reports(False)

All browser actions between the .disable_reports() statements should be excluded from reporting

Actual:
It appears that the failing of wait.until(expected_conditions.invisibility_of_element_located()) is not being excluded from the reporting.

Execution context:

• Windows: CLI command "systeminfo" output

OS Name:                   Microsoft Windows 10 Pro
OS Version:                10.0.19042 N/A Build 19042
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Workstation
OS Build Type:             Multiprocessor Free

• Python: python --version

Python 3.8.5

• TestProject Agent version = 0.64 running installed on Windows

Reproduce

• Using pipenv build up the Python env (pytest, selenium, testproject-python-sdk )

Sample code:

import os
import pytest

from src.testproject.sdk.drivers import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec


@pytest.fixture
def browser():
    os.environ['TP_DEV_TOKEN'] = '<insert_your_token_here>'
    driver = webdriver.Chrome()
    yield driver
    driver.quit()


@pytest.fixture()
def wait(browser):
    wait = WebDriverWait(browser, 2)
    yield wait


def test_disable_report(browser, wait):
    browser.get('http://www.google.com')
    search_field = (By.CSS_SELECTOR, "input[name='z']")  # => z should be q, thus simulating a non-present element

    browser.report().disable_reports(True)
    element_not_present = wait.until(ec.invisibility_of_element_located(search_field))  # still included in report
    assert element_not_present is True  # excluded from report
    browser.report().disable_reports(False)

• hint: might be related to issue 50

Running this test after the latest changes in the Python SDK:

import os
import pytest
import time

from src.testproject.sdk.drivers import webdriver


def test_example_on_native_android_app():
    desired_capabilities = {
        "appActivity": "host.exp.exponent.MainActivity",
        "appPackage": "com.example.tp",
        "udid": "11",
        "platformName": "Android",
        "unicodeKeyboard": "true",
        "resetKeyboard": "true"
    }

    driver = webdriver.Remote(desired_capabilities=desired_capabilities)
    time.sleep(5)
    driver.close_app()
    driver.quit()

if __name__ == "__main__":
    test_example_on_native_android_app()

If the SDK fails to find the connected device (Gave an incorrect UDID by choice) , it returns Browser Not found error.

src.testproject.sdk.exceptions.missingbrowserexception.MissingBrowserException: Requested browser could not be found on your system

I'm currently working with testproject-io which requires Appium-Python-Client==1.0.1 and it seems this is not available currently on the repo at following location https://pypi.org/project/Appium-Python-Client/#files

Is testproject-io compatible with Appium-Python-Client==1.0.2(latest version) ??
Can you help solving this issue.

Expectation:
Based on the SDK documentation: Explicit project and job names I expected the @report decorator to communicate the provided values to the testproject.io "Reports" section.

Actual:
It appears that the @report decorator does not work when (at least) the "project" and "job" arguments are provided.

Execution context:

• Windows: CLI command "systeminfo" output

OS Name:                   Microsoft Windows 10 Pro
OS Version:                10.0.19042 N/A Build 19042
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Workstation
OS Build Type:             Multiprocessor Free

• Python: python --version

Python 3.8.5

• TestProject Agent version = 0.64 running installed on Windows

Reproduce

• Using pipenv build up the Python env (pytest, selenium, testproject-python-sdk )

Sample code:

import os
import pytest

from src.testproject.sdk.drivers import webdriver
from src.testproject.decorator import report


@pytest.fixture
def browser():
    os.environ['TP_DEV_TOKEN'] = '<insert_your_token_here>'
    driver = webdriver.Chrome()
    yield driver
    driver.quit()


@report(project='Python SDK project', job='Python job')
def test_report_decorator(browser):
    browser.get('http://www.google.com')
    assert 'Google' in browser.title

Current Behavior

When I run the test suite with multiple tests that before each test I recreate the driver, and after each test, I quit the driver,
The report is reported as 2 jobs (in case there are 2 tests for example).

This is true for unittest and pytest.

Unittest

import unittest

from src.testproject.sdk.drivers import webdriver


class UntitledTestCase(unittest.TestCase):

    def setUp(self):
        self.driver = webdriver.Chrome(jobname="Job with multiple tests")

    def tearDown(self):
        self.driver.quit()

    def test_untitled_test_case1(self):
        self.driver.get("http://google.com")


    def test_untitled_test_case2(self):
        self.driver.get("http://yahoo.com")


if __name__ == "__main__":
    unittest.main()

pytest

@pytest.fixture
def driver():
    # Will be executed before each test
    driver = Chrome(projectname="python sdk project",
                    jobname="before_after_each",
                    disable_reports=False)

    yield driver

    # Will be executed after each test
    driver.quit()


def test_some_test_case_1(driver):
    driver.get("http://google.com/")


def test_some_test_case_2(driver):
    driver.get("http://yahoo.com/")


if __name__ == "__main__":
    pytest.main()

When running one of these codes, the report I get is 2 jobs - first job have only the first test and the second job have the second test.

Wanted Behavior

I want that the code will be reported as one job with 2 tests (in these cases).

@basdijkstra suggested that we will have something like driver.report().start_job("job name") and driver.report().end_job()
So the idea is to call driver.report().start_job("job name") in code that execute only at the very beginning, and call to driver.report().end_job() in code that executes only at the final end of the execution.

Advantages

Currently, without the wanted behavior, the automation developer is forced to keep using the same driver (to be reported as one job) and this is against a very common best practice approach to make the tests independent on each other by NOT using the same driver. The current behavior is encouraging to reuse the same driver and it is considered as a bad practice.

By making this suggestion, we will allow the developer to write code that follows this good practice without having it reported in the wrong way.

@basdijkstra - Please take a look at this PR --> https://github.com/testproject-io/python-sdk/runs/1585216515?check_suite_focus=true

The problem comes from here:
https://github.com/testproject-io/python-sdk/blob/c17894162e28ca584d3912aa5f9d460ec53120ed/setup.py#L26

It fails to build because 'Appium-Python-Client' updated to v1.0.2 which requires a newer version of Selenium.

You personally made the change at --> b522217
Is there a particular reason ? can we revert it?

While creating my library is Robot Framework using the TestProject SDK, I have encounter a 'phantom' test called run_cli which is being reported in my Tests while using my Custom library.

This is probably coming from the Robot framework it self.

According to our QA @gileli121, the SDK currently ignores invert result.

Seems it ignores all the newly added StepSettings object.

Hello, we found an issue with the android examples

For some android devices, these test failed because the keyboard is opened and hiding the elements that the test interact with.

The solution we found is to add to the driver capabilities the following:

"unicodeKeyboard": "true",
"resetKeyboard": "true" 

This will disable the keyboard on the device.

Reference: https://discuss.appium.io/t/can-we-hide-android-soft-keyboard/6956/3
See comment from Telmo_Cardoso.

We created patch for the issue:
21290_0001-Fix-Android-examples-issues-with-keyboard.patch.txt

from addons.swipe_and_find_element import SwipeAndFindElement
from appium.webdriver.common.mobileby import MobileBy
from src.testproject.classes import DriverStepSettings, StepSettings
from src.testproject.decorator import report_assertion_errors
from src.testproject.enums import SleepTimingType
from src.testproject.sdk.drivers import webdriver
import pytest



@pytest.fixture()
def driver():
    capabilities = {
        "platformName": "iOS",
        "udid": "",
        "bundleId": "com.apple.Preferences",
    }
    driver = webdriver.Remote(token="",
                              project_name="",
                              job_name="iPhone Settings swipe",
                              desired_capabilities=capabilities)
    step_settings = StepSettings(timeout=15000,
                                 sleep_time=500,
                                 sleep_timing_type=SleepTimingType.Before)
    with DriverStepSettings(driver, step_settings):
        yield driver
    driver.quit()


@report_assertion_errors
def test_main(driver):

    # 1. Swipe 'Down' at Most '5' Times Until Element is Displayed
    accessibility = (MobileBy.CSS_SELECTOR,
                     "type == 'XCUIElementTypeCell' AND label == 'Accessibility' AND visible == 1")
    driver.addons().execute(
        SwipeAndFindElement.swipeuntilelementiosvertical(
            direction="Down",
            amountOfSwipes=5), *accessibility)

    # 2. Click 'Accessibility'
    accessibility = driver.find_element(MobileBy.NAME,
                                        "type == 'XCUIElementTypeCell' AND label == 'Accessibility' AND visible == 1")
    accessibility.click()

results in:

FAILED test_iphone_settings_swipe.py::test_main - src.testproject.sdk.exceptions.sdkexception.SdkException: Did not recognize locator strategy accessibility id

While using 0.63.18.dev14 with the example test

from src.testproject.sdk.drivers import webdriver


def simple_test():
    driver = webdriver.Chrome()

    driver.get("https://example.testproject.io/web/")

    driver.find_element_by_css_selector("#name").send_keys("John Smith")
    driver.find_element_by_css_selector("#password").send_keys("12345")
    driver.find_element_by_css_selector("#login").click()

    passed = driver.find_element_by_css_selector("#logout").is_displayed()

    print("Test passed") if passed else print("Test failed")

    driver.quit()


if __name__ == "__main__":
    simple_test()

I'm getting
InvalidTokenException: 401 Client Error: Unauthorized for url: http://127.0.0.1:8585/api/development/session

[ ERROR ] Failed to initialize a session with the Agent - invalid developer token supplied

Using the previous dev version or the official on Pypi works well.

self._driver.find_element(MobileBy.ANDROID_UIAUTOMATOR, "new UiSelector().text(\"SAVE\")").click()

Returns:

selenium.common.exceptions.WebDriverException: Message: An unknown server-side error occurred while processing the command. Original error: java.lang.ClassCastException: io.appium.uiautomator2.model.UiObjectElement cannot be cast to io.appium.uiautomator2.model.UiObject2Element

While applying the click() method.

When using Webdriber.Generic, no assertions will be reported to TestProject, only the first assertion will appear as skipped:

See PyTest code:

import pytest
import requests
import json
from src.testproject.decorator import report_assertion_errors
from src.testproject.sdk.drivers import webdriver

@pytest.fixture(scope="session", autouse=True)
def driver():
    driver = webdriver.Generic(projectname = "PyTest Reports", token = "TOKEN")
    yield driver
    driver.quit()

def get_response_status_code():
    response = requests.get("https://www.metaweather.com/api/location/search/?query=San Francisco")
    return response.status_code

def get_response_body():
    response = requests.get("https://www.metaweather.com/api/location/search/?query=San Francisco")
    return json.dumps(response.json())

@report_assertion_errors
def test_response_status_code():
    # Should be equal to 200.
    assert get_response_status_code() == 500

@report_assertion_errors
def test_response_location_type():
    # Should contain 'City' in response.
    assert ('Does not appear' in get_response_body())

Actual On any assertion failure, I'm getting the fail in my local console but it's not been propagated to the reports
Expected The test report should have the info about the failed test (assert failure)

To Reproduce

from src.testproject.sdk.drivers import webdriver
import unittest

class SampleTest(unittest.TestCase):

    def test_google_search(self):
        driver = webdriver.Chrome()
        driver.implicitly_wait(10)
        driver.get("https://www.google.com/")
        search_button = driver.find_element_by_xpath("//input[@name='q']")
        self.assertTrue(search_button.is_displayed()) 
        self.assertEqual(search_button.text, "google_search_lowercase")
        driver.quit()

if __name__ == "__main__":
    unittest.main()

Reports shows all green and giving false impression about the tests been passed.

The current logic does not properly report driver commands and other actions with the test method they belong to. This needs to be addressed.

Here's an example test using unittest that has some steps in the setUp and tearDown methods:

import unittest

from src.testproject.sdk.drivers import webdriver


class TestSetupAndTeardown(unittest.TestCase):

    def setUp(self):
        self.driver = webdriver.Chrome()
        self.driver.get("https://example.testproject.io/web/")
        self.driver.find_element_by_css_selector("#name").send_keys("John Smith")
        self.driver.find_element_by_css_selector("#password").send_keys("12345")
        self.driver.find_element_by_css_selector("#login").click()

    def test_login_was_successful(self):
        self.assertTrue(self.driver.find_element_by_css_selector("#greetings").is_displayed())

    def tearDown(self):
        self.driver.quit()

What I would like to see is that these steps are reported as part of the test, and not in separate setUp or tearDown tests:

Some assertions appear with additional unwanted characters such as line breaks in the reports, for example while running the following assertion:

def get_response_status_code():
response = requests.get("https://www.metaweather.com/api/location/search/?query=San Francisco")
return response.status_code

@report_assertion_errors
def test_response_status_code():
# Should be equal to 200.
assert get_response_status_code() == 500

In TestProject, the assertion in the report will appear as follows:

With some unwanted characters like '\n+'

The "capture screenshot" doesn't look enabled by default. I want to automatically capture the screenshots as part the TestProject Reports.

• Element locating failure
• Assert Failure

Is it supported? How to enable?
My snippet

from src.testproject.sdk.drivers import webdriver
from src.testproject.decorator import report
import unittest

class SampleTest(unittest.TestCase):

    @report(project='E-commerce', job='job1', test='Flight Search Test')
    def test_flight_search(self):
        driver = webdriver.Chrome()
        driver.implicitly_wait(10)
        driver.get("https://www.google.com/")
        sort_by = driver.find_element_by_xpath("//div[contains(@class,'fb padTB10 ico14')]") # meant to fail
        self.assertTrue(sort_by.is_displayed()) 
        self.assertEqual(sort_by.text, "No results for this search")

        driver.report().step(
        	description='Page: Search results', 
        	message='Taking screenshot',
        	passed=True,
        	screenshot=True
        	)  #this manual taking works but I need by default because I wouldn't know which step the test is failing
        #driver.report().test(name="Failing test", passed=False)
        driver.quit()


if __name__ == "__main__":
    unittest.main()

following a simple test which will be executed on two different browsers:

from src.testproject.sdk.drivers import webdriver

def simple_test():

    drivers = [ "Chrome", "Edge" ]
    for d in drivers:
        if d == "Chrome":
            driver = webdriver.Chrome(token='DEV_TOKEN')
        else:
            driver = webdriver.Edge(token="DEV_TOKEN")

        driver.get("https://example.testproject.io/web/")
        driver.find_element_by_css_selector("#name").send_keys("John Smith")
        driver.find_element_by_css_selector("#password").send_keys("12345")
        driver.find_element_by_css_selector("#login").click()
        passed = driver.find_element_by_css_selector("#logout").is_displayed()
        print("Test passed") if passed else print("Test failed")
        driver.quit()

if __name__ == "__main__":
    simple_test()

when navigating to the reports, we can see that both reports are combined in a single one:

Problem

I cannot seem to run tests using Safari as the browser on my macOS machine. Chrome and Firefox work just fine, however. I don't know if I set up something incorrectly or if there is a problem with the TestProject agent.

Agent Info

Error Messages

>     b = webdriver.Safari(**kwargs)

tests/conftest.py:58: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/src/testproject/sdk/drivers/webdriver/safari.py:30: in __init__
    super().__init__(
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/src/testproject/sdk/drivers/webdriver/base/basedriver.py:65: in __init__
    self._agent_client: AgentClient = AgentClient(
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/src/testproject/sdk/internal/agent/agent_client.py:72: in __init__
    if not self.__start_session():
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/src/testproject/sdk/internal/agent/agent_client.py:90: in __start_session
    start_session_response = self._request_session_from_agent()
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/src/testproject/sdk/internal/agent/agent_client.py:124: in _request_session_from_agent
    self.__handle_new_session_error(response)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

response = <src.testproject.executionresults.operationresult.OperationResult object at 0x108d94ac0>

    @staticmethod
    def __handle_new_session_error(response: OperationResult):
        """ Handles errors occurring on creation of a new session with the Agent
    
        Args:
            response (OperationResult): response from the Agent
        """
        if response.status_code == 401:
            logging.error("Invalid developer token supplied")
            logging.error(
                "Get your developer token from https://app.testproject.io/#/integrations/sdk?lang=Python"
                " and set it in the TP_DEV_TOKEN environment variable"
            )
            logging.error(f"Response from Agent: {response.message}")
            raise SdkException("Invalid developer token supplied")
        elif response.status_code == 406:
            logging.error(f"This SDK version ({ConfigHelper.get_sdk_version()}) is incompatible with your Agent version.")
            logging.error(f"Response from Agent: {response.message}")
            raise SdkException(f"Invalid SDK version {ConfigHelper.get_sdk_version()}")
        else:
            logging.error("Failed to initialize a session with the Agent")
            logging.error(f"Response from Agent: {response.message}")
>           raise SdkException("Failed to initialize a session with the Agent")
E           src.testproject.sdk.exceptions.sdkexception.SdkException: Failed to initialize a session with the Agent

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/src/testproject/sdk/internal/agent/agent_client.py:271: SdkException
------------------------------------------------------------- Captured log setup --------------------------------------------------------------
ERROR    root:agent_client.py:269 Failed to initialize a session with the Agent
ERROR    root:agent_client.py:270 Response from Agent: 500 Server Error: Internal Server Error for url: http://127.0.0.1:8585/api/development/session
=========================================================== short test summary info ===========================================================
ERROR tests/traditional/test_duckduckgo.py::test_basic_duckduckgo_search[panda] - src.testproject.sdk.exceptions.sdkexception.SdkException: ...

This class for example:

@pytest.mark.usefixtures("driver")
class DemoTests(unittest.TestCase):

    @pytest.fixture(autouse=True)
    def classSetup(self, driver):
        self.login_page = LoginPage(driver)
        self.profile_page = ProfilePage(driver)
        self.driver = driver

    def test_1(self):
        self.login_page.open().login_as("Test 1", "12345")
        assert self.profile_page.greetings_are_displayed() is True

    def test_2(self):
        self.login_page.open().login_as("Test 2", "12345")
        assert self.profile_page.greetings_are_displayed() is True

    def test_3(self):
        self.login_page.open().login_as("Test 3", "12345")
        assert self.profile_page.greetings_are_displayed() is True

Reports the following:

Instead of reporting the tests inside the class.

Using this test:

from addons.web_extensions import WebExtensions
from selenium.webdriver.common.by import By
from src.testproject.classes import DriverStepSettings, StepSettings
from src.testproject.decorator import report_assertion_errors
from src.testproject.sdk.drivers import webdriver
from src.testproject.sdk.drivers.actions import Actions
import pytest



@pytest.fixture()
def driver():
    driver = webdriver.Chrome(token="",
                              project_name="Test",
                              job_name="Test")
    step_settings = StepSettings(timeout=2000)
    with DriverStepSettings(driver, step_settings):
        yield driver
    driver.quit()


@report_assertion_errors
def test_main(driver):
    # Test Parameters
    ApplicationURL = "https://mobii.ai"  # Auto generated application URL parameter

    # 1. Navigate to '{ApplicationURL}'
    # Navigates the specified URL (Auto-generated)
    driver.get(f'{ApplicationURL}')

    # 2. Click 'IMG4'
    # step_settings = StepSettings(always_pass=True)
    # with DriverStepSettings(driver, step_settings):
    #     img4 = driver.find_element(By.XPATH,
    #                                "//swiper/div/button/img")
    #     img4.click()

    # 3. Set window size to ('760','1020')
    driver.addons().execute(
        WebExtensions.setwindowsize(
            width="760",
            height="1020"))

    # 4. Pause for '1000'ms
    driver.pause(milliseconds=1000)

    # 5. Click 'IMG31'
    img31 = driver.find_element(By.XPATH,
                                "//div[1]/div/div[5]/a//img")
    img31.click()

    # 6. Is 'IMG32' visible?
    img32 = driver.find_element(By.XPATH,
                                "//div/div/a/img")
    assert img32.is_displayed()

    # 7. Click 'IMG9'
    img9 = driver.find_element(By.XPATH,
                               "//a[5]/img[1]")
    img9.click()

Results in:

FAILED test_1_244_demo.py::test_main - AttributeError: 'CustomCommandExecutor' object has no attribute 'step_settings'

At step:

    # 4. Pause for '1000'ms
    driver.pause(milliseconds=1000)

Using SDK 0.65.2

Expectation:
Normal WebDriver api calls should not block the uploading of reports to the testproject.io service.

Actual:
It appears that when the set_window_size() method is being used in a test, the report is not uploaded to the testproject.io service

Execution context:

• Windows: CLI command "systeminfo" output

OS Name:                   Microsoft Windows 10 Pro
OS Version:                10.0.19042 N/A Build 19042
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Workstation
OS Build Type:             Multiprocessor Free

• Python: python --version

Python 3.8.5

• TestProject Agent version = 0.64 running installed on Windows

Reproduce

• Using pipenv build up the Python env (pytest, selenium, testproject-python-sdk )

Sample code:

import os
import pytest

from src.testproject.sdk.drivers import webdriver


@pytest.fixture
def browser():
    os.environ['TP_DEV_TOKEN'] = '<insert_your_token_here>'
    driver = webdriver.Chrome()
    yield driver
    driver.quit()


def test_report_step_screenshot(browser):
    browser.set_window_size(1920, 1080)
    browser.get('http://www.google.com')
    assert 'Google' in browser.title

• removing the statement: browser.set_window_size(1920, 1080) will actually make sure the report does get uploaded.
• hint: have a look at the agent log after running the test. It shows errors related to uploading the report.

Hi, this is the message I received when I ran the test with the Python SDK while the Agent was on recording mode with an Android device:

.testproject.sdk.exceptions.agentconnectexception.AgentConnectException: Agent responded with HTTP status 409: [Test Aborted: 'Chrome' driver cannot be created in 'Android' device.]

testdemo2.zip
logs (19).zip

Expected Behavior

After I execute the unittest class, I expected to see one job in the cloud report with 2 tests inside, each test have the name of it's method name.

Current Behavior

After the unittest class finished to execute, the report I got is the execution of 2 jobs, each job have one test case with name UntitledTestCase (and not the name of the method)

Possible Solution

I don't know about any work around for this. If I want to recreate the driver before each test, I don't have any options.

Steps to Reproduce

Try to run the code:

import unittest

from src.testproject.sdk.drivers import webdriver


class UntitledTestCase(unittest.TestCase):

    def setUp(self):
        self.driver = webdriver.Chrome(jobname="Job with multiple tests")

    def tearDown(self):
        self.driver.quit()

    def test_untitled_test_case1(self):
        self.driver.get("http://google.com")


    def test_untitled_test_case2(self):
        self.driver.get("http://yahoo.com")


if __name__ == "__main__":
    unittest.main()

Context (Environment)

Windows 10
Agent 0.63.1
Latest SDK version

Wanted Behavior

Test should look like this (with data source icon)

And the test steps should be:

Current Behavior

I get 2 jobs executions in report.
First job is:

Second job is:

Possible Solution

Don't know..
I want datasource report

Steps to Reproduce

Try to run the code:

import pytest
from src.testproject.sdk.drivers.webdriver import Chrome

@pytest.mark.parametrize('url', ['http://google.com', 'http://yahoo.com'])
def test_parametrize_test(url):
    driver = Chrome(projectname="python sdk project",
                    jobname="pytest_decorators",
                    disable_reports=False)

    driver.get(url)

    driver.quit()

    print("Done")

Context (Environment)

Windows 10
Agent 0.63.1
SDK Version 0.63.10rc3

Detailed Description

Like recorded test with data source selected, it should be the same from this code.
Note: I don't expect that the test names will be different. The test names are fine.
The issue is that it is not reported as execution with data source iterations.
Instead, it reported as 2 jobs that each job have different test

A Boolean flag should be added to the SDK driver to be able to control and suppress warnings.

For example, while building my own test, I control the flow of how the tests are reported, the following warning:

[ WARN ] Automatic reporting is enabled, disable this using disable_reports flag when creating a driver instance to avoid duplicates in the report

Is redundant.

Problem
To prepare simple, pure API (using e.g. module requests) tests based on Python+unittest I have to include webdriver (e.g. Chrome) to be able to use driver.report() to report test status.

Idea
Idea would be to prepare generic, browser-less and light driver, that would allow developer to use report() without initialization of any browser.

Details

To visualize the problem I have attached code of sample API test based on requests and unittest where I have to use webdriver.Chrome to report results to my TestProject project:

import unittest
import requests
from src.testproject.sdk.drivers import webdriver
from selenium import webdriver as seleniumWebdriver

import params


class ApiTests(unittest.TestCase):

    def setUp(self):
        options = seleniumWebdriver.ChromeOptions()
        options.add_argument('headless')
        self.driver = webdriver.Chrome(chrome_options=options, token=params.token)

    def test_get_google_check_status_code_equals_200(self):
        response = requests.get("https://www.google.com/")
        assert response.status_code == 200

        # and to report result we have to use self.driver.report(), e.g.:
        self.driver.report().step(description='Test result', message='I set test status to passed', passed=True)

    def tearDown(self):
        self.driver.quit()

Expectation:
The success of the wait.until(expected_conditions.ec.invisibility_of_element(_located)) will also be reported as a step which passed (= successful/green etc) in the testproject.io report

Actual:
The step is shown as failed in the testreport.io report. But running the test locally through the pytest execution will make the test pass.

testproject.io report:

Local testexecution result:

test_wait_with_ec_invisible.py::test_wait_with_ec_invisible PASSED       [100%]

============================= 1 passed in 10.88s ==============================
Process finished with exit code 0

Execution context:

• Windows: CLI command "systeminfo" output

OS Name:                   Microsoft Windows 10 Pro
OS Version:                10.0.19042 N/A Build 19042
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Workstation
OS Build Type:             Multiprocessor Free

• Python: python --version

Python 3.8.5

• TestProject Agent version = 0.64 running installed on Windows

Reproduce

• Using pipenv build up the Python env (pytest, selenium, testproject-python-sdk )

Sample code:

import os
import pytest

from src.testproject.sdk.drivers import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec


@pytest.fixture
def browser():
    os.environ['TP_DEV_TOKEN'] = '<insert_your_token_here>'
    driver = webdriver.Chrome()
    yield driver
    driver.quit()


@pytest.fixture()
def wait(browser):
    wait = WebDriverWait(browser, 2)
    yield wait


def test_wait_with_ec_invisible(browser, wait):
    browser.get('http://www.google.com')
    search_field = (By.CSS_SELECTOR, "input[name='z']")  # => z should be q, thus simulating a non-present element
    element_not_present = wait.until(ec.invisibility_of_element_located(search_field))
    assert element_not_present is True

Hello,

Issue is submitted due to the subject mentioned in the title.

When initializing the web driver using my dev token:
driver = webdriver.Chrome(token='MY_DEV_TOKEN', projectname='MY_PROJ_NAME')

It will still attempt to use a token in an environment variable.

Expected Behavior

I should see a report of one job with one test that failed

Current Behavior

The job I get have another test:

Possible Solution

The only workaround is to report a failed step

Steps to Reproduce

Try to run the code:

def test_fail_due_to_report_test_as_failed():
    driver = Chrome(projectname="python sdk project",
                    jobname="fail_tests",
                    disable_reports=False)

    driver.get("http://example.testproject.io/")

    driver.report().test(passed=False)

    driver.quit()

    print("done")

Environment

Windows 10
Agent 0.63.1
SDK version 0.63.10rc3

Detailed Description

What making the extra test is line:

driver.report().test(passed=False)

Example Test

@pytest.mark.parametrize('phrase', ['panda', 'python', 'polar bear'])
def test_basic_duckduckgo_search(phrase):
  pass

TestProject Reports

"panda" is okay:

"polar bear" is not:

Characters after the whitespace character are truncated.

The web page isn't dynamically shortening that line of text, either. The text is truncated in the HTML element:

<span class="tp-text-ellipsis ng-binding" ng-attr-title="{{ ::( test.name ) }}" title="test_basic_duckduckgo_search[polar">test_basic_duckduckgo_search[polar</span>

Related Issues

#8 is a feature for improved test parameter reporting. This issue may become a non-issue based on the outcome of #8.

In my Library I have the options to create Capabilities hardcoded as a string where I parse it to dictionary.
At the moment the driver takes only Options object because it calls the to_capabilities() method.

self._build_capabilities(WebDriverCreator(os.getcwd())._parse_capabilities( capabilities=desired_capabilities, browser=browser))

Returns a dictionary of capabilities which I wish to pass to the driver
webdriver.Chrome(desired_capabilities, dev_token, project_name, job_name, disabled_reports)

It looks like so:
{'desired_capabilities': {'browserName': 'chrome', 'platform': 'web', 'version': '86', 'name': 'RobotFramework Lambda Test'}}

It would be great if there was a generic solution to parse dictionary to desired_capabilities.

This is true for FireFox, IE and Chrome and Edge, since Safari already take the desired capabilities as a dict.

Expected Behavior

I should see 2 tests in the job, each test have one step

Current Behavior

I get job with one test that has 2 steps

Possible Solution

Don't know

Steps to Reproduce

Try to run the code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import NoAlertPresentException
import unittest, time, re

from src.testproject.decorator import report
from src.testproject.sdk.drivers.webdriver import Chrome, Firefox


class UntitledTestCase(unittest.TestCase):

    driver = None

    @classmethod
    def setUpClass(cls):
        cls.driver = Chrome( projectname="python sdk project",
                             jobname="python sdk job")  # With TestProject driver it will fail

        cls.base_url = "https://www.google.com/"
        cls.verificationErrors = []
        cls.accept_next_alert = True

    @classmethod
    def tearDownClass(cls):
        cls.driver.quit()

    @classmethod
    def test_untitled_test_case1(cls):
        driver = cls.driver
        driver.get("https://google.com/")

    @classmethod
    def test_untitled_test_case2(cls):
        driver = cls.driver
        driver.get("https://yahoo.com/")


if __name__ == "__main__":
    unittest.main()

Context (Environment)

Windows 10
Agent 0.63.1
Latest SDK version

Detailed Description

I try to reuse the same driver for all tests.
I should not get one test but 2 tests

After the setup of TestProject agent on my mac and developer setting on my android device, I am facing issues on loading the page of any app. It seems loader icon on the page has been changed to the refresh icon. To be sure I did reset to factory setting and the apps were working fine but as soon as I connected and ran the TestProject Agent, refresh icon appeared again with the complete blank screen while loading the page instead of the loader. I have tried the same on the other Android device but the same issue occurred.

When the Agent URL is specified with a trailing slash, e.g. http://127.0.0.1:8585/, SDK appends the endpoint path as /api/development/session which results in http://127.0.0.1:8585//api/development/session (note the double slashes) that returns a response code 404 when the request is sent to the Agent.

For a reference: https://forum.testproject.io/t/python-sdk-response-from-agent-404/2427/6

While running a test with the disable_reports = true, the agent returns 406 per driver command.

When using the @report annotation to decide the project name, the SDK ignores it and instead infers the name of the project to the pytest file.

How to reproduce?

@pytest.fixture()
def driver():
    driver = webdriver.Chrome(token="DEV_TOKEN")
    yield driver
    driver.quit()


# MAIN TEST #
@report_assertion_errors
@report(project="3. Bugs / New Tests - QA111", job="very nice job name", test="very simple web")
def test_normalized_function_test_name(driver):
    # STEP 1
    driver.get(url='https://www.google.co.il')

The following report is created with multiple issues :

1. Project name is ignored, instead we see the python file name (which is also missing a 'y', should be filename.py not filename.p)
2. Job name is ignored as well, just as with Project name.
3. Multiple tests are created, 1 is the provided test name in the @report annotation, the other 1 is the test function name.
   (Removing the 'test_' prefix from the function name isn't an option I assume because the pytest will no longer recognize the function as a test that needs to be ran, correct? @basdijkstra)

Using this test:

from src.testproject.sdk.drivers import webdriver

def simple_test():
    driver = webdriver.Safari()

    driver.get("https://example.testproject.io/web/")

    driver.find_element_by_css_selector("#name").send_keys("John Smith")
    driver.find_element_by_css_selector("#password").send_keys("12345")
    driver.find_element_by_css_selector("#login").click()

    passed = driver.find_element_by_css_selector("#logout").is_displayed()

    print("Test passed") if passed else print("Test failed")

    driver.quit()



if __name__ == "__main__":
    simple_test()

When the desired browser is not installed, the SDK returns the HTTP code instead of the message from the agent.

src.testproject.sdk.exceptions.agentconnectexception.AgentConnectException: Agent responded with HTTP status 404: [404 Client Error: Not Found for url: http://127.0.0.1:8585/api/development/session]

2020-11-19 10:02:33.405 [ERROR] i.t.a.d.a.h                              Failed to activate Developing state
io.testproject.agent.dto.exceptions.MissingBrowserException: Could not discover browser Safari version.