taverntesting / tavern Goto Github PK

How do you think this should work @michaelboulton? Can you come up with something and implement it with some examples in the docs site?

When excecuting (env) @math:tavern $ tavern-ci minimal_test.tavern.yaml an error is printed on console
and the test doesn't run.
Contents from minimal_test.tavern.yaml is the 'Simple test' from: https://taverntesting.github.io/examples

I instaled tavern with: pip install tavern

Error doesnt happen when using python3.6.

For example if we want to generate a random uuid from an external function and make sure it's in the response, we might want to access it using format keys like:

request:
  body:
    $ext:
      function: uuid:uuid4
response:
  body:
    returned_uuid: "{request_values.body.uuid}"

If there's a missing key, print something more useful than

13:43:24 [ERROR]: (tavern.request:57) Key not found in format

Number of total stages (not just tests), time for each request, etc.

Any chance of getting Jinja template style filters?
Would be nice to be able to do something like

body:
  json:
    mydata: "{saved_value | b64decode}"

At the moment they are validated on load with validate_extensions, but can we also call them and replace the value with the return value automatically?

if send

json:
        settings: []
        agentslist:
          - '{agent_id:d}'

Server define agent_in type of string. for example '2'

A public docker image would be useful for getting tests running faster, as mentioned in #46.

Let's work out how best to make a docker image which can access the test files, perhaps through a volume?

Would be nice to be able to define a base URL from the command line.

Earlier my most of work of api design is done in django and python 2.7.Can I use this for API testing in python 2.7 or will work in function with python 3 only ?

We are testing on JSON files like so: https://jsonplaceholder.typicode.com/users/1 . However, we don't want to test on every given parameter, but only on some of them in every stage.

Is there a way to ignore parameters, that I don't want to test?
In the given example I'd like to test for the street and for the Street only.
Therefor I tried:

test_name: Street
stages:
    - name: Test address-street = "Kulas Light"
      request:
        url: https://jsonplaceholder.typicode.com/users/1
        method: GET
      response:
        body:
          address:
            street: Kulas Light

This doesn't pass, since tavern expects me to list "suite","city", etc.

Any suggestions?

I think I've seen some kind of guide for this on github somewhere

Command

$ tavern-ci --debug some_file.tavern.yaml

Response in Terminal

Value mismatch: '159295' vs '159295'
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tavern/response.py", line 210, in _validate_block
    assert actual_val == expected_val
AssertionError

Test to save user_id from response

  - name: login
    request:
      url: some_test_url.com/login
      json:
        username: someUser
        password: somePassword
        grant_type: password
      method: POST
      headers:
        content-type: application/json
        accept: application/json
    response:
      status_code: 200
      body:
        token_type: bearer
        scope: user
        expires_in: 604800
      headers:
        content-type: application/json; charset=utf-8
      save:
        body:
          access_token: access_token
          refresh_token: refresh_token
          user_id: user_id

Test to use user_id in subsequent test

  - name: create_book
    request:
      url: some_test_url.com/books
      json:
        title: Some title of a book
      method: POST
      headers:
        content-type: application/json
        accept: application/json
        Authorization: "Bearer {access_token:s}" #working to reuse variable saved
    response:
      status_code: 201
      body:
        user_id: "{user_id}"
      headers:
        content-type: application/json; charset=utf-8

I'm trying to access the variable within a json response. The body contains a list of items, each with a collection :

response:
     status_code: 200
     body:
          results: 1
          items:
          - "u":
                "name": "CREATE_TEST_USER"
                "user_id": "{user_id}"

The output in server:

[{'u': {'user_id': '{user_id}'}]

The structure works fine without variables.

Hi,

I would like to hire some woth experience in wroting tests on top of this repo, and/or writting pull requests. please leave your details or write to me [email protected]

• Set travis up to deploy to pypi on a new tag if all the tests pass.
• Possibly shuffle the travis CIs file around a bit so it runs a few less integration tests on a normal commit, but also runs the mqtt integration tests on a tagged commit
• Auto deploy a docker image as well when #48 is done

Hey,
Can you use pytest parameters when you run tavern-cli?

Currently i would like to generate --junitxml file.

Thanks

Possibly also for pyresttest?

I want to validate my api response using tavern, but unfortunately I can't validate it because it throw exception ..

somehow if i remove validation for key "data" it was success ..

So the response is:

{
"data": { "foo": "bar", "foo": "bar"},
"meta": { "foo": "bar", "foo": "bar"}
}

I can validate meta key, but not with data key.

So if use this in response:

It will throw something like this:

I expect something like this:

I'm trying to use tavern for testing an API based on flask_restful and flask_httpauth. This modules use the auth header, and I think there is no support for it yet. I've done several tests and that's what i've figured out (sorry, I can't follow down your code... yet)

My server is capable of answering correctly the following CURL query:

curl -H "Content-type:application/json" \
    -u 'user:password' \
    -d '{"param":"value"}' \
    -X GET 'http://127.0.0.1:5000/api/v1.0/user'

Please notice that the login is redundant for the sake of this example
However, i've tried to include this information in the yaml file, within the request and headers block, but I haven't been able to receive any information on my server.
Many thanks in advance!

It seems like the integration test tool should use the integration tests

Something like this:

• https://pytest-flask.readthedocs.io/en/latest/
• https://pypi.python.org/pypi/pytest-localserver

We should add a bit of info to the readme about setting up for dev, running tests etc to make it easier for people to contribute.

Allow variables such as sensitive API keys or other auth tokens to be set as environment variables and substituted in config.yaml when parsed at runtime (i.e. with a $ENV{ENVNAME} type syntax). This is standard practice for cloud-based CI solutions such a Travis, Circle CI, etc.

Okay so I've been trying to find information on this problem however it seems that your package is fairly new so I can't find much online. Anyways, my problem is that I am trying to test my server however I keep ending up with badschemaerrors whenever I try to test my yaml file with pytest. Here is my yaml file:

# test_sz.tavern.yaml

---

test_name: Testing Authentication Module

stages:
 request:
  url: https://localhost:8443/api/authenticate
  method: POST
  data: 
   password: *****
   username: *****
   thirdPartyLogin: false
   forceLogin: false
  headers:
   content-type: application/x-www-form-urlencoded
 response:
  status_code: 200

---

and here is what I get in response to running py.test in the same directory as the file:

C:\Users\TON\Documents\Rest API Pulling Documentation>py.test
============================= test session starts =============================
platform win32 -- Python 3.6.4, pytest-3.3.2, py-1.5.2, pluggy-0.6.0
rootdir: C:\Users\TON\Documents\Rest API Pulling Documentation, inifile:
plugins: tavern-0.2.1
collected 0 items / 1 errors

=================================== ERRORS ====================================
____________________ ERROR collecting test_sz.tavern.yaml _____________________
..\..\appdata\local\programs\python\python36-32\lib\site-packages\tavern\schemas\files.py:38: in verify_generic
    verifier.validate()
..\..\appdata\local\programs\python\python36-32\lib\site-packages\pykwalify\core.py:176: in validate
    error_msg=u'.\n - '.join(self.validation_errors)))
E   pykwalify.errors.SchemaError: <SchemaError: error code 2: Schema validation failed:
E    - Value '{'request': {'data': {'forceLogin': False, 'password': 'Panduit12', 'thirdPartyLogin': False, 'username': 'admin'}, 'headers': {'content-type': 'application/x-www-form-urlencoded'}, 'method': 'POST', 'url': 'https://localhost:8443/api/authenticate'}, 'response': {'status_code': 200}}' is not a list. Value path: '/stages'.: Path: '/'>

The above exception was the direct cause of the following exception:
..\..\appdata\local\programs\python\python36-32\lib\site-packages\tavern\testutils\pytesthook.py:49: in collect
    verify_tests(test_spec)
..\..\appdata\local\programs\python\python36-32\lib\site-packages\tavern\schemas\files.py:82: in verify_tests
    verify_generic(test_tmp, schema_filename)
..\..\appdata\local\programs\python\python36-32\lib\site-packages\tavern\schemas\files.py:41: in verify_generic
    raise_from(BadSchemaError(), e)
..\..\appdata\local\programs\python\python36-32\lib\site-packages\future\utils\__init__.py:398: in raise_from
    exec(execstr, myglobals, mylocals)
<string>:1: in <module>
    ???
E   tavern.util.exceptions.BadSchemaError
------------------------------- Captured stderr -------------------------------
validation.invalid
 --- All found errors ---
["Value '{'request': {'data': {'forceLogin': False, 'password': '****', 'thirdPartyLogin': False, 'username': '****'}, 'headers': {'content-type': 'application/x-www-form-urlencoded'}, 'method': 'POST', 'url': 'https://localhost:8443/api/authenticate'}, 'response': {'status_code': 200}}' is not a list. Value path: '/stages'"]
Error validating C:\Users\TON\AppData\Local\Temp\tmp8irts1la.yaml
Traceback (most recent call last):
  File "c:\users\ton\appdata\local\programs\python\python36-32\lib\site-packages\tavern\schemas\files.py", line 38, in verify_generic
    verifier.validate()
  File "c:\users\ton\appdata\local\programs\python\python36-32\lib\site-packages\pykwalify\core.py", line 176, in validate
    error_msg=u'.\n - '.join(self.validation_errors)))
pykwalify.errors.SchemaError: <SchemaError: error code 2: Schema validation failed:
 - Value '{'request': {'data': {'forceLogin': False, 'password': '****', 'thirdPartyLogin': False, 'username': '****'}, 'headers': {'content-type': 'application/x-www-form-urlencoded'}, 'method': 'POST', 'url': 'https://localhost:8443/api/authenticate'}, 'response': {'status_code': 200}}' is not a list. Value path: '/stages'.: Path: '/'>
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.51 seconds ===========================

It keeps talking about how the value is not a list which makes no sense to me. Thanks for taking the time to take a look at this and I look forward to your responses! :)

I currently consider using tavern for testing our backend services and would appreciate some of your thoughts.

Is there any recommended way of setting up a (local) testing environment with flask? Flask itself provides a test client which can be used in unit tests.

Since spinning up a real test server running on localhost brings other issues with it (clean shutdown, port may already be used etc.), I would like to fall back to this test client.

As far as I can tell, tavern currently only supports requests using the requests (+ mqtt, which is not relevant to flask) module. I suppose one could statically add flask test-client support in the taverns.plugins module, but this seems a little bit hacky.

• Does anybody have experience in using tavern with a local flask setup, preferrably without spinning up a local server?
• Are there any plans and/or development ongoing regarding the plugin system?
• If no, is there any set of requirements or specification regarding a plugin system? If we decide to use tavern ourselves, contributing here might be of interest.

Hello gang!

I believe that there might be an issue reading integers from one file when they've been included in another.

For instance:
foo.yaml:
owner-id: 1001

bar.yaml:
!include: foo.yaml
owner-id: {owner-id}
--or
owner-id: "{owner-id:d}"

So far, I haven't been able to get this to return an int to the calling document. If explicitly casting to an int:

bar.yaml:
!include: foo.yaml
owner-id: !!int "{owner-id:d}"

The following error is produced: ValueError: invalid literal for int() with base 10: '{testid'
because the cast is happening before the variable is returned.

If I have an incorrect implementation somewhere, please let me know! Thanks!

Something like the default pytest one:

https://github.com/pytest-dev/pytest/blob/master/_pytest/terminal.py#L126

We want it to print out the yaml that caused the failure, the line, the file, etc. Colourised would be nice of course.

Cannot install tavern as pyyaml is one of its dependencies

Environment:

OS: Ubuntu 16.04.2 LTS
Python version: 3.6.2
Pip3 version: 9.0.1

Expected result:

pip3 install tavern to successfully install tavern and its dependencies

Actual result:

Failed building wheel for pyyaml

(env) banx@banx-Latitude-E6420:~/other_projects/api$ pip3 install tavern
Collecting tavern
Requirement already satisfied: requests in /home/banx/env_for_random_py_modules/lib/python3.6/site-packages (from tavern)
Collecting pyyaml (from tavern)
  Using cached PyYAML-3.12.tar.gz
Collecting pykwalify (from tavern)
  Using cached pykwalify-1.6.0-py2.py3-none-any.whl
Requirement already satisfied: python-box in /home/banx/env_for_random_py_modules/lib/python3.6/site-packages (from tavern)
Requirement already satisfied: future in /home/banx/env_for_random_py_modules/lib/python3.6/site-packages (from tavern)
Requirement already satisfied: contextlib2 in /home/banx/env_for_random_py_modules/lib/python3.6/site-packages (from tavern)
Requirement already satisfied: pyjwt in /home/banx/env_for_random_py_modules/lib/python3.6/site-packages (from tavern)
Requirement already satisfied: certifi>=2017.4.17 in /home/banx/env_for_random_py_modules/lib/python3.6/site-packages (from requests->tavern)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/banx/env_for_random_py_modules/lib/python3.6/site-packages (from requests->tavern)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /home/banx/env_for_random_py_modules/lib/python3.6/site-packages (from requests->tavern)
Requirement already satisfied: idna<2.7,>=2.5 in /home/banx/env_for_random_py_modules/lib/python3.6/site-packages (from requests->tavern)
Requirement already satisfied: docopt>=0.6.2 in /home/banx/env_for_random_py_modules/lib/python3.6/site-packages (from pykwalify->tavern)
Requirement already satisfied: python-dateutil>=2.4.2 in /home/banx/env_for_random_py_modules/lib/python3.6/site-packages (from pykwalify->tavern)
Requirement already satisfied: six>=1.5 in /home/banx/env_for_random_py_modules/lib/python3.6/site-packages (from python-dateutil>=2.4.2->pykwalify->tavern)
Building wheels for collected packages: pyyaml
  Running setup.py bdist_wheel for pyyaml ... error
  Complete output from command /home/banx/env_for_random_py_modules/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-crhlvw6w/pyyaml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpnbhy6217pip-wheel- --python-tag cp36:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/error.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/emitter.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/cyaml.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/constructor.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/__init__.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/events.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/representer.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/scanner.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/serializer.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/nodes.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/loader.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/dumper.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/parser.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/resolver.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/reader.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/tokens.py -> build/lib.linux-x86_64-3.6/yaml
  copying lib3/yaml/composer.py -> build/lib.linux-x86_64-3.6/yaml
  running build_ext
  creating build/temp.linux-x86_64-3.6
  checking if libyaml is compilable
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -I/home/banx/env_for_random_py_modules/include/python3.6m -c build/temp.linux-x86_64-3.6/check_libyaml.c -o build/temp.linux-x86_64-3.6/check_libyaml.o
  checking if libyaml is linkable
  x86_64-linux-gnu-gcc -pthread build/temp.linux-x86_64-3.6/check_libyaml.o -lyaml -o build/temp.linux-x86_64-3.6/check_libyaml
  building '_yaml' extension
  creating build/temp.linux-x86_64-3.6/ext
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -I/home/banx/env_for_random_py_modules/include/python3.6m -c ext/_yaml.c -o build/temp.linux-x86_64-3.6/ext/_yaml.o
  ext/_yaml.c:4:20: fatal error: Python.h: No such file or directory
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for pyyaml
  Running setup.py clean for pyyaml
Failed to build pyyaml
Installing collected packages: pyyaml, pykwalify, tavern
  Running setup.py install for pyyaml ... error
    Complete output from command /home/banx/env_for_random_py_modules/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-agtz8xbr/pyyaml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-yt34mhg5-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/banx/env_for_random_py_modules/include/site/python3.6/pyyaml:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/error.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/emitter.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/cyaml.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/constructor.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/__init__.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/events.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/representer.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/scanner.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/serializer.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/nodes.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/loader.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/dumper.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/parser.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/resolver.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/reader.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/tokens.py -> build/lib.linux-x86_64-3.6/yaml
    copying lib3/yaml/composer.py -> build/lib.linux-x86_64-3.6/yaml
    running build_ext
    creating build/temp.linux-x86_64-3.6
    checking if libyaml is compilable
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -I/home/banx/env_for_random_py_modules/include/python3.6m -c build/temp.linux-x86_64-3.6/check_libyaml.c -o build/temp.linux-x86_64-3.6/check_libyaml.o
    checking if libyaml is linkable
    x86_64-linux-gnu-gcc -pthread build/temp.linux-x86_64-3.6/check_libyaml.o -lyaml -o build/temp.linux-x86_64-3.6/check_libyaml
    building '_yaml' extension
    creating build/temp.linux-x86_64-3.6/ext
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -I/home/banx/env_for_random_py_modules/include/python3.6m -c ext/_yaml.c -o build/temp.linux-x86_64-3.6/ext/_yaml.o
    ext/_yaml.c:4:20: fatal error: Python.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Packages installed in attempt to fix issue:

libpq-dev python-dev libxml2-dev libxslt1-dev libldap2-dev libsasl2-dev libffi-dev

Eg if you set the status_code to 1234 it should raise a warning

pykwalify is still used in one of the built in 'extension' functions so we can't completely remove it, but it would be nice to use something other for validating test input, mainly just because it's the slowest part of the whole tool right now. Validating ~2600 lines of tests in one of our project takes about 5 seconds and running the tests takes about 10 seconds.

Tavern currently validates all the tests at discovery time, so a (stopgap) solution is to defer checking the schema until the test is run, which means we can do pytest -k the_test_i_want and discovery won't take 5 seconds when you just want to run 1 test.

I'm trying to use my variable value in the array but it doesn't work. E.g.:

json:
        settings: []
        testlist: ["{test_id:d}"]

I want to upload images just like I do it in Postman. See the screenshot of my Postman session.

Is there a way to specify fixtures for the pytest hook?

test_name: Get some fake data from the JSON placeholder API

stages:
  - name: Verify only id
    request:
      url: https://reqres.in/api/users/1
      method: GET
    response:
      status_code: 200
      body:
        data:
          id: 1
          first_name: null
          last_name: null
          avatar: null

The above test failed with error:
Value mismatch: got '{'id': 1, 'first_name': 'George', 'last_name': 'Bluth', 'avatar': 'https://s3.amazonaws.com/uifaces/faces/twitter/calebogden/128.jpg'}' (type: <class 'dict'>), expected '{'id': 1, 'first_name': None,
'last_name': None, 'avatar': None}' (type: <class 'dict'>)

Would be nice if we could use something like @pytest.mark.parametrize on tests. We can do stuff like this with pytest:

def get_login_stage():
    return {
        "request": {
            ...
        }
        "response": {
            ...
        }
    }

@pytest.mark.parametrize("test_a_thing", (True, False))
def test_something(test_a_thing):
    stages = []

    stages.append(get_login_stage())

    if test_a_thing:
        stages.append(get_optional_stage())
    else:
        stages.append(get_other_stage())

    stages.append(get_final_stage()

    tavern.run_test(stages)

It would be nice if there was a way to do this in the yaml.

This can probably be done with some kind of custom YAML constructor (!parametrize True False...)

What version will be support this?

as i'm currently seeking to solve #6, i noticed there is no documentation on the available global parameters. i'm still undecided for a generic approach as @michaelboulton proposed or a specific approach (tavern --baseurl service.example.org …), which i'd consider more user-friendly and preferable if there are only a few global parameters.

overall, the docs don't explain the concept of global settings or any aspects of it.

eg tavern-ci test_1.tavern.yaml test_2.tavern.yaml

We should come up with some naming convention for 'special' variables, eg

• #22
• #4
• #29

Using the Box library you can return what is essentially a dictionary where things can be accessed with dot notation for formatting - this is currently only possible in external (or tavern-internal) functions. We may want more of these in future. Possible options:

• Allow any key to be returned from a response/external function, but log a warning if a key in the current test_block_config is overridden by an external function (and maybe the return value of a 'response')
• set some naming convention, for example that all ext function keys are uppercased (for #29, regex->REGEX)

Would also be useful add a 'meta'/'magic' key which has environment variables (and maybe some other magic data in future), which can be used like

request:
  json:
    user: "{username}"
    password: "{meta.environment.password}"

and stop anyone from saving to the key 'meta'

@benhowes

There isn't going to be some kind of huge api for this in a first run through. This is already implemented (though hardcoded for 2 request types) in plugins.py. Assuming plugins are like pytest where you define a file called something like 'tavern_plugins.py' (loaded via setuptools entry point automatically). We do need to think of a good way for plugins to register themselves (possibly just a global variable in the file), and a particular setuptools entrypoint name (eg tavernplug = myextension.tavern_plugin)

file has to define 4 functions (run in this order):

• get_session(**kwargs) - takes any generic keyword arguments relevant to initialise the 'session'. For example, mqtt need mqtt host, port, authorization, tls settings, etc. etc. This will basically just be passed through as get_session(**test_spec[session_name]), where 'session_name' is a block that isn't one of the predefined ones (stages, test_name, includes). Has to return something that can be used as a context manager which will be used throughout the test.

• get_request_type(session, request_spec, test_block_variables) - get object with a run() method that actually runs the test and returns some kind of object corresponding to a response. session, is the session returned above (now inside context manager), request_spec is any of the dictionary sub-items for the request (eg for request this is url, headers, params. for mqtt_publish this is topic, payload, json.). test_block_variables is all the variables for using/formatting for that test.

• get_expected(response_spec, test_block_variables) - Get the expected result as a dictionary (in whatever format is required for the verifier). response_spec is any of the dictionary sub-items for the response (eg for response this is status_code, params, etc. for mqtt_response this is topic, payload, json).

• get_verifier(session, expected_response, test_block_variables) - get a object with a verify(response) method that takes the actual response and verifies it against the expected response. session, is the session returned above (now inside context manager). expected_response is the returned value from get_expected for that particular response type - there can be multiple responses per test, so there has to be one verifier for each response. test_block_variables is all the variables for using/formatting for that test.

Other things to do:

• update docs to emphasise that there is only 1 request per test stage, but 0 or more responses (for example, MQTT messages don't always expected a response, but HTTP requests always do - need some way to expose this in the plugins like require_response_block = True)
• Move existing request type (Requests/MQTT) into another subfolder (eg contrib/) and always have them available by default
• Optionally add another function like validate_schema which takes the request/response block and validates it. This can just be done in the get_request_type entrypoint so it might be redundant.
• Move 'MQTT' block (and any other plugin blocks) in test into another key like plugins, eg:

test_name: Do a thing

plugins:
  mqtt:
    connect:
       host: example.com

stages:
  - name: do something
    mqtt_publish:
      ...

I think doing this and making it work with requests and mqtt will be fairly short work, just shuffling the files around a bit.

Hello,

In the list of the requirements can you add pytest-runner

In my test.tavern.yaml file

  - name: Make sure signature is returned
    request:
      url: "{signature_url:s}"
      method: PUT
      headers:
        content-type: application/json
        content: {"ppddCode": "11","LIN": "123456789","correlationID":"{correlationId:s}","bodyTypeCode":"utv"}
        token: "{sessionToken:s}"
    response:
      status_code: 200
      save:
        body:
          signature: signature

I get the following error response:

E           requests.exceptions.InvalidHeader: Value for header {content: {'ppddCode': '11', 'LIN': '123456789', 'correlationID': '99019c36-4f49-4be8-815c-9ff5cbcc14ff', 'bodyTypeCode': 'utv'}} must be of type str or bytes, not <class 'dict'>

venv/lib/python3.6/site-packages/requests/utils.py:872: InvalidHeader

How to convert my 'content' in my header to a dict? Or any other solution?

the source files of the documentation are not tracked with the vcs. that makes it hard to adapt the docs to reflect changes of the software. :-/

test_name: Validating schema and data binding

stages:
  - name: Get car info
    request:
      url: https://reqres.in/api/cars/1
      method: GET
    response:
      status_code: 200
      body:
        # {  
        #   "data":{  
        #     "id":1,
        #     "name":"cerulean",
        #     "year":2000,
        #     "color":"#98B2D1",
        #     "pantone_value":"15-4020"
        #   }
        # }
        $ext:
          # http://pykwalify.readthedocs.io/en/master/
          function: tavern.testutils.helpers:validate_pykwalify
          extra_kwargs:
            schema:
              type: map
              required: True
              mapping:
                data:
                  type: map
                  required: True
                  mapping:
                    id:
                      type: int
                    name:
                      type: str
                    year:
                      type: int
                    color:
                      type: str
                      # using regexp to match hex color code, {6} not working
                      pattern: '#[A-Fa-f0-9]{6}'
                    pantone_value:
                      type: str
                      # {n} not working
                      pattern: '\d\d-\d\d\d\d'

error:
Traceback (most recent call last):
File "C:\Python36\Scripts\tavern-ci-script.py", line 11, in
load_entry_point('tavern==0.6.0', 'console_scripts', 'tavern-ci')()
File "c:\python36\lib\site-packages\tavern\entry.py", line 102, in main
exit(not run(**vargs))
File "c:\python36\lib\site-packages\tavern\core.py", line 167, in run
run_test(in_file, test_spec, global_cfg)
File "c:\python36\lib\site-packages\tavern\core.py", line 92, in run_test
expected = get_expected(stage, test_block_config, sessions)
File "c:\python36\lib\site-packages\tavern\plugins.py", line 125, in get_expected
f_expected = format_keys(r_expected, test_block_config["variables"])
File "c:\python36\lib\site-packages\tavern\util\dict_util.py", line 28, in format_keys
formatted[key] = format_keys(val[key], variables)
File "c:\python36\lib\site-packages\tavern\util\dict_util.py", line 28, in format_keys
formatted[key] = format_keys(val[key], variables)
File "c:\python36\lib\site-packages\tavern\util\dict_util.py", line 28, in format_keys
formatted[key] = format_keys(val[key], variables)
[Previous line repeated 5 more times]
File "c:\python36\lib\site-packages\tavern\util\dict_util.py", line 31, in format_keys
formatted = val.format(**variables)
IndexError: tuple index out of range

In the entry.py file, line 63, the logger format is as follows:
{asctime:s} [{levelname:s}]: ({name:s}:{lineno:d}) {message:s}

Any logging output will just print the line above and not the actual variables.

A quick fix is by replacing it with the following:
%(asctime)s [%(levelname)s]: (%(name)s:%(lineno)d) %(message)s

I have only tested this on python 2.7.

Hey. I've discovered such issue that actually blocks me to even try this app:

1. Do pip install tavern
2. Run tavern-ci --help

I got such output:
Traceback (most recent call last): File "/usr/local/bin/tavern-ci", line 11, in <module> load_entry_point('tavern==0.1.1', 'console_scripts', 'tavern-ci')() File "/Library/Python/2.7/site-packages/pkg_resources/__init__.py", line 572, in load_entry_point return get_distribution(dist).load_entry_point(group, name) File "/Library/Python/2.7/site-packages/pkg_resources/__init__.py", line 2769, in load_entry_point return ep.load() File "/Library/Python/2.7/site-packages/pkg_resources/__init__.py", line 2422, in load return self.resolve() File "/Library/Python/2.7/site-packages/pkg_resources/__init__.py", line 2428, in resolve module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/Library/Python/2.7/site-packages/tavern/entry.py", line 5, in <module> from .core import run File "/Library/Python/2.7/site-packages/tavern/core.py", line 6, in <module> from .util.loader import IncludeLoader File "/Library/Python/2.7/site-packages/tavern/util/loader.py", line 55 Resolver, metaclass=LoaderMeta): ^ SyntaxError: invalid syntax

I have:
$ python --version Python 2.7.10

What to do next?

I'm having some trouble implementing a session using tavern and am a little confused as to how that would work. Normally in python I would create a session object as follows:

with reqeusts.Session() as sesh:
  data = { "username" : "myName", "password" : "myPW"}   
  sesh.get(url,data = data, ... )
  etc.

and then after this just making all of the necessary calls as needed using sesh, however I'm just not sure how to approach this within tavern. My first thought was to create an external function that would act like the Sessions function and grab the cookies, auth, and something else (can't remember off the top of my head right now) and then simply copy and paste that code to each different test. My problem though was that the keywords for auth and for cookies has not yet been implemented.

My main questions are

1. Is there a way to currently implement a constant session with the way that tavern is right now?
2. If not, how could we implement this and what would need to be done in order to get this type of functionality?

Here is the code that I wanted to implement but cannot mainly because cookies has not yet been implemented:

# test_sz.tavern.yaml

---
test_name: Testing *** Module

stages:
- name:  %DOMAIN%/api/authenticate POST
  request:
    url: %DOMAIN%/api/authenticate
    method: POST
    data: 
      password: ***
      username: ***
      thirdPartyLogin: false
      forcelogin: false
    headers:
      content-type: application/x-www-form-urlencoded
      accept: application/json 
    verify: false
  response:
    save:
      $ext:
        function: helperFuncsTavern:getCookie
    status_code: 200

- name: Testing %DOMAIN%/api/search/category GET
  request:
    url:  %DOMAIN%/api/search/category 
    method: GET
    headers:
      content-type: application/json
      accept: application/json
    verify: false
    cookies: cookies.seshCookie
  response:
    status_code: 200

Here is the code from the getCookie external function:

def getCookie(response):
""" Grabs the seshCookie cookie from the response so that
    we can reuse it throughout the rest of the session.  """
    return {"cookies" : {"seshCookie" : response.cookies["seshCookie"]}}

Appreciate the help and hope to hear from you soon!

Edit: Looking at the code again I found that there is actually support for the auth keyword, so ignore that part from above about auth.

Validating this response:

[
    {
        "author": "Dan Brown",
        "isbn": "123456",
        "title": "Digital Fortress"
    },
    {
        "author": "JK Rowling",
        "isbn": "234567",
        "title": "Harry Potter and the Chamber of Secrets"
    }
]

With this test:

test_name: json list response

stages:
  - name: get list
    request:
      url: http://localhost:8080/
      method: GET
    response:
      status_code: 200
      body:
        - author: Dan Brown
          isbn: '123456'
          title: Digital Fortress
        - author: JK Rowling
          isbn: '234567'
          title: Harry Potter and the Chamber of Secrets

Gives this error:

2017-11-16 20:15:17,978 [ERROR]: (tavern.schemas.files:38) Error validating /var/folders/6p/d1m45dr1031800ldbk945jrc0000gn/T/tmpes45ny99.yaml
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/tavern/schemas/files.py", line 36, in verify_generic
    verifier.validate()
  File "/usr/local/lib/python3.6/site-packages/pykwalify/core.py", line 176, in validate
    error_msg=u'.\n - '.join(self.validation_errors)))
pykwalify.errors.SchemaError: <SchemaError: error code 2: Schema validation failed:
 - Value '[{'author': 'Dan Brown', 'isbn': '123456', 'title': 'Digital Fortress'}, {'author': 'JK Rowling', 'isbn': '234567', 'title': 'Harry Potter and the Chamber of Secrets'}]' is not a dict. Value path: '/stages/0/response/body'.: Path: '/'>