hpenetworking / topology_docker Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 15.0 3.37 MB

Docker based Platform Engine plugin for the Topology Modular Framework

Home Page: https://topology-docker.readthedocs.io/

License: Apache License 2.0

Python 98.58% Dockerfile 1.42%

topology_docker's People

Contributors

Stargazers

Watchers

Forkers

josedvq carlos-jenkins saenzpa agustin-meneses-fuentes ramesh-subramaniam laurent-albert taigin sergiosmcr deepgreenantennas etcalvo rodyce

topology_docker's Issues

Improve prompt regular expressions to avoid false possitives

Currently the overly simplified prompt regular expressions for all nodes have made possible that the prompt matches several lines, causing confusion with the output a some commands.

Building topology fails when there are dirty docker images

When building a topology got the following error:

2016-09-05 18:32:41,494:::CRITICAL:::Build failed at stage "add_node" with "argument of type 'NoneType' is not iterable". Calling plugin rollback routine...
Traceback (most recent call last):
  File "/home/diazdavi/Documents/testing/topology_docker/.tox/py34/bin/topology", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/home/diazdavi/Documents/testing/topology_docker/.tox/py34/src/topology/bin/topology", line 35, in <module>
    sys.exit(main(args))
  File "/home/diazdavi/Documents/testing/topology_docker/.tox/py34/src/topology/lib/topology/main.py", line 87, in main
    mgr.build()
  File "/home/diazdavi/Documents/testing/topology_docker/.tox/py34/src/topology/lib/topology/manager.py", line 239, in build
    enode = self._platform.add_node(node)
  File "/home/diazdavi/Documents/testing/topology_docker/.tox/py34/lib/python3.4/site-packages/topology_docker/platform.py", line 79, in add_node
    node.identifier, **node.metadata
  File "/home/diazdavi/Documents/testing/topology_docker/.tox/py34/lib/python3.4/site-packages/topology_docker/nodes/host.py", line 41, in __init__
    super(HostNode, self).__init__(identifier, image=image, **kwargs)
  File "/home/diazdavi/Documents/testing/topology_docker/.tox/py34/lib/python3.4/site-packages/topology_docker/node.py", line 131, in __init__
    self._autopull()
  File "/home/diazdavi/Documents/testing/topology_docker/.tox/py34/lib/python3.4/site-packages/topology_docker/node.py", line 216, in _autopull
    if self._image in tags:
TypeError: argument of type 'NoneType' is not iterable

Somehow docker images can return None on RepoTags

command fails with TIMEOUT error

Hi,

I am trying to execute a command on a switch using migrated API’s of the new test framework. This command involves “ip netns exec swns” and “echo” with the pipe ( “|” ) command. My test execution hangs in the middle and it errors with TIMEOUT.

The actual command that I am trying to run is:

echo_result = sw2(‘echo \'{"id":"1234", "method":"echo", "params":["hi123"]}\' | ip netns exec swns nc 10.0.10.2 6622’, shell='bash')

The expected output of this command execution is: {"id":"1234","result":["hi123"],"error":null}

Corresponding code snippet for this command is:

        echo_cmd = 'echo \'{"id":"1234", "method":"echo", "params":["hi123"]}\''
        echo_cmd += ' | ip netns exec swns nc 10.0.10.2 6622'
        print("echo cmd = %s" % echo_cmd)
        echo_result = sw2(echo_cmd, shell='bash')

If I run the same command manually on the switch’s bash shell then it works fine. But with the test framework my script hangs and eventually it fails. One important observation shows that the error message back trace (call stack) shows that the expected output for this command is available at the buffer which is hold by the test framework but it could not pass it to the output. These are the error message lines which holds the expected output.

E           searcher: None
E           buffer (last 100 chars): b'ip ne \rtns exec swns nc 10.0.10.2 6622\r\n{"id":"1234","result":["hi123"],"error":null}root@switch:~# '
E           before (last 100 chars): b'ip ne \rtns exec swns nc 10.0.10.2 6622\r\n{"id":"1234","result":["hi123"],"error":null}root@switch:~# '
E           after: <class 'pexpect.exceptions.TIMEOUT'>

I think this might be the issue with test framework. The detailed output of the command is shown as below and I have attached my test script.

Thank you,
Hitesh Padekar
test_echo_two_sws.zip
.

padekar@hpnsw3660:/ws/padekar/tests/openswitch_tests/test$ tox -- --topology-platform ostl --ostl-destroy false --ostl-rsvnid 63594690 -k test_echo_two_sws -s -v
doc installed: You are using pip version 8.1.0, however version 8.1.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,alabaster==0.7.7,apipkg==1.4,autoapi==1.2.2,Babel==2.2.0,decorator==4.0.9,docker-py==1.7.2,docutils==0.12,execnet==1.4.1,flake8==2.5.4,ipaddress==1.0.16,ipdb==0.9.0,ipython==4.1.2,ipython-genutils==0.1.0,Jinja2==2.8,MarkupSafe==0.23,mccabe==0.4.0,path.py==8.1.2,pep8==1.7.0,pep8-naming==0.3.3,pexpect==4.0.1,pickleshare==0.6,ptyprocess==0.5.1,py==1.4.31,pyflakes==1.0.0,Pygments==2.1.3,pynml==1.0.0,pyparsing==2.1.0,pytest==2.8.4,pytest-random==0.2,pytest-timeout==1.0.0,pytest-xdist==1.14,pytz==2016.1,requests==2.9.1,rfc3986==0.3.1,simplegeneric==0.8.1,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.3.6,sphinx-rtd-theme==0.1.9,sphinxcontrib-plantuml==0.7,topology==1.5.0,topology-docker==1.5.0,-e git+https://github.com/HPENetworking/topology_docker_openswitch.git@86ff05d21543e01a6779de6a74c1b3a93c7187cc#egg=topology_docker_openswitch,-e git+https://github.com/HPENetworking/topology_lib_ip.git@e109d1a90f3f36be0e3e0e7ac1646e2149840478#egg=topology_lib_ip,-e git+https://github.com/HPENetworking/topology_lib_iperf.git@972c52fe858a212d48b7fc18811e673377dc8e34#egg=topology_lib_iperf,-e [email protected]:psaenz/topology_lib_ixia.git@609860c035a8b9f8b06fd26c1f3a8310b83ab47b#egg=topology_lib_ixia,-e git+https://github.com/HPENetworking/topology_lib_ping.git@e967c36203ecfba41bedf4680fee9289e25191b3#egg=topology_lib_ping,-e git+https://github.com/HPENetworking/topology_lib_sflowtool.git@7011d17be53e6cf29b79915459c60c45e3e93671#egg=topology_lib_sflowtool,-e git+https://github.com/HPENetworking/topology_lib_tcpdump.git@073a6ba6b945c2c40ea58c5eb05874141c884ff4#egg=topology_lib_tcpdump,-e git+https://git.openswitch.net/openswitch/ops-topology-lib-vtysh@d3c3119b459e0730adc19d48562f1691d5b49642#egg=topology_lib_vtysh,-e [email protected]:hpe-networking/topology_ostl.git@10c0be897d176ddf9c8284ad63a3e2c03dd5e4e6#egg=topology_ostl,traitlets==4.2.1,websocket-client==0.35.0
doc runtests: PYTHONHASHSEED='58734270'
doc runtests: commands[0] | ln -s /ws/padekar/tests/openswitch_tests/test /ws/padekar/tests/openswitch_tests/.tox/doc/tmp/testsuite
doc runtests: commands[1] | sphinx-build -W -b html -d doctrees /ws/padekar/tests/openswitch_tests/doc/ html
Running Sphinx v1.3.6
making output directory...
loading pickled environment... not yet created
loading intersphinx inventory from https://docs.python.org/3.4/objects.inv...

building [mo]: targets for 0 po files that are out of date
building [html]: targets for 8 source files that are out of date
updating environment: 8 added, 0 changed, 0 removed
reading sources... [ 12%] autoapi/testsuite


reading sources... [ 25%] autoapi/testsuite.helpers

reading sources... [ 37%] autoapi/topology_lib_ip.library


reading sources... [ 50%] autoapi/topology_lib_vtysh.library
reading sources... [ 62%] autoapi/topology_lib_vtysh.parser
reading sources... [ 75%] index
reading sources... [ 87%] migration_oldapi
reading sources... [100%] migration_vsi

looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [ 12%] autoapi/testsuite
writing output... [ 25%] autoapi/testsuite.helpers
writing output... [ 37%] autoapi/topology_lib_ip.library
writing output... [ 50%] autoapi/topology_lib_vtysh.library
writing output... [ 62%] autoapi/topology_lib_vtysh.parser
writing output... [ 75%] index
writing output... [ 87%] migration_oldapi
writing output... [100%] migration_vsi

generating indices... genindex py-modindex
writing additional pages... search
copying images... [100%] _static/images/logo.png

copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded.
py34 installed: You are using pip version 8.1.0, however version 8.1.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,apipkg==1.4,decorator==4.0.9,docker-py==1.7.2,execnet==1.4.1,flake8==2.5.4,ipaddress==1.0.16,ipdb==0.9.0,ipython==4.1.2,ipython-genutils==0.1.0,mccabe==0.4.0,path.py==8.1.2,pep8==1.7.0,pep8-naming==0.3.3,pexpect==4.0.1,pickleshare==0.6,ptyprocess==0.5.1,py==1.4.31,pyflakes==1.0.0,pynml==1.0.0,pyparsing==2.1.0,pytest==2.8.4,pytest-random==0.2,pytest-timeout==1.0.0,pytest-xdist==1.14,requests==2.9.1,rfc3986==0.3.1,simplegeneric==0.8.1,six==1.10.0,topology==1.5.0,topology-docker==1.5.0,-e git+https://github.com/HPENetworking/topology_docker_openswitch.git@86ff05d21543e01a6779de6a74c1b3a93c7187cc#egg=topology_docker_openswitch,-e git+https://github.com/HPENetworking/topology_lib_ip.git@e109d1a90f3f36be0e3e0e7ac1646e2149840478#egg=topology_lib_ip,-e git+https://github.com/HPENetworking/topology_lib_iperf.git@972c52fe858a212d48b7fc18811e673377dc8e34#egg=topology_lib_iperf,-e [email protected]:psaenz/topology_lib_ixia.git@609860c035a8b9f8b06fd26c1f3a8310b83ab47b#egg=topology_lib_ixia,-e git+https://github.com/HPENetworking/topology_lib_ping.git@e967c36203ecfba41bedf4680fee9289e25191b3#egg=topology_lib_ping,-e git+https://github.com/HPENetworking/topology_lib_sflowtool.git@7011d17be53e6cf29b79915459c60c45e3e93671#egg=topology_lib_sflowtool,-e git+https://github.com/HPENetworking/topology_lib_tcpdump.git@073a6ba6b945c2c40ea58c5eb05874141c884ff4#egg=topology_lib_tcpdump,-e git+https://git.openswitch.net/openswitch/ops-topology-lib-vtysh@d3c3119b459e0730adc19d48562f1691d5b49642#egg=topology_lib_vtysh,-e [email protected]:hpe-networking/topology_ostl.git@10c0be897d176ddf9c8284ad63a3e2c03dd5e4e6#egg=topology_ostl,traitlets==4.2.1,websocket-client==0.35.0
py34 runtests: PYTHONHASHSEED='58734270'
py34 runtests: commands[0] | flake8 /ws/padekar/tests/openswitch_tests/test
py34 runtests: commands[1] | py.test -v --random --durations=10 --junitxml=tests.xml --topology-nml-dir topologies --topology-plot-dir topologies --topology-plot-format svg --topology-platform ostl --ostl-destroy false --ostl-rsvnid 63594690 -k test_echo_two_sws -s -v /ws/padekar/tests/openswitch_tests/test
========================================= test session starts =========================================
platform linux -- Python 3.4.3, pytest-2.8.4, py-1.4.31, pluggy-0.3.1 -- /ws/padekar/tests/openswitch_tests/.tox/py34/bin/python3.4
cachedir: ../../../.cache
Tests are shuffled using seed number 373555060484.
topology: platform='ostl'
          plot_dir='/ws/padekar/tests/openswitch_tests/.tox/py34/tmp/topologies' (svg)
          nml_dir='/ws/padekar/tests/openswitch_tests/.tox/py34/tmp/topologies'
rootdir: /ws/padekar/tests/openswitch_tests, inifile: tox.ini
plugins: topology-ostl-0.1.0, xdist-1.14, random-0.2, topology-1.5.0, timeout-1.0.0
collected 10 items 

../../../test/test_echo_two_sws.py::test_echo_two_sw 2016-03-28T14:11:05.874559 [sw1].send_command('conf ter', shell='vtysh') ::

2016-03-28T14:11:06.486437 [sw1].send_command('interface 1', shell='vtysh') ::

2016-03-28T14:11:06.605897 [sw1].send_command('routing', shell='vtysh') ::

2016-03-28T14:11:06.725625 [sw1].send_command('ip address 10.0.10.2/24', shell='vtysh') ::
IP address is already assigned to interface 1 as primary.
2016-03-28T14:11:06.851110 [sw1].send_command('no shut', shell='vtysh') ::

2016-03-28T14:11:06.971038 [sw1].send_command('end', shell='vtysh') ::

2016-03-28T14:11:07.292900 [sw2].send_command('conf ter', shell='vtysh') ::

2016-03-28T14:11:08.532320 [sw2].send_command('interface 1', shell='vtysh') ::

2016-03-28T14:11:08.649107 [sw2].send_command('routing', shell='vtysh') ::

2016-03-28T14:11:08.769509 [sw2].send_command('ip address 10.0.10.3/24', shell='vtysh') ::
IP address is already assigned to interface 1 as primary.
2016-03-28T14:11:08.894992 [sw2].send_command('no shut', shell='vtysh') ::

2016-03-28T14:11:09.015318 [sw2].send_command('end', shell='vtysh') ::

2016-03-28T14:11:10.137564 [sw1].send_command('ip netns exec swns ovs-appctl -v -t ovsdb-server ovsdb-server/add-remote ptcp:6622', shell='bash') ::
ip netns exec swns ovs-appctl -v -t ovsdb-server ovsdb-server/add -remote ptcp:6622
2016-03-23T01:33:39Z|00001|jsonrpc|DBG|unix:/var/run/openvswitch/ovsdb-server.274.ctl: send request, method="ovsdb-server/add-remote", params=["ptcp:6622"], id=0
2016-03-23T01:33:39Z|00002|poll_loop|DBG|wakeup due to 0-ms timeout
2016-03-23T01:33:39Z|00003|jsonrpc|DBG|unix:/var/run/openvswitch/ovsdb-server.274.ctl: received reply, result="", id=0
OVSDB cmd result = ip netns exec swns ovs-appctl -v -t ovsdb-server ovsdb-server/add -remote ptcp:6622
2016-03-23T01:33:39Z|00001|jsonrpc|DBG|unix:/var/run/openvswitch/ovsdb-server.274.ctl: send request, method="ovsdb-server/add-remote", params=["ptcp:6622"], id=0
2016-03-23T01:33:39Z|00002|poll_loop|DBG|wakeup due to 0-ms timeout
2016-03-23T01:33:39Z|00003|jsonrpc|DBG|unix:/var/run/openvswitch/ovsdb-server.274.ctl: received reply, result="", id=0
>>> [001] :: test_echo_two_sw:124
... opened a port on sw1
echo cmd = echo '{"id":"1234", "method":"echo", "params":["hi123"]}' | ip netns exec swns nc 10.0.10.2 6622
2016-03-28T14:11:11.083170 [sw2].send_command('echo '{"id":"1234", "method":"echo", "params":["hi123"]}' | ip netns exec swns nc 10.0.10.2 6622', shell='bash') ::
FAILED

============================================== FAILURES ===============================================
__________________________________________ test_echo_two_sw ___________________________________________

topology = <topology.manager.TopologyManager object at 0x7fad156755f8>
step = <topology.pytest.plugin.StepLogger object at 0x7fad153f37f0>

    def test_echo_two_sw(topology, step):
        """
        Set network addresses and static routes between nodes and ping h2 from h1.
        """
        sw1 = topology.get('sw1')
        sw2 = topology.get('sw2')

        assert sw1 is not None
        assert sw2 is not None

        # Configure IP and bring UP switch 1 interfaces
        sw1('conf ter', shell='vtysh')
        sw1('interface 1', shell='vtysh')
        sw1('routing', shell='vtysh')
        sw1('ip address 10.0.10.2/24', shell='vtysh')
        sw1('no shut', shell='vtysh')
        sw1('end', shell='vtysh')

        sw2('conf ter', shell='vtysh')
        sw2('interface 1', shell='vtysh')
        sw2('routing', shell='vtysh')
        sw2('ip address 10.0.10.3/24', shell='vtysh')
        sw2('no shut', shell='vtysh')
        sw2('end', shell='vtysh')

        sleep(1)

        # Open passive TCP port on OVSDB-Server for listening connections
        cmd = 'ip netns exec swns ovs-appctl -v -t ovsdb-server '
        cmd += 'ovsdb-server/add-remote ptcp:6622'
        result = sw1(cmd, shell='bash')
        print('OVSDB cmd result = %s' % result)

        step('opened a port on sw1')

        echo_cmd = 'echo \'{"id":"1234", "method":"echo", "params":["hi123"]}\''
        echo_cmd += ' | ip netns exec swns nc 10.0.10.2 6622'
        print("echo cmd = %s" % echo_cmd)
>       echo_result = sw2(echo_cmd, shell='bash')

../../../test/test_echo_two_sws.py:129: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../lib/python3.4/site-packages/topology/platforms/base.py:212: in __call__
    return self.send_command(cmd, shell=shell, silent=silent)
../lib/python3.4/site-packages/topology/platforms/base.py:390: in send_command
    response = self._shells[shell](cmd)
../src/topology-ostl/lib/topology_ostl/nodes.py:167: in bash
    result = self._telnet(command, r'\r\nroot@[-\w]+:~#')
../src/topology-ostl/lib/topology_ostl/shell.py:72: in __call__
    self.send_command(command, prompt)
../src/topology-ostl/lib/topology_ostl/shell.py:113: in send_command
    self._child.expect(prompt, timeout=self._timeout)
../lib/python3.4/site-packages/pexpect/spawnbase.py:315: in expect
    timeout, searchwindowsize, async)
../lib/python3.4/site-packages/pexpect/spawnbase.py:339: in expect_list
    return exp.expect_loop(timeout)
../lib/python3.4/site-packages/pexpect/expect.py:104: in expect_loop
    return self.timeout(e)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pexpect.expect.Expecter object at 0x7fad15675908>, err = TIMEOUT('Timeout exceeded.',)

    def timeout(self, err=None):
        spawn = self.spawn
        from . import TIMEOUT

        spawn.before = spawn.buffer
        spawn.after = TIMEOUT
        index = self.searcher.timeout_index
        if index >= 0:
            spawn.match = TIMEOUT
            spawn.match_index = index
            return index
        else:
            spawn.match = None
            spawn.match_index = None
            msg = str(spawn)
            if err is not None:
                msg = str(err) + '\n' + msg
>           raise TIMEOUT(msg)
E           pexpect.exceptions.TIMEOUT: Timeout exceeded.
E           <pexpect.pty_spawn.spawn object at 0x7fad153f32b0>
E           command: /usr/bin/telnet
E           args: ['/usr/bin/telnet', '16.93.48.133', '2007']
E           searcher: None
E           buffer (last 100 chars): b'ip ne \rtns exec swns nc 10.0.10.2 6622\r\n{"id":"1234","result":["hi123"],"error":null}root@switch:~# '
E           before (last 100 chars): b'ip ne \rtns exec swns nc 10.0.10.2 6622\r\n{"id":"1234","result":["hi123"],"error":null}root@switch:~# '
E           after: <class 'pexpect.exceptions.TIMEOUT'>
E           match: None
E           match_index: None
E           exitstatus: None
E           flag_eof: False
E           pid: 18004
E           child_fd: 8
E           closed: False
E           timeout: 30
E           delimiter: <class 'pexpect.exceptions.EOF'>
E           logfile: None
E           logfile_read: None
E           logfile_send: None
E           maxread: 2000
E           ignorecase: False
E           searchwindowsize: None
E           delaybeforesend: 0.05
E           delayafterclose: 0.1
E           delayafterterminate: 0.1

../lib/python3.4/site-packages/pexpect/expect.py:68: TIMEOUT
----------- generated xml file: /ws/padekar/tests/openswitch_tests/.tox/py34/tmp/tests.xml ------------
====================================== slowest 10 test durations ======================================
35.46s call     test/test_echo_two_sws.py::test_echo_two_sw
10.73s setup    test/test_echo_two_sws.py::test_echo_two_sw
0.15s teardown test/test_echo_two_sws.py::test_echo_two_sw
============================= 9 tests deselected by '-ktest_echo_two_sws' =============================
=============================== 1 failed, 9 deselected in 46.47 seconds ===============================
ERROR: InvocationError: '/ws/padekar/tests/openswitch_tests/.tox/py34/bin/py.test -v --random --durations=10 --junitxml=tests.xml --topology-nml-dir topologies --topology-plot-dir topologies --topology-plot-format svg --topology-platform ostl --ostl-destroy false --ostl-rsvnid 63594690 -k test_echo_two_sws -s -v /ws/padekar/tests/openswitch_tests/test'
_______________________________________________ summary _______________________________________________
  doc: commands succeeded
ERROR:   py34: commands failed
padekar@hpnsw3660:/ws/padekar/tests/openswitch_tests/test$ 
padekar@hpnsw3660:/ws/padekar/tests/openswitch_tests/test$

hostname command in switches

When try to execute the following command

sw1('hostname CT-TEST')

The test failed, because I am changing the prompt.

Speed up topology unbuild

Currently the topology unbuild is implemented as a sequential node destruction.

Node destruction is implemented as a destroy, remove, and wait sequence.

We can speed up the unbuild by calling the destroy and remove of all nodes first, then wait until all of them are removed.

This can implement this as threads (consider GIL), or refactor the workflow. We also need to consider the the hooks interfaces defined by the core framework.

Migrate Shells object to use new Shell low-level API.

A new Shell low-level API was merged in the base topology framework in HPENetworking/topology@06f9d98

This platform engine should migrate it's shell objects to use this new API.

Concurrent POST networks/create results in "Another app is currently holding the xtables lock"

check moby/moby#26111

This is seen when multiple topologies are build concurrently

Update node API

This platform engine uses a differentiated API for nodes by defining the pause() and unpause() methods.

The Topology base API defines since version 1.0 (e24ee6c2d2858af9f610c891b89e19742b2fc9c5) the enabled() and disable() methods.

This platform engine must conform the base API and implement those methods. A simple renaming of the methods should be sufficient, and updating the documentation off course.

Biport identifiers should be checked for platform-wide uniqueness

As explained on issue #44, adding the a new biport with a repeated identifier (at the platform level) will overwrite the old biport that used that same identifier.

The problem is that one would intuitively think that biport identifiers only need to be unique at the node-level (or even pair-of-nodes level), not at the whole platform level.

Given that the current behavior seems to be the way we want to go with, we should at least raise an error if an user tries to add an already existing biport identifier.

Specifying image in the test file causes unexpected behaviour

Hello,
If I specify the name of the Docker image to be used such as the following example:

[type=openswitch name="OpenSwitch 1"] ops1
[type=host name="Host 1" image="openswitch/genericx86-64"] hs1
[type=host name="Host 2"] hs2

The behaviour it causes is unexpected. WITHOUT this property the tests run just fine. The following is the stack trace and error message:

========================================================================================================================================================================================= ERRORS ==========================================================================================================================================================================================
_______________________________________________________________________________________________________________________________________________________________________________ ERROR at setup of test_vlan _______________________________________________________________________________________________________________________________________________________________________________
[gw1] linux -- Python 3.4.3 /ws/cambronw/openswitch_tests/.tox/py34/bin/python3.4
Error building topogy in module test_vlan:
Traceback (most recent call last):
  File "/ws/cambronw/openswitch_tests/.tox/py34/lib/python3.4/site-packages/docker/client.py", line 138, in _raise_for_status
    response.raise_for_status()
  File "/ws/cambronw/openswitch_tests/.tox/py34/lib/python3.4/site-packages/requests/models.py", line 840, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localunixsocket/v1.21/images/openswitch%2Fgenericx86-64%3Alatest/tag?repo=openswitch%2Fgenericx86-64&tag=latest&force=0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ws/cambronw/openswitch_tests/.tox/py34/lib/python3.4/site-packages/topology/pytest/plugin.py", line 157, in topology
    topomgr.build()
  File "/ws/cambronw/openswitch_tests/.tox/py34/lib/python3.4/site-packages/topology/manager.py", line 296, in build
    raise e
  File "/ws/cambronw/openswitch_tests/.tox/py34/lib/python3.4/site-packages/topology/manager.py", line 238, in build
    enode = self._platform.add_node(node)
  File "/ws/cambronw/openswitch_tests/.tox/py34/lib/python3.4/site-packages/topology_docker/platform.py", line 77, in add_node
    node.identifier, **node.metadata
  File "/ws/cambronw/openswitch_tests/.tox/py34/lib/python3.4/site-packages/topology_docker/nodes/host.py", line 41, in __init__
    super(HostNode, self).__init__(identifier, image=image, **kwargs)
  File "/ws/cambronw/openswitch_tests/.tox/py34/lib/python3.4/site-packages/topology_docker/node.py", line 77, in __init__
    self._autopull()
  File "/ws/cambronw/openswitch_tests/.tox/py34/lib/python3.4/site-packages/topology_docker/node.py", line 133, in _autopull
    if not self._client.tag(pull_name, image, tag):
  File "/ws/cambronw/openswitch_tests/.tox/py34/lib/python3.4/site-packages/docker/utils/decorators.py", line 21, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/ws/cambronw/openswitch_tests/.tox/py34/lib/python3.4/site-packages/docker/api/image.py", line 269, in tag
    self._raise_for_status(res)
  File "/ws/cambronw/openswitch_tests/.tox/py34/lib/python3.4/site-packages/docker/client.py", line 142, in _raise_for_status
    raise errors.APIError(e, response, explanation=explanation)
docker.errors.APIError: 409 Client Error: Conflict ("b'Conflict: Tag openswitch/genericx86-64:latest is already set to image d3db7be415c0, if you want to replace it, please use -f option'")

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured stderr setup ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Build failed at stage "add_node" with "409 Client Error: Conflict ("b'Conflict: Tag openswitch/genericx86-64:latest is already set to image d3db7be415c0, if you want to replace it, please use -f option'")". Calling plugin rollback routine...
-------------------------------------------------------------------------------------------------------------------------------------------------------- generated xml file: /ws/cambronw/openswitch_tests/.tox/py34/tmp/tests.xml --------------------------------------------------------------------------------------------------------------------------------------------------------
================================================================================================================================================================================ slowest 10 test durations ================================================================================================================================================================================
15.51s setup    test_vlan.py::test_vlan
0.00s teardown test_vlan.py::test_vlan
================================================================================================================================================================================ 1 error in 16.67 seconds =================================================================================================================================================================================
ERROR: InvocationError: '/ws/cambronw/openswitch_tests/.tox/py34/bin/py.test --random --durations=10 --junitxml=tests.xml --topology-nml-dir topologies --topology-plot-dir topologies --topology-plot-format svg --topology-platform docker -n=auto -v /ws/cambronw/openswitch_tests/test'
_________________________________________________________________________________________________________________________________________________________________________________________ summary _________________________________________________________________________________________________________________________________________________________________________________________
ERROR:   py34: commands failed

Place node-specific code in separate repos

The implementation of nodes for this platform engine resides in this very same repo, they should be moved to completely separate the node development from this.

Host node interfaces with numeric name

When running a test case using an interface of a host node type named with only numeric characters (e.g.: 1), it fails when trying to access that interface for configuration. For example:

Given the following node:

[type=host name="Host 1"] hs1
hs1:1

When test tries to configure the interface,

hs1("ip link set 1 up")

The following error appears:

Cannot find device "1"

Although the interface is created correctly under the netns front_panel (Already verified with ip link show), the configuration fails with the message mentioned before.

Add support to services API

In upcoming Topology 1.8 a new API for the nodes will be implemented: the services API.

The Docker platform engine for Topology must implement the required interfaces on the base node if required.

The openswitch node should stop vtysh gracefully

It looks like today after having processed all vtysh commands, the vtysh session isn't ended properly (i.e with 'exit').

I am trying to measure code coverage (gcov) and need the vtysh binary to gracefully end after having executed all my tests.

Release 1.7

@carlos-jenkins seems like we are good to go for 1.7!

Handle interfaces in host node with legacy compatibility

When using interfaces in a host it is now a requirement to put the "bash_front_panel" instead of just bash. Is there a way we can keep using the following?
hs1("ip link set dev eth0 up")
Instead of:
hs1("ip link set dev eth0 up", shell="bash_front_panel")

And, in the background the framework will do the necessary to work with the front_panel netns.

This will prevent to change hundreds of current tests in production, and is very important not to do any modifications on those.

Docker-py update conflicts with requests

Latest docker-py requires the requests module on a version before 2.11

pkg_resources.ContextualVersionConflict: (requests 2.11.1 (...), Requirement.parse('requests<2.11,>=2.5.2'), {'docker-py'})

Remove imports from deprecated module

As explained in:

HPENetworking/topology#25

The module topology.platforms.base is now deprecated. Need to change to imports as explained in the PR.

Platform-level non-unique biport identifiers

On line 130 of platform.py, we use the biport's identifier as the key to add it to the platform's nmlbiport_iface_map:

self.nmlbiport_iface_map[biport.identifier] = {

Which means that a newer biport with the same identifier as one that's been already added to the platform will effectively replace it in the nmlbiport_iface_map. Because of this, one would think that having non-unique biport identifiers on the same platform shouldn't be supported, but on lines 185 through 193 of test/test_topology_docker_platform.py we are defining two biports with identifiers '3' and '4', adding them to the platform, and then defining two different biports with those same identifiers and adding them to the platform too:

s1p1 = BidirectionalPort(identifier='3')
s1p2 = BidirectionalPort(identifier='4')
platform.add_biport(s1, s1p1)
platform.add_biport(s1, s1p2)

s2p1 = BidirectionalPort(identifier='3')
s2p2 = BidirectionalPort(identifier='4')
platform.add_biport(s2, s2p1)
platform.add_biport(s2, s2p2)

Which would make one think that this is indeed a supported scenario, so, are non-unique biport identifiers on the same platform supported? If yes, platform.py should be fixed, if no, the test should be fixed and we should probably add some kind of error if someone tries to add an already existing biport identifier to the platform.

Need the openswitch.py node to accept a configurable parameter to start docker with an extra volume

RIght now the openswitch.py node is hardcoded to use the following volumes:

openswitch.py,

         # Add binded directories
         binds = [
             '{}:/tmp'.format(shared_dir),
             '/dev/log:/dev/log',
             '/sys/fs/cgroup:/sys/fs/cgroup:ro'
         ]

I am working in adding code coverage support and as a requirement I need to pass an extra bind (volume) to Docker and it needs to be programatically (it cant be hardcoded).

Using an environment variable is an option. For example, before calling tox I'd like to set the following variable

export VSI_COV_DATA_DIR=/some/location/for/coverage

and then after calling tox, this should result in something similar to:

         # Add binded directories
         binds = [
             '{}:/tmp'.format(shared_dir),
             '/dev/log:/dev/log',
             '/sys/fs/cgroup:/sys/fs/cgroup:ro'
             '/some/location/for/coverage:/some/location/for/coverage'
         ]

It must be read/write.

Thanks

topology_docker won't pull from private docker registry

It looks like when a private registry is specified and no local image is found, docker-py client fails to pull the image from the private registry:

File "/usr/local/lib/python3.4/site-packages/topology_docker/node.py", line 133, in __init__
    self._autopull()
  File "/usr/local/lib/python3.4/site-packages/topology_docker/node.py", line 322, in _autopull
    raise Exception(status['error'])
Exception: Error: image xxxxx/yyyyy:latest not found

Notice I am using xxxxx/yyyyy instead of the actual image name in the example above.

Support for nodes management interfaces (oobm network with participating host)

This was reported by Wassermann, Gary [email protected]

I’m trying to do some interactive testing with docker switch images. To do that, I used test_ping.py to create a topology with two switches and two workstations. I used the debugger to pause the test and manually interact with the switches. What I would like to do now is use the management interface to copy files from my VM to the running switches, but I can’t find the management interface. On docker switch instances that I create through my normal manual process, the management (oobm) interface is eth0. From the bash shell, if I run “ifconfig –a”, I only see lo. If I run “ip netns exec swns ifconfig –a”, I see many interfaces including ovs-netdev and bridge_normal, but I don’t think those are what I’m looking for. I ran “dmesg | grep eth0” and the output looked essentially the same as on my docker switch instance that does have eth0.

Vtysh ``speed ?`` command makes the shell out of sync

When I send the command speed ?, the following commands are delayed, so I try to send an enter to do that it works, but it is not working correctly. This is an output of the issue:

2016-02-19T14:41:02.737199 [sw1].send_command(interface 1) ::

2016-02-19T14:41:02.843174 [sw1].send_command(speed ?) ::
speed 
  1000   Mb/s supported
  10000  Mb/s not supported
  40000  Mb/s not supported
  auto   Auto negotiate speed (Default)
2016-02-19T14:41:02.944841 [sw1].send_command( ) ::
speed 
% Command incomplete.
2016-02-19T14:41:03.047972 [sw1].send_command(set interface 1 hw_intf_info:speeds="1000,10000") ::

2016-02-19T14:41:10.797321 [sw1].send_command(speed ?) ::

2016-02-19T14:41:10.898454 [sw1].send_command( ) ::
speed 
  1000   Mb/s supported
  10000  Mb/s supported
  40000  Mb/s not supported
  auto   Auto negotiate speed (Default)
FAILED

The FAILED is because I trying to do an assert to speed ? command, but it is empty, as you can see.

Add a differed topology unbuild in pytest plugin

Currently, the unbuild of a topology is linear. Before a new suite starts, the previous needs to be unbuild. Depending of the topology being built, the unbuild step can be a large part of the time spent executing the suite.

Given the fact that topology_docker can run multiple suites in parallel, we can differ the unbuild of the topologies until some heuristic or event happen. In this way, the data of the test execution is available sooner.

If this mode is enabled, the unbuild will be differed to a specific "garbage collection" subprocess that will trigger the stop and removal of all the containers upon request or activation by an event or heuristic.

At the implementation details, the plugin can submit to the garbage collector all the topology objects to be discarted. The garbage collector process will query from time to time for resources, and decide to kick in if some parameter of disk space, number of containers, ram or processing is matched.

As part of this optimization, it is important to consider that the current implementation of the stop command is blocking:

https://github.com/HPENetworking/topology_docker/blob/master/lib/topology_docker/node.py#L188
https://github.com/HPENetworking/topology_docker/blob/master/lib/topology_docker/platform.py#L211

An optimization could be that once the topology is submitted to the garbage collector, the garbage collector subprocess request stop of all the containers, but do not wait or block until done. Once the event trigger the removal, the garbage collector will then wait if not stopped and then remove them.
Wait and removal of the container can be done in different threads (consider the GIL), and until all of them finish then the garbage collector will be done.