ansible / pylibssh Goto Github PK

View Code? Open in Web Editor NEW

59.0 41.0 28.0 84.92 MB

Python bindings specific to Ansible use case for libssh https://www.libssh.org/

Home Page: https://ansible-pylibssh.rtfd.io

License: GNU Lesser General Public License v2.1

Python 33.10% Shell 15.59% Dockerfile 0.78% Cython 50.53%

pylibssh's Introduction

https://img.shields.io/badge/license-LGPL+-blue.svg?maxAge=3600

pylibssh: Python bindings to client functionality of libssh specific to Ansible use case

Nightlies @ Dumb PyPI @ GitHub Pages

We publish nightlies on tags and pushes to devel. They are hosted on a GitHub Pages based index generated by dumb-pypi.

The web view is @ https://ansible.github.io/pylibssh/.

$ pip install \
    --extra-index-url=https://ansible.github.io/pylibssh/simple/ \
    --pre \
    ansible-pylibssh

Requirements

You need Python 3.6+

pylibssh requires libssh to be installed in particular:

libssh version 0.9.0 and later.

To install libssh refer to its Downloads page.

Building the module

In the local env, assumes there's a libssh shared library on the system, build toolchain is present and env vars are set properly:

$ git clone https://github.com/ansible/pylibssh.git
$ cd pylibssh
$ pip install tox
$ tox -e build-dists

manylinux-compatible wheels:

$ git clone https://github.com/ansible/pylibssh.git
$ cd pylibssh
$ pip install tox
$ tox -e build-dists-manylinux1-x86_64  # with Docker

# or with Podman
$ DOCKER_EXECUTABLE=podman tox -e build-dists-manylinux1-x86_64

# to enable shell script debug mode use
$ tox -e build-dists-manylinux1-x86_64 -- -e DEBUG=1

License

This library is distributed under the terms of LGPL 2 or higher, see file LICENSE.rst in this repository.

pylibssh's People

Contributors

Stargazers

Watchers

pylibssh's Issues

[TODO] Implement automatic test constraints pinning

Introduce constraints files
Integrate pip-tools
Integrate dependabot
Use hashes

Trove classifiers in dist metadata are invalid

If you try to publish a dist with trove classifiers (https://github.com/ansible/pylibssh/blob/362a7e3/setup.py#L22-L26) that aren't in the list https://pypi.org/classifiers/, it'll be rejected.

[TODO] Rewrite manylinux build script to use Python

(moved out of #66)

[TODO] Enable faulthandler in pytest

Facilitate inspecting the crashed tests:

https://docs.pytest.org/en/stable/usage.html#fault-handler

[TODO] Implement building/packaging and testing RPMs

Figure out what to target and the build env
Define test containers
Integrate into tox+GHA

SFTP issue with libssh

SUMMARY

sftp is not working with libssh on iosxr platform

ISSUE TYPE

Bug Report

PYLISSH and LIBSSH VERSION

(py3.6.10) amhatre@ashwinis-MacBook-Pro playbooks % pip show ansible-pylibssh
Name: ansible-pylibssh
Version: 0.2.0
Summary: Python bindings for libssh client specific to Ansible use case
Home-page: https://github.com/ansible/pylibssh
Author: Ansible, Inc.
Author-email: info+github/ansible/[email protected]
License: LGPLv2+
Location: /Users/amhatre/ansible_venvs/py3.6.10/lib/python3.6/site-packages
Requires: 
Required-by:

OS / ENVIRONMENT

IOSXR 6.1.3
IOSXR 7.0.2

STEPS TO REPRODUCE

- name: test iosxr with pylibssh
  hosts: iosxr
  connection: ansible.netcommon.network_cli
  gather_facts: no
  tasks:
    - name: replace config
      register: result
      cisco.iosxr.iosxr_config:
        src: 'running_cfg_iosxr.txt'
        replace: config

EXPECTED RESULTS

changed: [127.0.0.1] => {
    "changed": true,
    "commands": [
        "load harddisk:/ansible_config.txt"
    ],
    "diff": {
        "prepared": "Building configuration...\n!! IOS XR Configuration 7.0.2\n+  interface preconfigure GigabitEthernet0/0/0/0\n   !\n+  interface preconfigure GigabitEthernet0/0/0/1\n   !\n+  interface preconfigure GigabitEthernet0/0/0/2\n   !\nend"
    },
    "invocation": {
        "module_args": {
            "admin": false,
            "after": null,
            "backup": false,
            "backup_options": null,
            "before": null,
            "comment": "configured by iosxr_config",
            "config": null,
            "exclusive": false,
            "force": false,
            "label": null,
            "lines": null,
            "match": "line",
            "parents": null,
            "provider": null,
            "replace": "config",
            "src": "hostname test1\nbanner motd \"hell\"\ntelnet vrf default ipv4 server max-servers 10\nusername vagrant\n group root-lr\n group cisco-support\n secret 10 $6$Zf8Qze0rDkx1z...$/1ly7j2qqH5vYIpGEG2a8khtTdEEGS3NnbRv6izmWCLteCQSFt2YbBCVQOfETj4A0IPquaPj8yWEqH1vXck06/\n!\nusername amhatre\n group root-lr\n group cisco-support\n!\ntpa\n vrf default\n  address-family ipv4\n   update-source dataports MgmtEth0/RP0/CPU0/0\n  !\n !\n!\ncall-home\n service active\n contact smart-licensing\n profile CiscoTAC-1\n  active\n  destination transport-method http\n !\n!\nnetconf-yang agent\n ssh\n!\ninterface Loopback888\n!\ninterface Loopback999\n!\ninterface MgmtEth0/RP0/CPU0/0\n ipv4 address dhcp\n!\ninterface preconfigure GigabitEthernet0/0/0/0\n!\ninterface preconfigure GigabitEthernet0/0/0/1\n!\ninterface preconfigure GigabitEthernet0/0/0/2\n!\nprefix-set ebpg_filter\n  192.168.0.0/16 ge 15 le 30\nend-set\n!\nprefix-set ebpg_filter2\n  192.168.0.0/16 ge 17 le 30\nend-set\n!         \nrouter static\n address-family ipv4 unicast\n  0.0.0.0/0 MgmtEth0/RP0/CPU0/0 10.0.2.2\n !        \n!         \nrouter bgp 1\n!         \nssh server v2\nssh server vrf default\nssh server netconf vrf default\nend\n"
        }
    }
}
META: ran handlers
META: ran handlers

PLAY RECAP ***********************************************************************************************************************************************************************
127.0.0.1                  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

ACTUAL RESULTS

<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/amhatre/.ansible/tmp/ansible-local-34893sr5i_bwc/ansible-tmp-1622724615.099243-34897-263822561108764/ /Users/amhatre/.ansible/tmp/ansible-local-34893sr5i_bwc/ansible-tmp-1622724615.099243-34897-263822561108764/AnsiballZ_iosxr_config.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/amhatre/ansible_venvs/py3.6.10/bin/python /Users/amhatre/.ansible/tmp/ansible-local-34893sr5i_bwc/ansible-tmp-1622724615.099243-34897-263822561108764/AnsiballZ_iosxr_config.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/amhatre/.ansible/tmp/ansible-local-34893sr5i_bwc/ansible-tmp-1622724615.099243-34897-263822561108764/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
  File "/var/folders/j6/fqn4v5252b9_4ck6xxq4_rqm0000gn/T/ansible_cisco.iosxr.iosxr_config_payload_sjx_ttxa/ansible_cisco.iosxr.iosxr_config_payload.zip/ansible_collections/cisco/iosxr/plugins/module_utils/network/iosxr/iosxr.py", line 599, in copy_file
    conn.copy_file(source=src, destination=dst, proto=proto)
  File "/var/folders/j6/fqn4v5252b9_4ck6xxq4_rqm0000gn/T/ansible_cisco.iosxr.iosxr_config_payload_sjx_ttxa/ansible_cisco.iosxr.iosxr_config_payload.zip/ansible/module_utils/connection.py", line 195, in __rpc__
    raise ConnectionError(to_text(msg, errors='surrogate_then_replace'), code=code)
fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "admin": false,
            "after": null,
            "backup": false,
            "backup_options": null,
            "before": null,
            "comment": "configured by iosxr_config",
            "config": null,
            "exclusive": false,
            "force": false,
            "label": null,
            "lines": null,
            "match": "line",
            "parents": null,
            "provider": null,
            "replace": "config",
            "src": "hostname test1\nbanner motd \"hell\"\ntelnet vrf default ipv4 server max-servers 10\nusername vagrant\n group root-lr\n group cisco-support\n secret 10 $6$Zf8Qze0rDkx1z...$/1ly7j2qqH5vYIpGEG2a8khtTdEEGS3NnbRv6izmWCLteCQSFt2YbBCVQOfETj4A0IPquaPj8yWEqH1vXck06/\n!\nusername amhatre\n group root-lr\n group cisco-support\n!\ntpa\n vrf default\n  address-family ipv4\n   update-source dataports MgmtEth0/RP0/CPU0/0\n  !\n !\n!\ncall-home\n service active\n contact smart-licensing\n profile CiscoTAC-1\n  active\n  destination transport-method http\n !\n!\nnetconf-yang agent\n ssh\n!\ninterface Loopback888\n!\ninterface Loopback999\n!\ninterface MgmtEth0/RP0/CPU0/0\n ipv4 address dhcp\n!\ninterface preconfigure GigabitEthernet0/0/0/0\n!\ninterface preconfigure GigabitEthernet0/0/0/1\n!\ninterface preconfigure GigabitEthernet0/0/0/2\n!\nprefix-set ebpg_filter\n  192.168.0.0/16 ge 15 le 30\nend-set\n!\nprefix-set ebpg_filter2\n  192.168.0.0/16 ge 17 le 30\nend-set\n!         \nrouter static\n address-family ipv4 unicast\n  0.0.0.0/0 MgmtEth0/RP0/CPU0/0 10.0.2.2\n !        \n!         \nrouter bgp 1\n!         \nssh server v2\nssh server vrf default\nssh server netconf vrf default\nend\n"
        }
    },
    "msg": "Writing to remote file [b'/harddisk:/ansible_config.txt'] failed"
}

PLAY RECAP ***********************************************************************************************************************************************************************
127.0.0.1

[TODO] Document how development and testing works

How to run all the tests
How to build dists
How to make pytest select a specific test to run
How the dists are autopublished
How to make a pre-release
How to make a "stable" release
How coverage works
Publish things to RTD for convinience
How GHA is set up
How the linting works

[TODO] Migrate to `sphinxcontrib-towncrier` once it's published

[TODO] Build libssh under macOS instead of using brew

This would allow us to control the version of libssh (it's currently what brew ships and that's it — they remove old versions when adding new ones, hence no control at all).

Also, having control over the building allows us to single-source libssh version for manulinux+macos instead of hoping that they are the same.

command requiring privilege escalation for linux host not working.

Description:

On issuing privilege escalation command (sudo) the password prompt is not received on the channel, as a result, the become password is not written on channel and privilege escalation fails.

Test Snippet to reproduce the issue: libssh_priveledge_escalation.py

# libssh_priveledge_escalation.py:
import os

from pylibsshext.session import Session
from pylibsshext.errors import LibsshSessionException
from ansible.module_utils._text import to_bytes, to_native, to_text
ssh = Session()

HOST = "<changeme>"
USER = "<changeme>"
PASSWORD = "<changeme>"
BECOME_PASSWORD = "<changeme>"

try:
    ssh.connect(
        host=HOST,
        user=USER,
        password=PASSWORD,
        timeout=30,
        port=22
    )
except LibsshSessionException as ex:
    print(str(ex))

print(ssh.is_connected)


def exec_command(cmd, in_data=None, sudoable=True):
    ''' run a command on the remote host '''
    bufsize = 4096
    try:
        chan = ssh.new_channel()
    except Exception as e:
        text_e = to_text(e)
        msg = u"Failed to open session"
        if text_e:
            msg += u": %s" % text_e
        raise Exception(to_native(msg))

    cmd = to_text(cmd, errors='surrogate_or_strict')
    become_output = b''

    # sudo usually requires a PTY (cf. requiretty option), therefore
    # we give it one by default (pty=True in ansible.cfg), and we try
    # to initialise from the calling environment when sudoable is enabled
    chan.request_pty_size(terminal=to_bytes(os.getenv('TERM', 'vt100')), col=int(os.getenv('COLUMNS', 0)), row=int(os.getenv('LINES', 0)))

    try:
        count = chan.write(to_bytes(cmd))
        print(count)
        if 1:
            passprompt = False
            become_sucess = False
            while not (become_sucess or passprompt):
                print('Waiting for Privilege Escalation input')

                chan.poll(timeout=9000)

                chunk = chan.recv(bufsize)
                print("chunk is: %s" % to_native(chunk))
                if not chunk:
                    if b'unknown user' in become_output:
                        n_become_user = to_native("root")
                        raise Exception('user %s does not exist' % n_become_user)
                    else:
                        break
                become_output += chunk

                for l in become_output.splitlines(True):
                    print(l)
                    if 'password' in str(l).lower():
                        become_sucess = True
                if passprompt:
                    chan.sendall(b'%s' % BECOME_PASSWORD + b'\n')

    except Exception as e:
        text_e = to_text(e)
        msg = u"Failed to execute command"
        if text_e:
            msg += u": %s" % text_e
        raise Exception(to_native(msg))


exec_command("sudo")

Ref: ansible-collections/ansible.netcommon#165

GitHub doen't detect LGPL as this repo's license

And this is why: licensee/licensee#287 (comment)

Implement coverage reporting for Cython modules

SUMMARY

$sbj.

Another thing no think about is whether we are going to test builds with TRACE or w/o it, or both.

ISSUE TYPE

Feature Idea

ADDITIONAL INFORMATION

Refs:

Basically need to have plugins = Cython.Coverage in .coveragerc->[run], compile with CYTHON_TRACE=1 (or even CYTHON_TRACE_NOGIL=1) and # cython: linetrace=True to all *.pyx files.

[TODO] Replace `set-env` and `add-path` with env files in GHA

These are deprecated and will need to be replaced ASAP. Here's why: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/

Password authentication fails with Arista vEos

SUMMARY

ISSUE TYPE

Bug Report

PYLISSH and LIBSSH VERSION

devel

OS / ENVIRONMENT

STEPS TO REPRODUCE

clone repo and build the binaries
Refer: https://github.com/ansible/pylibssh#building-the-module

Add binary file PYTHONPATH

$ export PYTHONPATH="${PYTHONPATH}:<path-to-cloned-repo>/src"

Run below python script

import os

from pylibsshext.session import Session
from pylibsshext.errors import LibsshSessionException

ssh = Session()

HOST = "<changeme>"
USER = "<changeme>"
PASSWORD = BECOME_PASSWORD = "<changeme>"


try:
    ssh.connect(
        host=HOST,
        user=USER,
        password=PASSWORD,
        timeout=30,
        port=22,
        look_for_keys=False
    )
except LibsshSessionException as ex:
    print(str(ex))

print(ssh.is_connected)

EXPECTED RESULTS

Password authentication should work

ACTUAL RESULTS

Password authentication fails

[TODO] Refactor workflow artifacts management in GHA

[TODO] Add deadsnakes/action into the GHA matrix (Python 3.10-dev)

[BUG] Flaky SEGFAULT while testing `exec_command`

https://github.com/ansible/pylibssh/runs/725098816?check_suite_focus=true#step:10:146:

[gw1] [ 16%] PASSED tests/unit/channel_test.py::test_exec_command Fatal Python error: Segmentation fault

Thread 0x00007f13f32a4700 (most recent call first):
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 400 in read
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 432 in from_io
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 967 in _thread_receiver
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 220 in run
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn

Current thread 0x00007f13f58cf680 (most recent call first):
  File "/home/runner/work/pylibssh/pylibssh/tests/unit/channel_test.py", line 22 in ssh_channel
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/fixtures.py", line 800 in _teardown_yield_fixture
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/fixtures.py", line 871 in finish
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 318 in _callfinalizers
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 328 in _teardown_with_finalization
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 310 in _pop_and_teardown
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 350 in _teardown_towards
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 342 in teardown_exact
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 148 in pytest_runtest_teardown
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 217 in <lambda>
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 244 in from_call
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 217 in call_runtest_hook
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 186 in call_and_report
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 101 in runtestprotocol
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/runner.py", line 85 in pytest_runtest_protocol
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/xdist/remote.py", line 88 in run_one_test
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/xdist/remote.py", line 71 in pytest_runtestloop
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/main.py", line 247 in _main
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/main.py", line 191 in wrap_session
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/xdist/remote.py", line 261 in <module>
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 1084 in executetask
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 220 in run
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 267 in integrate_as_primary_thread
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 1060 in serve
  File "/home/runner/work/pylibssh/pylibssh/.tox/test-binary-dists/lib/python3.7/site-packages/execnet/gateway_base.py", line 1554 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>
[gw1] node down: Not properly terminated
[gw1] [ 91%] FAILED tests/unit/channel_test.py::test_exec_command 
replacing crashed worker gw1

[TODO] Adjust `get-pip.py` in the CI workflow definitions

SUMMARY

$sbj

ISSUE TYPE

Bug Report

PYLISSH and LIBSSH VERSION

N/A

OS / ENVIRONMENT

STEPS TO REPRODUCE

N/A

EXPECTED RESULTS

pip being installed

ACTUAL RESULTS

get-pip.py's syntax is no longer compatible with py2 and py3.5

[TODO] Make manylinux build script splittable for GHA matrix

(and potentially convert it to Python)

[TODO] Produce wheels with `musllinux` tags

SUMMARY

Recently, a new PEP 656 has been accepted. It allows producing platform-specific wheels that can be unpacked under Alpine.

We should integrate producing and testing them into our CI/CD once there's enough ecosystem support in place.

ISSUE TYPE

Feature Idea

ADDITIONAL INFORMATION

Blocked on pypa/auditwheel#305.

[TODO] Add dev Python 3.10 to GHA matrix under macOS too

Similar to #68.

[TODO] Set up DumbPyPI for nightlies on GitHub Pages using GitHub Actions workflows

Explore running pytest with `catchsegv`

I've seen it suggested in a few debugging guides but still need to figure out how it works. The idea is that it will then print out more details for further debugging whenever segfaults will happen in the CI, in particular.

Things to learn:

~~Should it be prependend before tox or pytest?~~ All examples I've found put it before [pytest]
Does it work well with subprocesses of pytest-xdist? What about pytest-forked? [unclear]
~~Does macOS have it or is it GNU/Linux only?~~ It's GNU-specific.

[BUG] Flaky `exec_command` output on SSH channel

SUMMARY

pylibsshext.channel.Channel.exec_command() returns empty output from time to time. One invocation in tests returned an empty string for exec_command('echo -n Hello World').stdout.

ISSUE TYPE

Bug Report

PYLISSH and LIBSSH VERSION

N/A

OS / ENVIRONMENT

Currently observed in GHA under Python 3.5 + macOS. But it's not persistent so I assume that it may be reproducible in other envs.

STEPS TO REPRODUCE

Restart tests/unit/channel_test.py::test_exec_command until it fails.

EXPECTED RESULTS

The test is always green.

ACTUAL RESULTS

The test is occasionally failing:

=================================== FAILURES ===================================
______________________________ test_exec_command _______________________________
[gw1] darwin -- Python 3.5.9 /Users/runner/runners/2.263.0/work/pylibssh/pylibssh/.tox/test-binary-dists/bin/python

ssh_channel = <pylibsshext.channel.Channel object at 0x108861410>

    def test_exec_command(ssh_channel):
        """Test getting the output of a remotely executed command."""
        u_cmd_out = ssh_channel.exec_command('echo -n Hello World').stdout.decode()
>       assert u_cmd_out == u'Hello World'  # noqa: WPS302
E       AssertionError: assert '' == 'Hello World'
E         - Hello World

ssh_channel = <pylibsshext.channel.Channel object at 0x108861410>
u_cmd_out  = ''

tests/unit/channel_test.py:28: AssertionError

Ref: https://github.com/ansible/pylibssh/runs/724505447?check_suite_focus=true#step:10:212

[BUG] `test_exec_command` occasionally crashes with SIGBUS (signal 10) on macOS under Python 2.7

SUMMARY

Re-runs of the CI on devel result in different macOS + Python 2.7 jobs failing flakily on the test_exec_command trying to access some unaddressable memory (SIGBUS).

ISSUE TYPE

Bug Report

PYLISSH and LIBSSH VERSION

devel

OS / ENVIRONMENT

STEPS TO REPRODUCE

Build dists, switch to Python 2.7 and run the following under macOS:

# must have the dists in `dist/` folder
$ tox -e test-binary-dists -- tests/unit/channel_test.py::test_exec_command
# OR:
$ tox -e test-source-dists -- tests/unit/channel_test.py::test_exec_command
# The follow-up runs could be (after tox initialized the envs
$ PYTHONPATH=bin .tox/test-binary-dists/bin/python -m pytest tests/unit/channel_test.py::test_exec_command

EXPECTED RESULTS

Green CI

ACTUAL RESULTS

=================================== FAILURES ===================================
______________________________ test_exec_command _______________________________
[gw0] darwin -- Python 2.7.18 /Users/runner/runners/2.263.0/work/pylibssh/pylibssh/.tox/test-binary-dists/bin/python
:-1: running the test CRASHED with signal 10

Refs:

[TODO] Move dists metadata verification to an async job in GHA (parallel to tests)

Add support to pass private key file path to connect() API

SUMMARY

Add support to pass the private key file path to the session connect() API

ISSUE TYPE

Feature Idea

ADDITIONAL INFORMATION

from pylibsshext.session import Session
from pylibsshext.errors import LibsshSessionException

ssh = Session()
    ssh.connect(
        host=HOST,
        user=USER,
        password=PASSWORD,
        private_key_filename= PRIVATE_KEY_FILEPATH,    # new argument
        timeout=30,
        port=22,
        look_for_keys=False
    )

Add support for SCP libssh API's

SUMMARY

Add support for SCP API's to be invoked from Ansible libssh connection plugin

get_transport()
put()
get()

ISSUE TYPE

Feature Request

[TODO] Switch to using built-in Cython coverage plugins once it's fixed upstream

Refs:

Decide on testing strategies

SUMMARY

We need to decide and document how we do testing.

Right now we have a strong linter suite integrated and running in the CI.
It's mostly enough for its purposes but I should note that some of the checks simply don't support Cython so if we discover good Cython linters we should totally add them into the suite. We have some support for the coverage collection but Cython plugin is currently disabled.

We also have pytest which at the moment does almost nothing. It's a smoke test in its current incarnation, it only checks imports and object construction. Also, 2/3 of the smoke tests are failing on object instantiation.

Besides, we have a few more smoke tests in the form of imports after building dists here and there.

Build/test/lint steps are wrapped with tox to be easier and better reproducible. This is also very helpful when mapping them to CI and making sure that they run there in the same manner as locally.

Now, pytest-based tests need to be specified more clearly. As I see it, we should have unit and integration tests. I think we can cover both types using pure pytest (maybe with plugins). For integration tests, we can use OpenSSH that is usually already pre-installed on the systems that we have interest in supporting.

@ganeshrn also mentioned adding some docker containers for integration tests but I don't have a clear understanding of why we can't test things w/o them. This needs clarification. FWIW it's totally possible to run Docker on Linux workers in GitHub Actions Workflows if really needed.

ISSUE TYPE

Feature Idea

Not able to skip host identity check

Need option in session to skip host identity.

In ansible.cfg we can use.

host_key_checking=false

can we have same parameter like this?

[TODO] Build the platform-specific dists based on an sdist instead of the source checkout

SUMMARY

$sbj

ISSUE TYPE

Feature Idea

ADDITIONAL INFORMATION

https://twitter.com/webknjaz/status/1390920203266056196

[TODO] Fix PEP517 in the main toxenv

Ref: tox-dev/tox#1575 (comment)

[TODO] Make GHA compile tagged releases w/o profiling

SEGFAULT with executing remote commands without LF

SUMMARY

I discovered that sending two commands — one without an LF (\n) and followed by a similar exec_command() invocation — causes a SEGFAULT.

ISSUE TYPE

Bug Report

PYLISSH and LIBSSH VERSION

The latest wheel on PyPI (v0.2.0) but I expect it to be problematic on devel too.

OS / ENVIRONMENT

GNU/Linux (haven't checked on others)

STEPS TO REPRODUCE

Basically, if you comment out most of the blocks in the snippet below but keep two, so that one of them doesn't have a \n and the next one does, you'll see a SEGFAULT.

(change username/key/host to something you can access)

# ansible-pylibssh-test.py
from contextlib import contextmanager
from pathlib import Path
from time import perf_counter

from pylibsshext.errors import LibsshSessionException
from pylibsshext.session import AutoAddPolicy, Session

host = '127.0.0.1'
user = 'wk'
key_file = Path(__file__).parent / "id_rsa_test"
key_data = key_file.read_bytes()

ssh = Session()
ssh.set_missing_host_key_policy(AutoAddPolicy())
try:
    ssh.connect(
        host=host,
        user=user,
        timeout=30,
        port=22,
        private_key=key_data
    )
except LibsshSessionException as ssh_exc:
    print(f'Failed to connect to {host} over SSH: {ssh_exc!s}')
 
print(f'Connection status: {ssh.is_connected}')

@contextmanager
def perf_check(notice=""):
    toc = perf_counter()
    try:
        yield
    finally:
        print(f"Elapsed {notice} time: {perf_counter() - toc:0.4f} seconds")

with perf_check(notice="total"):
    with perf_check(notice="channel create"):
        ssh_channel = ssh.new_channel()
    try:
        with perf_check(notice="execution of write() w/o LF"):
            cmd_resp = ssh_channel.write(b'ls')
        print(f"Command object: {cmd_resp}")
        with perf_check(notice="responce check"):
            for item in ["stdout", "stderr", "returncode"]:
                if not hasattr(cmd_resp, item):
                    continue
                print(f'{item}:\n{getattr(cmd_resp, item)}\n')

        with perf_check(notice="execution of write() w/ LF"):
            cmd_resp = ssh_channel.write(b'ls\n')
        print(f"Command object: {cmd_resp}")
        with perf_check(notice="responce check"):
            for item in ["stdout", "stderr", "returncode"]:
                if not hasattr(cmd_resp, item):
                    continue
                print(f'{item}:\n{getattr(cmd_resp, item)}\n')

        with perf_check(notice="execution of poll()"):
            ssh_channel.poll()
        with perf_check(notice="execution of read_bulk_response()"):
            ssh_channel.read_bulk_response()
        with perf_check(notice="execution of sendall() w/ LF"):
            cmd_resp = ssh_channel.sendall(b'ls\n')
        print(f"Command object: {cmd_resp}")
        with perf_check(notice="responce check"):
            for item in ["stdout", "stderr", "returncode"]:
                if not hasattr(cmd_resp, item):
                    continue
                print(f'{item}:\n{getattr(cmd_resp, item)}\n')

        with perf_check(notice="execution of exec_command() w/o LF"):
            cmd_resp = ssh_channel.exec_command(u'ls')
        print(f"Command object: {cmd_resp}")
        with perf_check(notice="responce check"):
            for item in ["stdout", "stderr", "returncode"]:
                if not hasattr(cmd_resp, item):
                    continue
                print(f'{item}:\n{getattr(cmd_resp, item)}\n')

        with perf_check(notice="execution of exec_command() w/ LF"):
            cmd_resp = ssh_channel.exec_command(u'ls\n')
        print(f"Command object: {cmd_resp}")
        with perf_check(notice="responce check"):
            for item in ["stdout", "stderr", "returncode"]:
                if not hasattr(cmd_resp, item):
                    continue
                print(f'{item}:\n{getattr(cmd_resp, item)}\n')
    finally:
        ssh_channel.close()

EXPECTED RESULTS

No traceback.

ACTUAL RESULTS

Example log:

$ python ansible-pylibssh-test.py
Connection status: 1
Elapsed channel create time: 0.0236 seconds
Elapsed execution of poll() time: 0.0000 seconds
Elapsed execution of read_bulk_response() time: 0.0056 seconds
Elapsed execution of sendall() w/ LF time: 30.0226 seconds
Command object: 0
Elapsed responce check time: 0.0000 seconds
Elapsed execution of exec_command() w/o LF time: 0.0284 seconds
Command object: CompletedProcess(args='ls', returncode=0, stdout=b'Downloads\nPictures\nantsibull.log\nprof-29d8\nprof-31a3\nprof-4b69\nprof-74be\nprof-8685\nprof-8ed1\nprof-ac12\nprof-ef2b.main.cpuprofile.txt\nsecure\nsrc\nsystem-auth.pam.d.bak\nsystem-login.pam.d.bak\n', stderr=b'')
stdout:
b'Downloads\nPictures\nantsibull.log\nprof-29d8\nprof-31a3\nprof-4b69\nprof-74be\nprof-8685\nprof-8ed1\nprof-ac12\nprof-ef2b.main.cpuprofile.txt\nsecure\nsrc\nsystem-auth.pam.d.bak\nsystem-login.pam.d.bak\n'

stderr:
b''

returncode:
0

Elapsed responce check time: 0.0000 seconds
[1]    1198435 segmentation fault (core dumped)  venv/bin/python ansible-pylibssh-test.py

[TODO] Integrate towncrier

Also, optionally, integrate https://github.com/apps/psf-chronographer.

[TODO] Move making tagged releases in GHA to a separate tag created trigger

Installs from sdists produce broken C-extensions

SUMMARY

It's really low-priority right now (because it's unlikely that anybody would hit it) but installs from sdist under Ubuntu produce C-extensions that explode with missing symbols:

E   ImportError: /home/runner/work/pylibssh/pylibssh/.tox/test-source-dists/lib/python2.7/site-packages/pylibsshext/session.so: undefined symbol: ssh_disconnect

ISSUE TYPE

Bug Report

OS / ENVIRONMENT

https://github.com/ansible/pylibssh/pull/102/checks?check_run_id=879173917#step:12:178

[DESIGN] Improve DX/UX of the public APIs

Define the public APIs
Separate internal APIs

Thoughts 💭

It's not clear what are the usage scenarios of the lib (API-wise).
It is too low-level (mostly).
Most of the exposed classes implement managing resources and require calls to open/connect and close/disconnect. The idiomatic way of working with them is to implement context manager interfaces that guarantee that the managed resources are closed once unneeded, no matter what.
The usage sequence seems to require passing extra initialization params to the connect method. This is an antipattern: objects must be ready to use right away after the initialization. So things like credentials should be passed to init, not connect.

[BUG] Importing `pylibsshext.channel` fails under Python 2

SUMMARY

pylibsshext.channel contains from subprocess import CalledProcessError, CompletedProcess which causes it to explode with ImportError.
This is because both CalledProcessError and CompletedProcess were first introduced in Python 3.5.

ISSUE TYPE

Bug Report

PYLISSH and LIBSSH VERSION

devel / N/A

OS / ENVIRONMENT

N/A

STEPS TO REPRODUCE

[root@e18235c278ea ~]# /tmp/ansible_pylibssh-manylinux1-build.zaaAa46362/venvs/cp27-cp27m-1/bin/python
Python 2.7.17 (default, Jan 13 2020, 10:48:20) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pylibsshext.channel import Channel
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "src/pylibsshext/channel.pyx", line 23, in init pylibsshext.channel
ImportError: cannot import name CompletedProcess
>>> from pylibsshext.channel import Channel

(interestingly, Channel gets imported on the first try and the second attempt gets it from cache)

[root@e18235c278ea ~]# /tmp/ansible_pylibssh-manylinux1-build.zaaAa46362/venvs/cp27-cp27m-1/bin/python
Python 2.7.17 (default, Jan 13 2020, 10:48:20) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pylibsshext.channel
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "src/pylibsshext/channel.pyx", line 23, in init pylibsshext.channel
ImportError: cannot import name CompletedProcess
>>> import pylibsshext.channel
>>> import pylibsshext.channel
>>>

EXPECTED RESULTS

(no ImportError)

ACTUAL RESULTS

(see the repro section)

Publish a platform-specific wheel for Apple M1 (macOS ARM64) on PyPI

SUMMARY

add Package for macos arm64

ISSUE TYPE

Feature Idea

ADDITIONAL INFORMATION

N/A

GHA generates an invalid dist version on devel pushes

https://github.com/ansible/pylibssh/runs/808731189?check_suite_focus=true#step:15:66

I guess Git under macOS is older and the "fetch history" step does not retrieve all the required commits chain that lead to the previous tag.

Add support for GSSAPI based authentication

SUMMARY

Add libssh API support for GSSAPI based authentication https://api.libssh.org/stable/libssh_tutor_authentication.html

ISSUE TYPE

Feature Idea

ADDITIONAL INFORMATION

Building docs only work the first time (for non-doc modifications)

SUMMARY

Maybe I missed something but, (probably depending on the changes you make,) tox -e build-docs only builds the doc the 1st time. Then, it relies on .tox/.tmp/.doctrees, and will read the source again only if a non-changelog-fragment rst file was modified.

ISSUE TYPE

Bug Report

PYLISSH and LIBSSH VERSION

457c1e6 and libssh is irrelevant

OS / ENVIRONMENT

Not sure anything is relevant here, but if need be, MacOS.

STEPS TO REPRODUCE

$ tox -e build-docs
$ echo "Hey **there**" > docs/changelog-fragments/12.mist.rst
$ tox -e build-docs

EXPECTED RESULTS

On the second build, the changelog fragment should appear

ACTUAL RESULTS

It doesn't

CAUSED BY

Sphinx relies on .tox/.tmp/.doctrees. If it exists and (I assume) docs rst files have not been modified, then the doc parsing does not occur.
If this folder is deleted then it works normally.
A few things that can modify the doc output and are not triggering a new build:

The python code under _ext
Changelog fragments

[TODO] Refactor manylinux scenario to work with internal git clone

[TODO] Set up Sphinx docs

Integrate Sphinx into tox and GHA
Integrate RTD (almost ready -- https://readthedocs.org/projects/ansible-pylibssh/, need @ganeshrn to resync the webhook @ https://readthedocs.org/dashboard/ansible-pylibssh/integrations/127409/)
Enable RTD build status reports on PRs
Integrate RTD link into the dist metadata
Check in the Sphinx skeleton using https://github.com/ansible-community/sphinx_ansible_theme
Integrate https://github.com/sphinx-contrib/apidoc

[TODO] Enable testing sdists under Ubuntu in GHA

pylibssh takes surprisingly longer time than paramiko to receive the same data

SUMMARY

Related to ansible-collections/cisco.nxos#145
The above issue is caused because pylibssh takes ~300 times longer than paramiko to read the output of show vlan | json from the same target device. This resulted in the command_timeout value to work for paramiko but not libssh.
This was realised by adding time.perf_counter() in relevant places within network_cli code as defined in this patch.
Based on multiple passes, paramiko takes ~3 secs while pylibssh takes ~1000 secs to receive the same data.
Data from one such pass is added here - https://gist.github.com/NilashishC/62cf8e945ac9a49e743fb2b553b8716d.
On bumping the buffer size in read_non_blocking() to 8096 bytes and removing the upper cap of 1024 bytes, this delay was reduced to ~280 secs.

ISSUE TYPE

Bug Report

PYLISSH and LIBSSH VERSION

$ pip show ansible-pylibssh
Name: ansible-pylibssh
Version: 0.2.0
Summary: Python bindings for libssh client specific to Ansible use case
Home-page: https://github.com/ansible/pylibssh
Author: Ansible, Inc.
Author-email: info+github/ansible/[email protected]
License: LGPLv2+
Location: /Users/nchakrab/venvs/dev/lib/python3.6/site-packages
Requires:
Required-by:

>>> import pylibsshext
>>> pylibsshext.__libssh_version__
'0.9.5'

OS / ENVIRONMENT

ProductName: Mac OS X
ProductVersion: 10.15.7

STEPS TO REPRODUCE

Configure 500+ vlans on a Cisco Nexus appliance.
Run the following playbook with first paramiko and then libssh.

---
- hosts: nxos
  gather_facts: no
  tasks:
    - cisco.nxos.nxos_vlans:
        config:
          - vlan_id: 10
            enabled: True
        state: merged

EXPECTED RESULTS

The playbook run doesn't take an exorbitantly high time to complete.

ACTUAL RESULTS

When command_timeout < 1100 it results in a timeout. With a higher command_timeout set, the playbook run completes as expected.

ansible / pylibssh Goto Github PK

pylibssh's Introduction

pylibssh: Python bindings to client functionality of libssh specific to Ansible use case

Nightlies @ Dumb PyPI @ GitHub Pages

Requirements

Building the module

License

pylibssh's People

Contributors

Stargazers

Watchers

Forkers

pylibssh's Issues

SUMMARY

ISSUE TYPE

PYLISSH and LIBSSH VERSION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

SUMMARY

ISSUE TYPE

ADDITIONAL INFORMATION

SUMMARY

ISSUE TYPE

PYLISSH and LIBSSH VERSION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

SUMMARY

ISSUE TYPE

PYLISSH and LIBSSH VERSION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

SUMMARY

ISSUE TYPE

ADDITIONAL INFORMATION

SUMMARY

ISSUE TYPE

PYLISSH and LIBSSH VERSION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

SUMMARY

ISSUE TYPE

PYLISSH and LIBSSH VERSION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

SUMMARY

ISSUE TYPE

ADDITIONAL INFORMATION

SUMMARY

ISSUE TYPE

SUMMARY

ISSUE TYPE

SUMMARY

ISSUE TYPE

ADDITIONAL INFORMATION

SUMMARY

ISSUE TYPE

PYLISSH and LIBSSH VERSION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

SUMMARY

ISSUE TYPE

OS / ENVIRONMENT

Thoughts 💭

SUMMARY

ISSUE TYPE

PYLISSH and LIBSSH VERSION

OS / ENVIRONMENT

STEPS TO REPRODUCE