Giter VIP home page Giter VIP logo

salt-ext-modules-vmware's Introduction

Salt Extension Modules for VMware

This is a collection of Salt-maintained extension modules for use with VMware vSphere, vCenter, ESXi, and friends.

Security

If you think you've found a security vulnerability, see Salt's security guide.

User Documentation

This README is more for contributing to the project. If you just want to get started, check out the User Documentation. Note: See the Managing VMC SDDC with Salt section for more information about how to configure properties required for VMC operations

Contributing

The salt-ext-modules-vmware project team welcomes contributions from the community. If you wish to contribute code and you have not signed our contributor license agreement (CLA), our bot will update the issue when you open a Pull Request. For any questions about the CLA process, please refer to our FAQ.

The Salt Contributing guide has a lot of relevant information, but if you'd like to jump right in here's how to get started:

# Clone the repo
git clone --origin salt [email protected]:saltstack/salt-ext-modules-vmware.git

# Change to the repo dir
cd salt-ext-modules-vmware

# Create a new venv, after sourcing activate `python` will refer to python3.
python3 -m venv env --prompt vmw-ext
source env/bin/activate

# On mac, you may need to upgrade pip
python -m pip install --upgrade pip

# On WSL or some flavors of linux you may need to install the `enchant`
# library in order to build the docs
sudo apt-get install -y enchant

# Install extension + test/dev/doc dependencies into your environment
python -m pip install -e .\[tests,dev,docs\]

# Run tests!
python -m nox -e tests-3

# skip requirements install for next time
export SKIP_REQUIREMENTS_INSTALL=1

# Build the docs, serve, and view in your web browser:
python -m nox -e docs && (cd docs/_build/html; python -m webbrowser localhost:8000; python -m http.server; cd -)

# If you want to run tests against an actual vCenter:

# 1. Make a local salt dir
mkdir -p local/etc/salt

# 2. Make a local dir for salt state files
mkdir -p local/srv/salt

# 3. Make a local dir for salt pillar files
mkdir -p local/srv/pillar

# 4. Create a minion config
cat << EOF> local/etc/salt/minion
user: $(whoami)
root_dir: $PWD/local/
file_root: $PWD/local
master: localhost
id: saltdev
master_port: 55506
pillar_roots:
  base:
    - $PWD/local/srv/pillar
EOF

# 5. Make a Saltfile
cat << EOF> Saltfile
salt-call:
  local: true
  config_dir: local/etc/salt
EOF

# 6. Create a pillar file for you configuration
cat << EOF> local/srv/my_vsphere_conf.sls
# vCenter
saltext.vmware:
  # Or use IP address, e.g. 203.0.113.42
  host: vsphere.example.com
  password: CorrectHorseBatteryStaple
  user: BobbyTables
EOF

# 7. Create a pillar top file
cat << EOF>  local/srv/pillar.sls
base:
  saltdev:
    - my_vsphere_conf
EOF

# 8. (deprecated but not removed yet) If you're contributing to the project and need to run the tests, create a test config file:
python tools/test_value_scraper.py -c local/vcenter.conf

# 9. (deprecated but not removed yet) Create a test config file for VMC:
python tools/test_value_scraper_vmc.py --help
This command will return the required information.

For code contributions, as part of VMware we require a signed CLA. If you've already signed the VMware CLA, you're probably good to go.

Of course, writing code isn't the only way to contribute! We value contributions in any of these areas:

You could also contribute in other ways:

  • Writing blog posts
  • Posting on social media about how you used Salt+VMware to solve your problems, including videos
  • Giving talks at conferences
  • Publishing videos
  • Asking/answering questions in IRC, Slack, or email groups

Any of these things are super valuable to our community, and we sincerely appreciate every contribution!

For more information, build the docs and head over to http://localhost:8000/ โ€” that's where you'll find the rest of the documentation.

salt-ext-modules-vmware's People

Contributors

abhi1693 avatar cheburakshu avatar chruck avatar cmcmarrow avatar dhiltonp avatar dmurphy18 avatar garethgreenaway avatar ggiesen avatar jain-prerna avatar jgangel avatar joechainz avatar kdsalvy avatar navalkp avatar sagetherage avatar scriptautomate avatar shaikres avatar shaswat91 avatar waynew avatar zruzhinov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

salt-ext-modules-vmware's Issues

Automate release testing

In order to actually focus on automating the release tasks, we need an issue for it.

This issue is to track the automation of part of the release process

The high level design is that we should have a single command that can be executed, and the end result is that we should have a pass/fail, and especially for pass we MUST have the accompanying release artifact. We should follow the release process, but basically this looks like:

  • checkout the most recent salt/main code (maybe create a worktree for it?? Oh! No - can checkout the contents of the repo in another directory without actually tracking it with git. I forget what this is called.)
  • build the wheel aka test artifact
  • create a virtualenv and install the test artifact to it, along with test/doc requirements
  • build the docs - errors in docs build mean the build has failed
  • run the tests against the build/installed artifact - failures in tests mean the build was failed
  • at this point if there are any failures, we should halt and bail out
  • update the version.py file to the desired change, add & commit
  • re-run the build/install/test workflow with this final release candidate -- I suppose in theory we could bump the version prior to running the build/test cycle? However, building from the latest main branch ensures that failures are completely unrelated ๐Ÿค”
  • display pass/fail, and path of the release artifact that was tested

I think we should wait on automating creating the tag on pass - but that's something that we could do once we have towncrier for building the changelog, since the annotation on the tag should just be the contents of the changelog.

This means that we could do something like:

$ create-release 2021-11-10
...
PASS - dist/saltext.vmware-21.11.10-py3-none-any.whl
$ git tag -a 21.11.10
$ twine upload dist/saltext.vmware-21.11.10-py3-none-any.whl
$ git push salt 21.11.10

Followed by creating a release on GitHub.

tests.integration.modules.test_nsxt_ip_pools : test_nsxt_ip_pools_execution_module_crud_operations

Failed with a fresh VMC_CONFIG file from the scraper

    @pytest.fixture(autouse=True)
    def setup(nsxt_config):
        hostname, username, password = _get_server_info(nsxt_config)
>       ip_pools_from_nsxt = _get_ip_pool_by_display_name_using_nsxt_api(
            hostname, username, password, "IP_Pool_Salt_FT"
        )

hostname   = 'nsxt-hostname/ip'
nsxt_config = {'cert': 'path-to-nsxt-cert.pem', 'compute_manager_server': 'compute-manager-server-ip', 'credential': {'credential_ty...ompute-manager-server-thumbprint', 'username': 'compute-manager-server-username'}, 'hostname': 'nsxt-hostname/ip', ...}
password   = 'nsxt-password'
username   = 'nsxt-username'

tests.integration.states.test_nsxt_ip_blocks : test_nsxt_ip_blocks_state_module

Fresh VMC_CONFIG file

    @pytest.fixture(autouse=True)
    def setup(nsxt_config):
        hostname, username, password = _get_server_info(nsxt_config)
>       ip_block_from_nsxt = _get_ip_block_by_display_name_using_nsxt_api(
            hostname, username, password, "IP_Block_Salt_State_FT"
        )
        

hostname   = 'nsxt-hostname/ip'
nsxt_config = {'cert': 'path-to-nsxt-cert.pem', 'compute_manager_server': 'compute-manager-server-ip', 'credential': {'credential_ty...ompute-manager-server-thumbprint', 'username': 'compute-manager-server-username'}, 'hostname': 'nsxt-hostname/ip', ...}
password   = 'nsxt-password'
username   = 'nsxt-username'

tests.integration.modules.test_esxi : test_get_dns_config

    def test_get_dns_config(service_instance):
        """
        Test get dns configuration on ESXi host
        """
        ret = esxi.get_dns_config(
            service_instance=service_instance,
            datacenter_name="Datacenter",
            cluster_name="Cluster",
        )
        assert ret
        for host in ret:
>           assert ret[host]["ip"]
E           assert []

tests.integration.modules.test_nsxt_license : test_apply_license

Failed with fresh VMC_CONFIG file

@pytest.fixture
    def delete_license():
        """
        Sets up test requirements:
        Queries nsx api for licenses
        Deletes license if exists
        """
        url = BASE_URL.format(management_host=hostname)
        licenses_dict = nsxt_request.call_api(
            method="GET", url=url, username=username, password=password, verify_ssl=_verify_ssl
        )
    
>       if licenses_dict["result_count"] != 0:
E       KeyError: 'result_count'

licenses_dict = {'error': 'Error occurred while calling NSX-T API https://nsxt-hostname/ip/api/v1/licenses. Please check logs for more details.'}
url        = 'https://nsxt-hostname/ip/api/v1/licenses'

tests/integration/modules/test_nsxt_license.py:34: KeyError

tests.integration.modules.test_esxi : test_manage_service

    def test_manage_service(service_instance):
        """
        Test manage services on esxi host
        """
        SSH_SERVICE = "TSM-SSH"
        ret = esxi.manage_service(
            service_name=SSH_SERVICE,
            service_instance=service_instance,
            state="start",
            datacenter_name="Datacenter",
            cluster_name="Cluster",
        )
        assert ret
    
        ret = esxi.list_services(
            service_name=SSH_SERVICE,
            service_instance=service_instance,
            datacenter_name="Datacenter",
            cluster_name="Cluster",
        )
        for host in ret:
>           assert ret[host][SSH_SERVICE]["state"] == "running"
E           KeyError: 'TSM-SSH'

Not sure what happened here, but... clearly failing

exclusively use pillar data for vmc credentials

We need to extract the function arguments used for vmc connections from pretty much everywhere, and change them to be pulled exclusively from pillar data. Conveniently this will mostly be deletions with just a few additions to vmc_reques::call_api :)

I suggest this structure, with these 6 parameters, but @waynew will have a good perspective on the normal naming and organization of pillar data.

vmc_connection_details:
    console_host: hostname
    api_key: key
    org_id: UUID
    sddc_id: UUID
    nsx_host: hostname
    verify_ssl: False

verify_ssl: (Optional) True/False or string with path to CA bundle. Defaults to True.

We should also delete cert at the same time.

We will need to add documentation on how to find the correct values - generating an api key, browsing the UI to find the org_id and nsx_host etc.

This documentation will need to be linked in the module-level docstring of all vmc files.

tests.integration.modules.test_nsxt_transport_zone : test_check_transport_zone

    def test_check_transport_zone(nsxt_config, salt_call_cli):
        ret_create = salt_call_cli.run(
            "nsxt_transport_zone.create",
            hostname=nsxt_config["hostname"],
            username=nsxt_config["username"],
            password=nsxt_config["password"],
            host_switch_name="nsxDefaultHostSwitch",
            transport_type="OVERLAY",
            display_name="Test_Create_Transport_Zone-IT",
            verify_ssl=False,
        )
    
        assert ret_create is not None
        result_as_json_create = ret_create.json
>       assert result_as_json_create["display_name"] == "Test_Create_Transport_Zone-IT"
E       KeyError: 'display_name'

tests.integration.states.test_nsxt_transport_node : test_state_transport_node_verify

fresh VMC_CONFIG file

    def test_state_transport_node_verify(nsxt_config, salt_call_cli):
        response_create = salt_call_cli.run(
            "state.single",
            "nsxt_transport_node.present",
            name="create transport node",
            hostname=nsxt_config["hostname"],
            username=nsxt_config["username"],
            password=nsxt_config["password"],
            verify_ssl=False,
            display_name=_display_name,
            description="Create-Transport-Node",
            node_deployment_info=_node_deployment_info,
        )
    
        result_create = dict(list(response_create.json.values())[0])["result"]
        comment_create = dict(list(response_create.json.values())[0])["comment"]
    
>       assert result_create is True
E       assert False is True

comment_create = 'Failed to get the transport nodes : Error occurred while calling NSX-T API https://nsxt-hostname/ip/api/v1/transport-nodes. Please check logs for more details.'

tests.integration.modules.test_nsxt_ip_blocks : test_nsxt_ip_blocks_execution_module_crud_operations

Might be an issue in the scraper


    @pytest.fixture(autouse=True)
    def setup(nsxt_config):
        hostname, username, password, cert = _get_server_info(nsxt_config)
>       ip_block_from_nsxt = _get_ip_block_by_display_name_using_nsxt_api(
            hostname, username, password, "IP_Block_Salt_FT"
        )

cert       = 'path-to-nsxt-cert.pem'
hostname   = 'nsxt-hostname/ip'
nsxt_config = {'cert': 'path-to-nsxt-cert.pem', 'compute_manager_server': 'compute-manager-server-ip', 'credential': {'credential_ty...ompute-manager-server-thumbprint', 'username': 'compute-manager-server-username'}, 'hostname': 'nsxt-hostname/ip', ...}
password   = 'nsxt-password'
username   = 'nsxt-username'

Warn/error when service instance cannot be created

Currently if config values aren't setup for creating the service instance, an unhelpful message is returned:

salt-call --local vmware_datacenter.list 

Passed invalid arguments: expected string or bytes-like object.

Usage:

    Returns a list of datacenters for the specified host.

    .. code-block:: bash

        salt '*' vmware_datacenter.list

If an argument is passed then it's uh... much worse ๐Ÿ™ƒ

๐Ÿ˜ฑ

  File "/usr/sbin/salt-call", line 8, in <module>                                                                                                                                                                                   [20/1883]
    sys.exit(salt_call())                                                                                             
  File "/usr/lib/python3.9/site-packages/salt/scripts.py", line 449, in salt_call
    client.run()                                           
  File "/usr/lib/python3.9/site-packages/salt/cli/call.py", line 58, in run
    caller.run()                                           
  File "/usr/lib/python3.9/site-packages/salt/cli/caller.py", line 112, in run
    ret = self.call()                                                                                                 
  File "/usr/lib/python3.9/site-packages/salt/cli/caller.py", line 219, in call
    ret["return"] = self.minion.executors[fname](                                                                     
  File "/usr/lib/python3.9/site-packages/salt/loader.py", line 1241, in __call__
    return self.loader.run(run_func, *args, **kwargs)      
  File "/usr/lib/python3.9/site-packages/salt/loader.py", line 2274, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.9/site-packages/salt/loader.py", line 2289, in _run_as
    return _func_or_method(*args, **kwargs)                
  File "/usr/lib/python3.9/site-packages/salt/executors/direct_call.py", line 12, in execute
    return func(*args, **kwargs)       
  File "/usr/lib/python3.9/site-packages/salt/loader.py", line 1241, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.9/site-packages/salt/loader.py", line 2274, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.9/site-packages/salt/loader.py", line 2289, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/saltext/vmware/modules/datacenter.py", line 41, in list_
    return utils_datacenter.list_datacenters(service_instance)
  File "/usr/lib/python3.9/site-packages/saltext/vmware/utils/common.py", line 648, in list_datacenters
    return list_objects(service_instance, vim.Datacenter)
  File "/usr/lib/python3.9/site-packages/saltext/vmware/utils/common.py", line 299, in list_objects
    item_list = get_mors_with_properties(service_instance, vim_object, properties)
  File "/usr/lib/python3.9/site-packages/saltext/vmware/utils/common.py", line 219, in get_mors_with_properties
    content = get_content(*content_args, **content_kwargs)
  File "/usr/lib/python3.9/site-packages/saltext/vmware/utils/common.py", line 89, in get_content
    container_ref = get_root_folder(service_instance)
  File "/usr/lib/python3.9/site-packages/saltext/vmware/utils/common.py", line 37, in get_root_folder
    return service_instance.RetrieveContent().rootFolder
AttributeError: 'str' object has no attribute 'RetrieveContent'

That's undesirable. A good approach would be for https://github.com/saltstack/salt-ext-modules-vmware/blob/main/src/saltext/vmware/utils/connect.py to detect missing values and alert on the missing config values.

vmc api: raise exceptions on error

Currently vmc_request.call_api returns {"error": "reason"} on errors.

Errors should be thrown up the stack. We need to define a few Exception types and raise those.

Managing SDDC State

When I create a SDDC using a state file like this and then add arguments for num_hosts, hostname, etc:

create_sddc:
module.run:
- name: vmc_sddc.create

It creates the SDDC in VMC.
However if I make a change to an argument to change the config, like num_hosts and re-run the state it creates a new SDDC instead of updating the existing named SDDC. Is there a way to enforce a state of a SDDC vs. creating a new one yet? Thanks.

Fix Test Failures

These tests failed for me when running the full test suite (i.e. python -m tests-3) with a fresh vcenter.conf and vmc.conf for testing:

tests.integration.modules.test_nsxt_compute_manager : test_register_update_and_remove

Failed with:

failed on setup with "OSError: Could not find a suitable TLS CA certificate bundle, invalid path: path-to-nsxt-cert.pem"

    @pytest.fixture
    def delete_compute_manager(nsxt_config):
        """
        Sets up test requirements:
        Queries nsx api for compute manager 10.206.243.180
        Deletes compute manager if exists
        """
        hostname = nsxt_config["hostname"]
        username = nsxt_config["username"]
        password = nsxt_config["password"]
        cert = nsxt_config.get("cert", False)
        compute_manager_server = nsxt_config["compute_manager_server"]
    
        url = "https://{management_host}/api/v1/fabric/compute-managers".format(
            management_host=hostname
        )
>       response = requests.get(url=url, auth=(username, password), verify=cert)

This is probably an issue with the test value scraper, or the tests

tests.integration.modules.test_esxi : test_acceptance_level

    def test_acceptance_level(service_instance):
        """
        Test acceptance level on esxi host
        """
        ret = esxi.set_acceptance_level(
            acceptance_level="community",
            service_instance=service_instance,
            datacenter_name="Datacenter",
            cluster_name="Cluster",
        )
        for h in ret:
            assert ret[h] == "community"
    
        ret = esxi.get_acceptance_level(
            acceptance_level="community",
            service_instance=service_instance,
            datacenter_name="Datacenter",
            cluster_name="Cluster",
        )
>       assert set(ret.values()) == {"community"}
E       AssertionError: assert set() == {'community'}
E         Extra items in the right set:
E         'community'
E         Full diff:
E         - {'community'}
E         + set()

get rid of `vmc_constants.py`

80% of the variables in vmc_constants.py are only used one time, and 95% of the time the variables are re-used, they are like GET_REQUEST_METHOD = "get", and it's better to use "get" in the code directly.

"get" as a http method is not changing, and request(method="get" is perfectly clear.

tests.integration.modules.test_datacenter : test_get

This test is failing


    def test_get(vmware_datacenter, service_instance):
        """
        Test scenarios for get datacenter.
        """
    
        # Get a non existent datacenter. Should return False
        dc1_name = str(uuid.uuid4())
>       dc1 = datacenter.get(cluster_name=dc1_name, service_instance=service_instance)
E       AttributeError: module 'saltext.vmware.modules.datacenter' has no attribute 'get'

create test fixtures to build vCenter/vSphere config to expected values

Currently we're using the test_value_scraper.py as a workaround.

We need to replace test_value_scraper.py with proper fixtures that take the service instance and ensure that specific datacenters, vms, etc. are all existing.

This should potentially be fixtures using other fixtures, probably with scope='session' ๐Ÿค”

Extract out function for boot order list creation

This iteration happens several times. Perhaps it would better be extracted to something like

src/saltext/vmware/utils/vm.py

boot_order_list = []
device_types = (vim.vm.device.VirtualCdrom, vim.vm.device.VirtualDisk, 
existing_order = [device for device in vm.config.hardware.device if isinstance(device, (vim.vm.device.VirtualCdrom, vim.vm.device.VirtualDisk, vim.vm.device.VirtualEthernetCard, vim.vm.device.VirtualFloppy))]
for device_name in order:
    ...

Looking closer, though, I notice that for ethernet and disk we're only paying attention to the first item in the list ๐Ÿค” I wonder if there's a better way for this generally.

tests.integration.modules.test_nsxt_license : test_get_licenses

Failed with fresh VMC_CONFIG

@pytest.fixture
    def get_licenses():
        url = BASE_URL.format(management_host=hostname)
        response = nsxt_request.call_api(
            method="GET", url=url, username=username, password=password, verify_ssl=_verify_ssl
        )
    
>       assert "error" not in response
E       AssertionError: assert 'error' not in {'error': 'Error occurred while calling NSX-T API https://nsxt-hostname/ip/api/v1/licenses. Please check logs for more details.'}

response   = {'error': 'Error occurred while calling NSX-T API https://nsxt-hostname/ip/api/v1/licenses. Please check logs for more details.'}
url        = 'https://nsxt-hostname/ip/api/v1/licenses'

tests/integration/modules/test_nsxt_license.py:60: AssertionError

Fix broken smoke tests

Not sure what or why they're broken, but each time I've run the integration tests I'm seeing errors like this:

ERROR tests/integration/modules/test_vmc_dhcp_profiles.py::test_update_dhcp_profile_smoke_test - requests.exceptions.HTTPError: 400 Client Error:  for url: https://...
ERROR tests/integration/modules/test_vmc_networks.py::test_update_network_smoke_test - requests.exceptions.HTTPError: 400 Client Error:  for url: https://...

This error seems to be happening across PRs, and is blocking PRs with clearly unrelated changes like #217

Currently this is blocking:

Also probably

tests.integration.modules.test_esxi : test_esxi_get_lun_ids_should_return_lun_NAA_ids

    def test_esxi_get_lun_ids_should_return_lun_NAA_ids(service_instance, integration_test_config):
        expected_lun_ids = integration_test_config["esxi_datastore_disk_names"]
        actual_ids = esxi.get_lun_ids(service_instance=service_instance)
>       assert actual_ids == expected_lun_ids
E       AssertionError: assert ['mpx.vmhba0:...ba0:C0:T0:L0'] == ['mpx.vmhba0:C0:T0:L0']
E         Left contains one more item: 'mpx.vmhba0:C0:T0:L0'
E         Full diff:
E         - ['mpx.vmhba0:C0:T0:L0']
E         + ['mpx.vmhba0:C0:T0:L0', 'mpx.vmhba0:C0:T0:L0']

Might be worth creating a test fixture to specify/create the lun/NAA. Otherwise this is probably an issue with the scraper.

tests.integration.modules.test_nsxt_transport_node_profiles : test_nsxt_transport_node_profiles_execution_module

Fresh VMC_CONFIG file

nsxt_config = {'cert': 'path-to-nsxt-cert.pem', 'compute_manager_server': 'compute-manager-server-ip', 'credential': {'credential_ty...ompute-manager-server-thumbprint', 'username': 'compute-manager-server-username'}, 'hostname': 'nsxt-hostname/ip', ...}

    @pytest.fixture
    def setup(nsxt_config):
        """
        Sets up test requirements:
        Queries nsx api for transport node profiles with display name
        Deletes if exists
        Also queries and deletes transport zones which will be used in the IT
        """
>       transport_node_profiles = _get_transport_node_profiles_by_display_name_from_nsxt(
            nsxt_config, _display_name
        )

Missing exception handling in `get_access_token`

@shaikres not sure who is responsible for this section, but I came across the issue when reviewing #22

Feel free to assign whoever would be best for fixing this.


I used this state

blerp:
  vmc_security_rules.present:
    - hostname: "blerp"
    - refresh_key: "werd"
    - authorization_host: "okay"
    - org_id: "nope"
    - sddc_id: lolwat
    - domain_id: imperial dominion
    - rule_id: 13

and rather than giving me a helpful message such as "that refresh key doesn't exist" or "unable to get access token" I got a much less helpful message:

local:
----------
          ID: blerp
    Function: vmc_security_rules.present
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
                  conn = connection.create_connection(
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
                  for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
                File "/usr/lib/python3.9/socket.py", line 953, in getaddrinfo
                  for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
              socket.gaierror: [Errno -2] Name or service not known
              
              During handling of the above exception, another exception occurred:
              
              Traceback (most recent call last):
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
                  httplib_response = self._make_request(
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
                  self._validate_conn(conn)
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
                  conn.connect()
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/urllib3/connection.py", line 353, in connect
                  conn = self._new_conn()
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/urllib3/connection.py", line 181, in _new_conn
                  raise NewConnectionError(
              urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fa7f862ca60>: Failed to establish a new connection: [Errno -2] Name or service not known
              
              During handling of the above exception, another exception occurred:
              
              Traceback (most recent call last):
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
                  resp = conn.urlopen(
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
                  retries = retries.increment(
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
                  raise MaxRetryError(_pool, url, error or ResponseError(cause))
              urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='okay', port=443): Max retries exceeded with url: /csp/gateway/am/api/auth/api-tokens/authorize?refresh_token=werd (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa7f862ca60>: Failed to establish a new connection: [Errno -2] Name or service not known'))
              
              During handling of the above exception, another exception occurred:
              
              Traceback (most recent call last):
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/salt/state.py", line 2171, in call
                  ret = self.states[cdata["full"]](
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/salt/loader.py", line 1235, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/salt/loader.py", line 2268, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/salt/loader.py", line 2283, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/salt/loader.py", line 2316, in wrapper
                  return f(*args, **kwargs)
                File "/home/wayne/programming/salt-ext-modules-vmware/src/saltext/vmware/states/vmc_security_rules.py", line 222, in present
                  get_security_rule = __salt__["vmc_security_rules.get_by_id"](
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/salt/loader.py", line 1235, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/salt/loader.py", line 2268, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/salt/loader.py", line 2283, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/home/wayne/programming/salt-ext-modules-vmware/src/saltext/vmware/modules/vmc_security_rules.py", line 188, in get_by_id
                  return vmc_request.call_api(
                File "/home/wayne/programming/salt-ext-modules-vmware/src/saltext/vmware/utils/vmc_request.py", line 110, in call_api
                  headers = get_headers(refresh_key, authorization_host)
                File "/home/wayne/programming/salt-ext-modules-vmware/src/saltext/vmware/utils/vmc_request.py", line 54, in get_headers
                  access_token = get_access_token(refresh_key, authorization_host)
                File "/home/wayne/programming/salt-ext-modules-vmware/src/saltext/vmware/utils/vmc_request.py", line 38, in get_access_token
                  response = requests.post(url, params=params, headers=headers)
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/requests/api.py", line 119, in post
                  return request('post', url, data=data, json=json, **kwargs)
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/requests/api.py", line 61, in request
                  return session.request(method=method, url=url, **kwargs)
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
                  resp = self.send(prep, **send_kwargs)
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
                  r = adapter.send(request, **kwargs)
                File "/home/wayne/programming/salt-ext-modules-vmware/env/lib/python3.9/site-packages/requests/adapters.py", line 516, in send
                  raise ConnectionError(e, request=request)
              requests.exceptions.ConnectionError: HTTPSConnectionPool(host='okay', port=443): Max retries exceeded with url: /csp/gateway/am/api/auth/api-tokens/authorize?refresh_token=werd (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa7f862ca60>: Failed to establish a new connection: [Errno -2] Name or service not known'))
     Started: 18:52:23.095975
    Duration: 3680.228 ms
     Changes:   

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   3.680 s

(typically when one sees a stack trace, that's a sign that something is broken -- and it's definitely broken here)

use pytest fixture for test datastore lookup instead of integration_test_config

tests/integration/modules/test_datastore.py and
tests/integration/states/test_datastore.py both use integration_test_config["datastores"].

We should modify those tests to get a datastore name (cached using a pytest fixture?), then modify it/restore its state; something like:

@pytest.fixture(scope="session")
def datastore(service_instance, integration_test_config):
     host = integration_test_config["esxi_host_name"]

    datastores = datastore.get(
        service_instance=service_instance,
        host_name=host,
    )

    return datastores[0]

Audit for and deprecate plural state/module names

For example, see Salt module - file is for managing files, apache is for managing apache servers.

It recently came to my attention that some of our states/modules have plural names, so we need to check the list of existing names and deprecate the plural ones unless it makes sense (i.e. it actually has to operate on a collection of things)

tests.integration.states.test_folder : test_move_folder

Fixture should probably remove the folder that it's going to be moved to(?)

ย  | pyVmomi.VmomiSupport.vim.fault.DuplicateName: (vim.fault.DuplicateName) {    dynamicType = <unset>,    dynamicProperty = (vmodl.DynamicProperty) [],    msg = "The name 'top_folder' already exists.",    faultCause = <unset>,    faultMessage = (vmodl.LocalizableMessage) [],    name = 'top_folder',    object = 'vim.Folder:group-v41' }

    def test_move_folder(patch_salt_globals_folder_state):
        """
        test move folder
        """
>       folder.manage("top_folder", "create", "Datacenter", "vm")

patch_salt_globals_folder_state = None

tests/integration/states/test_folder.py:85: 

tests.integration.modules.test_esxi : test_get_firewall_config

    def test_get_firewall_config(service_instance):
        """
        Test get firewall configuration on ESXi host
        """
        ret = esxi.get_firewall_config(
            service_instance=service_instance,
            datacenter_name="Datacenter",
            cluster_name="Cluster",
        )
>       assert ret
E       assert {}

ret        = {}

tests.integration.modules.test_nsxt_manager : test_nsxt_manager_get_and_set_manager_config

fresh vmc_config file

    def test_nsxt_manager_get_and_set_manager_config(nsxt_config, salt_call_cli):
        """
        nsxt_manager.get_manager_config
        nsxt_manager.set_manager_config
        """
        hostname = nsxt_config["hostname"]
        username = nsxt_config["username"]
        password = nsxt_config["password"]
>       response_json = _get_manager_config_from_nsxt_api(hostname, username, password)

...

    def _get_manager_config_from_nsxt_api(hostname, username, password):
        response_json = nsxt_request.call_api(
            method="GET",
            url=BASE_URL.format(hostname),
            username=username,
            password=password,
            verify_ssl=False,
        )
    
>       assert "error" not in response_json
E       AssertionError: assert 'error' not in {'error': 'Error occurred while calling NSX-T API https://nsxt-hostname/ip/api/v1/configs/management. Please check logs for more details.'}


hostname   = 'nsxt-hostname/ip'
password   = 'nsxt-password'
response_json = {'error': 'Error occurred while calling NSX-T API https://nsxt-hostname/ip/api/v1/configs/management. Please check logs for more details.'}
username   = 'nsxt-username'

tests.integration.modules.test_nsxt_transport_node : test_check_transport_nodes

Fresh VMC_CONFIG file

    def test_check_transport_nodes(nsxt_config, salt_call_cli):
        # Create of the transport node
        ret_create = salt_call_cli.run(
            "nsxt_transport_node.create",
            hostname=nsxt_config["hostname"],
            username=nsxt_config["username"],
            password=nsxt_config["password"],
            verify_ssl=_verify_ssl,
            transport_zone_endpoints=[],
            maintenance_mode="DISABLED",
            node_deployment_info=_node_deployment_info,
            is_overridden=False,
            resource_type="TransportNode",
            display_name="Create-Transport-Node-IT",
        )
    
        assert ret_create is not None
        result_as_json_create = ret_create.json
>       assert result_as_json_create["node_id"] is not None
E       KeyError: 'node_id'

tests.integration.states.test_datacenter : test_present

ย  | TypeError: expected string or bytes-like object
    def test_present(vmware_datacenter):
        """
        Test scenarios for datacenter.present state run
        """
    
        # datacenter.present on an existing datacenter. No changes should be made.
>       ret = datacenter.present(name=vmware_datacenter)

tests.integration.states.test_nsxt_transport_zone : test_state_transport_zone_verify

Fresh VMC_CONFIG file

    def test_state_transport_zone_verify(nsxt_config, salt_call_cli):
        response_create = salt_call_cli.run(
            "state.single",
            "nsxt_transport_zone.present",
            name="create transport zone",
            hostname=nsxt_config["hostname"],
            username=nsxt_config["username"],
            password=nsxt_config["password"],
            host_switch_name="nsxDefaultHostSwitch",
            transport_type="OVERLAY",
            verify_ssl=False,
            display_name=_display_name,
            description=_description,
        )
    
        result_create = dict(list(response_create.json.values())[0])["result"]
        comment_create = dict(list(response_create.json.values())[0])["comment"]
>       display_name_response = dict(dict(list(response_create.json.values())[0])["changes"])["new"][
            "display_name"
        ]
E       KeyError: 'new'

tests.integration.modules.test_esxi : test_advanced_config

Integration test failed

    def test_advanced_config(service_instance):
        """
        Test advanced config on esxi host
        """
>       ret = esxi.set_advanced_config(
            config_name="Annotations.WelcomeMessage",
            config_value="testing",
            service_instance=service_instance,
            datacenter_name="Datacenter",
            cluster_name="Cluster",
        )

service_instance = 'vim.ServiceInstance:ServiceInstance'

tests/integration/modules/test_esxi.py:276: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/saltext/vmware/modules/esxi.py:639: in set_advanced_config
    return set_advanced_configs(
        cluster_name = 'Cluster'
        config_name = 'Annotations.WelcomeMessage'
        config_value = 'testing'
        datacenter_name = 'Datacenter'
        host_name  = None
        service_instance = 'vim.ServiceInstance:ServiceInstance'
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

config_dict = {'Annotations.WelcomeMessage': 'testing'}, datacenter_name = 'Datacenter', cluster_name = 'Cluster', host_name = None
service_instance = 'vim.ServiceInstance:ServiceInstance'

    def set_advanced_configs(
        config_dict,
        datacenter_name=None,
        cluster_name=None,
        host_name=None,
        service_instance=None,
    ):
        """
        Set multiple advanced configurations on matching EXSI hosts.
    
        config_dict
            Set the configuration key to the configuration value. Eg: {"Annotations.WelcomeMessage": "Hello"}
    
        datacenter_name
            Filter by this datacenter name (required when cluster is specified)
    
        cluster_name
            Filter by this cluster name (optional)
    
        host_name
            Filter by this ESXi hostname (optional)
    
        service_instance
            Use this vCenter service connection instance instead of creating a new one. (optional).
    
        .. code-block:: bash
    
            salt '*' vmware_esxi.set_advanced_config config_name=Annotations.WelcomeMessage config_value=Hello
    
        Returns:
    
        .. code-block:: json
    
            {
                "host1": {
                    "Annotations.WelcomeMessage": "HelloDemo"
                },
            }
    
        """
        log.debug("Running vmware_esxi.set_advanced_configs")
        ret = {}
        if not service_instance:
            service_instance = get_service_instance(opts=__opts__, pillar=__pillar__)
        hosts = utils_esxi.get_hosts(
            service_instance=service_instance,
            host_names=[host_name] if host_name else None,
            cluster_name=cluster_name,
            datacenter_name=datacenter_name,
            get_all_hosts=host_name is None,
        )
    
        try:
            for h in hosts:
                config_manager = h.configManager.advancedOption
                ret[h.name] = {}
                if not config_manager:
                    continue
    
                supported_configs = {}
                for opt in config_manager.supportedOption:
                    if opt.key not in config_dict:
                        continue
                    supported_configs[opt.key] = opt.optionType
    
                advanced_configs = []
                for opt in config_dict:
>                   opt_type = supported_configs[opt]
E                   KeyError: 'Annotations.WelcomeMessage'

Use `name` for the primary id of VMC states

VMC is currently using name for salt logging and not much else.

In salt it's normal for name in state functions to be the static identifier for the object being modified. Something like the file name you are chmod-ing, or the host you are checking for connectivity, or the command line string you are executing.

As an example, right now we have dhcp_profile_id as well as name.

We need to delete dhcp_profile_id from the function argument list, replacing its usages with name, then update the docstring of name to be something like: The dhcp profile id, any static unique string identifying the rule. Also used for the display_name by default.

update get_datastores to work with recursive folders

utils/vmware.py::get_datastore uses some fancy traversal to make finding datastores more efficient.

Unfortunately, when searching "everywhere" it doesn't search nested folders.

This should probably be made to work with nested folders. In the meantime, modules/datastore.py is using a wrapper function that gets all datastores and searches by name if nothing exists. This should work but be less efficient.

tests.integration.modules.test_esxi : test_esxi_get


    def test_esxi_get(service_instance):
        """
        Test get configuration on ESXi host
        """
        ret = esxi.get(
            service_instance=service_instance,
            datacenter_name="Datacenter",
            cluster_name="Cluster",
        )
        assert ret
        for host in ret:
            assert ret[host]["cpu_model"]
            assert ret[host]["capabilities"]
>           assert ret[host]["nics"]
E           assert {}

Sync up esxi.get to match core grains API

Currently the implementation of esxi.get in #129 does not quite match up with the API in Salt's grains.core. For now this is fine but at some point in the future we want to bring these in sync so that it's trivial for a user of this extension module to upgrade to either a deltaproxy approach or some other as-yet-to-be-developed style of accessing esxi grain-style information.

Installation instructions don't work on a VMware SaltStack Config appliance

I tried to follow the installation instructions listed here on a vRealize Automation SaltStack Config 8.6.1 appliance:
https://github.com/waynew/salt-ext-modules-vmware/blob/quickstart-docs/docs/quickstart.rst
Installation went fine, but the command to successfully test it didn't find the module:
salt-call vmware_datacenter.list

After some research we found out that the module directory has to be specified in the minion configuration file e.g. by creating /etc/salt/minion.d/module_dirs.conf file with following content:
module_dirs:

  • /usr/lib/python3.7/site-packages/saltext/vmware

After "service salt-minion restart" above command was executed successfully.

Static or relatively unchanging info should probably be grains

What it says on the tin. There are some bits of information that we're only exposing via a .get_info or something to that effect, that really should be grains.

I don't have an exhaustive list or even a single thing that I'm currently confident about, but... we should be considering grains.

tests.integration.modules.test_vm : test_ovf_deploy

salt.exceptions.VMwareApiError: ['Host did not have any virtual network defined.']
    def test_ovf_deploy(integration_test_config, patch_salt_globals_vm):
        """
        Test deploy virtual machine through an OVF
        """
>       res = virtual_machine.deploy_ovf(
            vm_name="test1",
            host_name=integration_test_config["esxi_host_name"],
            ovf_path="tests/test_files/centos-7-tools.ovf",
        )

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.