wahlnetwork / vester Goto Github PK

This does not prevent tests from running, its simply a reporting bug.

It could be resolved by replacing '$cfg.vcenter.vc' with '$global:DefaultVIServer' in 'VesterTemplate.Tests.ps1' where the '$InventoryList' is created.

Would this cause problems for the future if you allow multiple vcenter connections?

The new unit tests that are being written fail when a value of false is specified in the config.json file. Setting the value to true and later remediating that test works.

Comparing the existing tests that have false values that work, like $cfg.host.esxsyslogfirewallexception, I noticed that the value is Boolean, whereas the tests that are failing are simple strings. I would suspect that a string of false should match another string of false, however I'm not seeing that. I will continue looking into it but wanted to log the issue until it gets cleared up.

First learned of this in the VMworld session today and I'd like to help out with extending the capability. Right now, I have a need to continuously audit a 800+ host environment and part of that includes storage device configuration. More specifically the path selection policy, round robin IOPS, and perennial reservation status. Including these things in the current state of this project seems to require further scoping of host in config. I was thinking a scope such as host.storage and move along the lines of how esxcli is constructed.

Any thoughts on expanding this project to include those storage device configuration pieces? For my non-Auto Deployed hosts that run MSCS workloads, this would help get the information out of the host profile.

Currently, each test file checks one setting.
This proposal is to allow multiple settings to be able to be checked at once.

This would be beneficial for things like advanced settings or other large groups of settings that could be pulled all at once.

Personally, when you run Invoke-Vester you are running it against hosts, clusters, etc that should all be the same. Thereby, a new Config.json file should be specified for environments/hosts/clusters/etc that are different.

By grouping these large subset of settings all at once it should:

• Prevent us the labor of making each individual tests
• Prevent us from having to maintain those tests
• The ability to check a large array of values with very little effort.
• (There are probably more, but I'm not a good salesman/wordsmith)

For my NFS settings test, there are 23 settings, which would result in 23 individual tests.
vCenter has upwards of 840 advanced settings. I don't know if we will add more of those, but i wouldn't want to make 840 individual tests.

Expected Behavior

To entertain the idea of allowing multiple tests per test file

Current Behavior

One test per test file
VesterTemplate.Tests.ps1 is more or less expecting one test
(currently you get one failure test and you are out of the Try/Catch loop. More on this later)

Possible Solution

Gist link to possible solution

Test File - $Desired

$Desired will result in the value from the Config.json being imported as a 'PSCustomObject'. This is just host the ConvertFrom-Json cmdlet works.

(in the gist we convert $Desired into a 'hashtable' so we can compare it with $Actual)

Test File - $Type and $Actual

Quick side note: From my testing, $Actual should result with an output of a hashtable (meaning only 'key/name' and 'value')

I have $Actual result in an [ordered] hashtable
Note: $Type can be 'hashtable' or 'PSCustomObject' in this scenario. It doesnt matter. But i used 'hashtable' for consistency.

$All = [ordered]@{}
$NFS = Get-AdvancedSetting -Entity $Object | Where-Object {$_.Name -Match "NFS"} | Sort Name | Select Name,Value
$NFS | Foreach-Object { $All.Add($_.Name, $_.Value) }
$All

Which results in Config.json files like this:

                 "nfssettings":  {
                                     "NFS.HeartbeatFrequency":  12,
                                     "NFS.ReceiveBufferSize":  256,
                                     "NFS.VolumeRemountFrequency":  30,
                                     "NFS.LogNfsStat3":  0
                                     }

If we made the $Actual output as a 'PSCustomobject' using just this:
Get-AdvancedSetting -Entity $Object | Where-Object {$_.Name -Match "NFS"} | Sort Name | Select Name,Value
The Config.json output looks like this:

                 "nfssettings":  [
                                     "@{Name=NFS.ApdStartCount; Value=3}",
                                     "@{Name=NFS.DiskFileLockUpdateFreq; Value=10}",
                                     "@{Name=NFS.HeartbeatDelta; Value=5}",
                                     "@{Name=NFS.HeartbeatFrequency; Value=12}"
                                     ]

Note: $Type MUST BE 'PSCustomObject' in this scenario, or the results will be null

Test File - $Fix - AKA the Magic

One line to rule them all (in my scenario - and probably most that will use advanced settings):
Get-AdvancedSetting -Entity $Object -Name $DesiredObject.Name | Set-AdvancedSetting -Value $DesiredObject.Value -Confirm:$FALSE

We will get into this more below but when the test file is dot sourced, $Fix can be run and it references $DesiredObject for whatever the current value is (we loop through all of the $DesiredObjects individually)

So when a (multiple) test is run, it verifies $DesiredObject is equal to $ActualObject. If it isn't and you use -Remediate, it will push the current $DesiredObject.Value to the specific $DesiredObject.Name setting.

Now onto the VesterTemplate.Test.ps1

In my gist i have a few changes that i have issues open for, but regardless.

Everything is basically the same until line 104. I added an if/else that says:
if $Desired is a 'PSCustomObject' it's a test with multiple tests in it
else, do what we have been doing.

Pull in the results from $Actual

We get the results from $Actual and convert them to a 'PSCustomObject' (so we can compare them with $Desired as it too is a 'PSCustomObject').

Loop through all the objects in $Desired

First thing is i had to move the It in here.
Otherwise, the very first test that failed, Pester would no longer proceed with the rests of the tests.
and it just considered them to all be one large test, instead of individual ones.
Note: i added : $($DesiredObject.Name) after Title in the It clause so we could better see the individual test names.

First thing in the loop is to get the value from $ActualObjects (the 'PSCustomObject' from $Actual) that has the same name of the current $DesiredObject (the object we are checking against currently).

Now that we have our $DesiredObject and $ActualObject

If $DesiredObject is null, we still want to make sure it is null
else we compare the two objects. Since they are both 'PSCustomObjects', it runs Compare-Object with the -Property Name,Value specified.

Everything else is the same.
if Compare-Object fails, it goes into the Catch block.

(I think that covers everything. it's kind of late here after writing all this).
As a side note, i have Write-Host in there just for testing/debugging. We can for sure remove those.

Steps to Reproduce (for bugs)

Not really a bug

Context

Accomplish: Simplicity of test writing for large subsets of settings

Your Environment

• Operating System: Windows 2012 R2
• PowerShell version: 4.0
• PowerCLI version: 6.5 Release 1 build 4624819
• Vester module version used: 1.0.0
• Pester version: 4.0.2

Currently these tests are just seeing if the CPU/Memory limit is set to unlimited.
And the remediate is just to allow it to be set to unlimited

This isn't anything major, it is just to recommend allowing users to specify the limit or to specify that they want it to be unlimited.

These are my suggested updates:
Memory: https://gist.github.com/Midacts/8f0c03394325f598eb6b46a4def40729
CPU: https://gist.github.com/Midacts/b010caa61489bd31636dfae3a0f63ebc

Need to create some method to validate:

1. That the config file has valid params defined
2. That the param values are within some sort of range

Optionally, it would be nice to accept a range of inputs and auto-generate a config.

I'm curious what everyone thinks is remaining to retire the current master branch and focus efforts on the module branch (thus making it master). I tend to support a "just enough functionality" model in which we start with a few tests and grow from there, even if we don't have test parity with the current master branch.

So - what are the big blockers preventing us from using the module today? @cars @equelin @brianbunke @mattallford

Also, I'm not a huge fan of the Project tab. Seems like too much overhead.

Currently if $Desired is $null, it skips testing whether the value is still $null.
Current

We should wait for #114 to be resolved, but i have been testing with this and it works:
https://gist.github.com/Midacts/3f6f3304dcfad54765f8faa11ecb05d8

It adds the check to make sure $null still equals $null inside of the It blocks Try statement.

This is more of a FYI and making sure that this is the best way to handle it.

Currently $Desired = $cfg.... is the only time $cfg is allowed to be referenced in a test files do to these
lines (specifically the lines referencing $Cfgline).
(and it then being passed to Set-VesterConfigValue.ps1)

I was able to work around this in Test Files by doing things like this:

$Save = Get-ChildItem Variable: | Where-Object {$_.Value -Match "host"}
$DCUIAccess = $Save.Value.Host.dcuiaccess

Are you guys okay with using the above types of references in test files?
I think it is easier than refactoring parts of New-VesterConfig.ps1

We're long overdue for a PowerShell Gallery version publish. I plan to:

• Resolve #149
• Add a CHANGELOG.md file for release notes
  • Finished this tonight, but I'd like to do #149 first
• Update the version in Vester.psd1 to 1.1.0
  • Trying to follow SemVer
• Get release notes into Vester.psd1 somehow
  • Could manually copy the text in
  • Thought there was a slicker way to do this, but June Blender's sweet article from last year seems to have vanished from the Sapien site. Want to follow up on that
• Manually publish to the PS Gallery as v1.1.0, leaving 1.0.1 visible

Making this public in case there are any questions, and if we need to refer back later.

Hey there!

Vester is fairly stable right now, with a version 1.0.1 published on the PS Gallery! (read: I'm done drastically screwing with things for the time being)

Now seems like a pretty good time to ask everyone to try it out again, and to please speak up if you like or don't like certain things.

If you're not sure how to wrap your head around everything...

1. http://vester.readthedocs.io/en/latest/
2. help about_Vester
3. I believe @chriswahl will be posting a blog soon
   • vSphere Configuration Management with Vester
4. I've got documentation to review/update and a couple blogs on the changes to bang out as well
   • Part 1 - Introducing Vester
   • Part 2 - Tips & Tricks
   • Part 3 - Write (& Contribute!) Your Own Tests

Keep an eye out for those if you're interested, but in the meantime, try this on for size:

# Assuming you've used the PowerShell Gallery. You could just DL the GitHub zip if preferred
Install-Module Vester
Import-Module Vester
# Module requirements "Pester" & "VMware.VimAutomation.Core" automatically load into the session

Connect-VIServer your.vcenter.com

# Do you care about Distributed Switches?
# PowerCLI doesn't do implicit module loading yet, so manually import any other needed modules
Import-Module VMware.VimAutomation.Vds

# This step requires you to be running PowerShell as admin
# Because default PS Gallery install location is inside C:\Program Files\
New-VesterConfig
# Config.json now exists in the module's \Config folder

# By default, run all included `.Vester.ps1` tests using the newly created config.json
Invoke-Vester

If you have general feedback, feel free to reply on this issue ⬇️ , or let us know in the VMware Code #vester Slack channel. If you find a bug or have an enhancement request, please also feel free to just open a new issue.

Thanks! Hope you enjoy!

Am I missing something or the second call of Get-Cluster is redondant?

Process {
    # Tests
    Describe -Name 'Cluster Configuration: DRS Settings' -Tags @("vcenter","cluster") -Fixture {
        # Variables
        . $Config
        [string]$drsmode = $config.cluster.drsmode
        [int]$drslevel = $config.cluster.drslevel

        foreach ($cluster in (Get-Cluster -Name $config.scope.cluster)) <<<<<<<<<<< Look here!
        {
            It -name "$($cluster.name) Cluster DRS Mode" -test {
                $value = (Get-Cluster $cluster).DrsAutomationLevel  <<<<<<<<<<< And here!
                try 

It could be replace by $value = $cluster.DrsAutomationLevel

Is there a need for the user to define -Config? I don't think it's a stretch to call it directly from "$PSScriptRoot\Config.ps1" in each test, and scold the user in the event of moving it out of the same directory.

Just to be up front about the real reason behind opening this issue. The current Invoke-Pester examples with the nested hashtables are ugly, and impossible to discover if you don't know to scroll down the Github readme. Removing the need for both variables to be supplied in that fashion would be amazing, but this issue is focused on the easy one. :)

The JSON file lists "0" for the VDS ProtoLink when the correct value is "CDP"

Need unit tests for the various tests found within the module. The lead contributor on this is @brianbunke but marking this as help wanted to attract the various crawler engines out there.

Are you interested in adding non-remediating tests to the project, or would you prefer that -Remediate stays "global" against the test suite? I think any admin could be interested in things like "network adapters that aren't VMXNET3" or "disks that aren't thin provisioned," but those examples either incur downtime or aren't easy to fix in an automated fashion. Just wondering about your ideal scope. Thanks!

VMware.PowerCLI is on the PowerShell Gallery now. We should tweak Vester to recognize the differences:

• VMware.PowerCLI can now be referred to as a RequiredModule (like we do for Pester) instead of an ExternalModuleDependency
  • Update Vester.psd1 L110 & Vester.Tests.ps1 L43
• PowerCLI now supports implicit module loading when commands are called, so no more manually importing modules
  • Update VesterTemplate.Tests.ps1 L79

Self-assigning, and will look at this soon. Feel free to comment if any questions come to mind.

running powercli 5.1 on windows 10 x64 using a simple script;
Add-PSSnapin VMware.VimAutomation.Core

write-host -ForegroundColor Green "Connecting to VirtualCenter Server"
Connect-VIServer -Server vcenter -Session $sessionId -ErrorVariable currentError

Invoke-Pester .\

[-] Error occurred in test script 'C:\Users\u376504\Desktop\vblock capacity charts\Vester-master\Configs\Config.Tests.ps1' 37ms
ScriptRequiresException: The script 'Config.Tests.ps1' cannot be run because the following modules that are specified by the "#requires" statements of the script are missing: VMware.VimAutomation.Core.
at , C:\Program Files\WindowsPowerShell\Modules\Pester\3.4.0\Pester.psm1: line 297
at Invoke-Pester, C:\Program Files\WindowsPowerShell\Modules\Pester\3.4.0\Pester.psm1: line 310
at , : line 6

seems to be no module version of VMware.VimAutomation.Core? can we just use the snapin

Running Invoke-Vester -Remediate -Whatif allows some changes to execute. If $Fix code for a test uses cmdlets without -WhatIf support, real changes may be made. For example, VIB-AcceptanceLevel.Vester.ps1 is susceptible to this problem due to its use of Get-EsxCli.

Rather than rely on each devs test to handle Should execution properly, I'm wondering if some wrapper logic would be appropriate before executing the $Fix block. But I'm not sure how Pester handles that exactly so...

Documentation should be easy to write, contribute to, and publish and allow for auto-generated reference documentation.

Expected Behavior

Contributors to documentation should be able to write that documentation in markdown, the de facto documentation markup standard for PowerShell. They should also be able to update the documents without having to fiddle too much with configuration. The documentation should be deployable to ReadTheDocs or another static site via CI and include testing. The documentation should also support exporting the existing reference documentation for public functions so that that documentation is not being updated in multiple places.

Current Behavior

Currently, the docs are manually written in ReStructured Text (RST) and published to ReadTheDocs. No reference documentation is included.

Possible Solution

Use mkdocs for documentation, allowing the contributors to write docs in markdown, theme and publish to readthedocs with minimal effort. Reference documentation can be generated from the module using PlatyPS.

Context

This would make contributing to documentation easier.

Your Environment

• Module version used: Vester 1.0.1
• Operating System: Windows 10.0.14393
• PowerShell version: 5.1.14393.693

Add a public function to the module for creating the necessary skeleton file for a Vester Test.

Expected Behavior

As a user of the Vester module, I should be able to call a command to create the minimum necessary code for editing into a vester-functional test so that I don't have to copy existing tests over and edit them.

Current Behavior

Currently, users copy existing tests and modify them.

Possible Solution

Create a new function which provides the functionality and existing code style used by the project.

Context

This would make writing tests easier/faster for people and help standardize the boilerplate code without requiring people to hopefully copy things correctly.

Your Environment

• Module version used: Vester 1.0.1
• Operating System: Windows 10.0.14393
• PowerShell version: 5.1.14393.693

What do you think about creating a slack channel for Vester ? It'll allow to discuss more freely than on GIthub's issues and emojis are way cooler on slack (:parrotparty: !!!)

VMware {code}'s slack could be a good choice.

See PR #28.

This issue is to discuss pros/cons of:

1. Converting Vester to a PowerShell module
2. Introducing Invoke-Vester

Pros:

• No more hashtables to pass -Remediate and -Config
• Free to order tests (and rearrange) as desired
  • In reference to #12, where Config.Tests.ps1 was running first because of #AlphabeticPrivilege
  • And can prompt user to define Config.ps1 if Invoke-Pester is called against a default file
• Can call VMs/Hosts/whatever once up front, instead of within every test
• Support for -WhatIf
• Tags can auto-suggest/complete thanks to ValidateSet
• Better docs. Comment-based help on Invoke-Pester, help about_Vester, etc.

Cons:

• Have to manually duplicate each Pester parameter? I think
  • And would have to manually update when Pester adds new features
• ^ That's a really big con

Examples of use:

Invoke-Vester
Invoke-Vester -Remediate -WhatIf
# Note auto-complete when adding tags
Invoke-Vester -Tag cluster,host -ExcludeTag nfs

The NFS test, instead of testing that each actual value perfectly matches its Config counterpart, just makes sure that the actual value matches any one in the array of the six configured numbers. You can see this in the thrown error of a failed test:

[-] esxi01 NFS Settings 815ms
  Expected: {20 30 256 32 1536 12}
  But was:  {8}
  43:                     $value | Should Be $compare
   at <ScriptBlock>, C:\Github\Vester\Tests\Update-NFS.Tests.ps1: line 43

A fix should correctly loop through the config values, testing them 1:1 against actual. (or vice versa)

This is super low priority, and maybe should just sit idle until the other NFS performance improvements are merged.

So even though I can now use tags to exclude the NFS tests in my environments. I'm wondering if NFS seems really slow for others that do use it?

I've been playing with the NFS Test and it's averaging 65-70 seconds (yes seconds not ms) per host in an environment that doesn't use NFS. Wondering if same is true for others? I've made some changes to the test that seem to get it down 10-12 seconds per host but wondering if this something specific to my environment or not?

-CRT

In each hosts tests (like Update-DNS) we do not look at the cluster scope

foreach ($server in (Get-VMHost -Name $config.scope.host)) {
...
}

Instead, I propose do to something like:

foreach ($server in (Get-Cluster -Name $config.scope.cluster | Get-VMHost -Name $config.scope.host)) {
...
}

What do you think about that ?

PS C:\> Install-Module Vester
PS C:\> Import-Module Vester

Import-Module : The specified module 'Vester' was not loaded because no valid module file was found in any module directory.
At line:1 char:1
+ Import-Module Vester
+ ~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (Vester:String) [Import-Module], FileNotFoundException
    + FullyQualifiedErrorId : Modules_ModuleNotFound,Microsoft.PowerShell.Commands.ImportModuleCommand

Upon inspecting my default install directory, C:\Program Files\WindowsPowerShell\Modules\, I noticed two things:

1. The top-level Vester folder was included in the gallery publish, causing the above error to replicate from #63
   1. Hopefully we can publish one level down? No need for that top level to be included, if possible
2. There is no version folder. My other modules look like C:\Program Files\WindowsPowerShell\Modules\Pester\3.4.3\{contents}
   1. Maybe this fixes itself once we iterate past 1.0.0?

I have not yet published anything to the gallery, so I'm pretty helpless here.

When a specific test fails, readers of the output should be able to see what the actual discovered setting was and be able to understand why it failed the test to some extent.

Expected Behavior

When reviewing a test failure, I get back the specific setting currently found for that test/target.
This helps me to drill down into the problem without having to go read the tests to see what it is doing.

A similar construct for this is used by DSC - you have the option to get information about infrastructure, the option to test if the infrastructure is in compliance, and the option to set the infrastructure such that it becomes compliant.

Vester lets us test and set but not get it looks like.

Current Behavior

Using pester v4.0.2, all test failures return the same information:

      Expected: value to be empty but it was { }
      at line: 94 in C:\Users\lombarm\Documents\WindowsPowerShell\Modules\vester\1.0.1\Private\Template\VesterTemplate.Tests.ps1
      94:                         Should BeNullOrEmpty

Using Pester v3.4.6, all test failures return a similarly unhelpful response:

Expected: value to be empty but it was {@{InputObject=True; SideIndicator==>}}
   94:                         Should BeNullOrEmpty
   at <ScriptBlock>, C:\Users\lombarm\Documents\WindowsPowerShell\Modules\Vester\1.0.1\Private\Template\VesterTemplate.Tests.ps1: line 93

Neither of these options is particularly useful for troubleshooting.
Currently, if I need to investigate, I have to go run commands on my own after reviewing the test code to see what it was checking for specifically.

Possible Solution

• Modify `VesterTemplate.Tests.ps1 to return useful errors
• Look to create specific error messages from within the tests themselves
• Add a get type option for each test that returns the inspected value; pass that through the template tests on a failure.

Context

I was hoping to be able to use the vester test suite + reportunit to provide a simple HTML5 dashboard for our engineers to be able to review, updating the dashboard on a schedule or letting them interactively pull the information.

Unfortunately, the value of the dashboard is largely around seeing the failing tests and being able to extract value from them. For some tests, this isn't a problem - I can guess that if a VM fails the CDROM ISO File Test what's going on there, but for others it means I have to go find the test, see what/how it's checking, and then do some further investigation just to find out why the test failed.

It's better if we can give the engineers the immediate information and context to go solve problems wherever possible.

Your Environment

• Vester module version used: 1.0.1
• PowerCLI version: 6.5.1.5377412
• Pester version: 3.4.6 / 4.0.2
• Operating System and PowerShell version: Windows 10.0.14393, PowerShell 5.1.14393.1198

Hello,

Using Invoke-Expression with a path with spaces like C:\Program Files\ throw an error.

I think that you can correct this by escaping spaces that might be in $vars:

# Pull in vars
$vars = (Get-Item $PSScriptRoot).Parent.FullName + "\vars.ps1"
Invoke-Expression ($vars -replace ' ', '` ')

If you are interested, I can do that for you with a PR.

The scenario given in example 3 doesn't work. My real-world code:

$syslog = gci .\Vester\Tests -Recurse -Filter '*syslog*'
Invoke-Vester -Test $syslog

That code should run two tests, one for each syslog test (Syslog-Firewall & Syslog-Server). Instead, it was only running Syslog-Server for me.

Creating this just to log it. Self-assigning, and will investigate soon.

Need to write tests for ...

• DRS enabled
• DRS automation level
• DRS mode

Currently, only three types are documented in the tests/code for types consumable by json: bool, string, and int.

Expected Behavior

The available types should support arrays of string and int and perhaps other types as well.

Current Behavior

In a test you can specify string[] as a type but this is undocumented even though it works.

Possible Solution

Decide on which types are acceptable and document them; at least document arrays.

Context

This would enable people to write better/more specific tests, especially in cases where the type should be an array of strings. This functionality already exists, it's just undocumented and people tend to assume that only documented things are possible.

Your Environment

• Module version used: Vester 1.0.1
• Operating System: Windows 10.0.14393
• PowerShell version: 5.1.14393.693

I see that Update-Template.ps1 is no longer current, but still referenced in the documentation here http://vester.readthedocs.io/en/latest/contribution.html. Can we get a quick update to the docs so it is apparent the broken link is a known issue, without digging through the other multi-paragraph issue?

Also, let me know the appropriate etiquette for this kind of thing - am I supposed to just fork and submit a pull request to fix it myself?

So, 1cebb28 ("Move VMware module to external dependencies for PoSh Gallery") introduced an interesting problem.

When VMware.VimAutomation.Core was a "Required Module," I could open a console and do this:

Import-Module .\Vester
# Note that VMware.VimAutomation.Core automatically loaded as part of the Vester import
Connect-VIServer vcsa.home.lab

Now, after moving to an "External Module Dependency," it doesn't work:

Connect-VIServer : The term 'Connect-VIServer' is not recognized as the name of a cmdlet, [blah blah blah]

Furthermore, Door A:

Import-Module .\Vester
Connect-VIServer vcsa.home.lab

Connect-VIServer : The term 'Connect-VIServer' is not recognized as the name of a cmdlet, [blah blah blah]

# Fine, I'll manually import
Import-Module VMware.VimAutomation.Core
Connect-VIServer vcsa.home.lab

Connect-VIServer : The term 'Connect-VIServer' is not recognized as the name of a cmdlet, [blah blah blah]

And Door B:

# Guess I'll try bringing in PowerCLI first?
Import-Module VMware.VimAutomation.Core
Import-Module .\Vester
Connect-VIServer vcsa.home.lab

# lol jk I'm fine

Any quick suggestions on how we should ideally handle this within the .psd1 file?

In a previous PR I submitted NFC-UseSSL.Vester.ps1 that checks if config.nfc.useSSL is set to True or False. The odd thing about this setting is that the key its checking doesn't exist by default, and this may be a setting that few people care to enable. My new idea is that I add a few additional comments to the test itself explaining why it exists and how it can be used (what to add to config.json, etc.) as well as modifying the new-vesterconfig.ps1 file removing the following line so that value doesn't exist in the generated config.json file.

nfcusessl = $vcenterHash['config.nfc.useSSL']

Thoughts? I don't want to force the hardening guide standards on people that don't want them and so far this is the only test that results in an error after the initial config.json file is generated followed by invoke-vester

Invoke-Vester could be a lot faster. You all are really polite for going this entire month of March trying out New Vester and not commenting on the performance. 😄

Currently, when each new test is ready to run, it checks its scope and then pulls the objects it will apply to. So, if a VM test is about to run, part that test's prep is to Get-VM and then ForEach the test against all applicable VMs.

This sucks when running the full test suite. For many VM tests in a row, you have to Get-VM at the beginning of each one.

There are two options:

1. Current implementation
2. Get- all inventory objects once, store them in variables, and pass them into each test

I did 1 initially because 2 means even if you're only running Host tests, you still have to Get-VM, Get-Cluster, etc. I'm over caring about that at this point. (and I may be able to weasel around that anyway, haha)

As revealed with the HA tests, there are certain types of tests that end up being dependencies of themselves. As an example:

• Issue1: HA should be on - as such, other settings should be configured such as Admission Control and Heartbeat Datastores
• Issue2: HA should be off - as such, other settings should not be tested because they are not applicable such as Admission Control and Heartbeat Datastores
• Issue3: HA should be on but we don't care about the other settings such as Admission Control and Heartbeat Datastores

Issue1 is relatively easy and is how Vester works today.

Issue2 could be solved in a few different ways. Probably the easiest is to simply wrap up the sub-tests (that check Admission Control and whatnot) with a boolean test to see if the user wants HA on in the first place. If HA enable $true, we run all the tests. If $false, we only check the state of HA.

Issue3 is a bit more interesting. There's a few ways we could handle this. One is to allow people to comment out configurations that aren't desired or perhaps set the value to $null. The Config.Tests file would need to allow for this. The .Tests file would need to then handle configs that are $null or missing and be told to skip those tests.

What say you @cars @equelin @brianbunke ?

Caveat: I don't use NFS in my environments.

When I run the following command, I don't see NFS.DeleteRPCTimeout anywhere. 6.0 U1 & U2.

Get-VMHost '*01*' | Get-AdvancedSetting | Sort Name

That key is one of the default NFS configuration settings in Config.ps1. Is it there, and I'm not cool enough to know about it? Can @chriswahl or anyone confirm that the key is returned when they run the above command on their environment?

I'm searching a way for doing the config management of our vCenters and vester looks very interesting. After doing some test I have some questions:

• In the usage Instructions is described, that the config is in a config.ps1 but when I generate the config it's creating a json file. Is the documentation wrong?
• How can I disable some tests for a specific config? Example I don't would like to do all VM tests, so I tried to change the scope in the config.json (null, remove, empty) but I'm always getting an error. Get-VM : 12.01.2017 14:47:08 Get-VM VM with name 'null' was not found using the specified filter(s)
• Tags: In the example tags are used with "Invoke-Pester .\Vester\Tests -Tag host -ExcludeTag nfs", but invoke-vester is not supporting tags. So how I can use tags for invoke-vester.
• I would like to create new custom tests, in the documentation you are referring to a template, but this file doesn't exist. Do I have just to write a new ps1 file in vester/tests for the needed category to start?

Expected Behavior

When running a vester test with the -remediate flag, the command should re-execute the initial condition test after the remediation action - this ensures that the test results are actually as-expected post-remediation. This should be true for all published/included tests.

Current Behavior

At least some tests do not rerun the comparison post-remediation.

Possible Solution

Update the template to rerun the test again - this means test-writers don't have to think about it, they get it for free from the template.

Context

This would ensure that if a remediation completes without throwing an error the test will still fail if the conditions specified are not met.

Your Environment

• Module version used: Vester 1.0.1
• Operating System: Windows 10.0.14393
• PowerShell version: 5.1.14393.693

Attempting to run the code with PSVersion 4.0 on Win Server 2012 R2. Line 146 in invoke-vester needed to be updated to "$cfg = Get-Content $ConfigFile -raw| ConvertFrom-Json" for conversion to complete. -RAW was not needed when running on Windows 10 with version 5.0 but was compatible with both versions.

For some interesting reason, the JSON file created by New-VesterConfig is generating an int value for drsmode (based on $cluster.DrsAutomationLevel). The value being pulled from vCenter is definitely FullyAutomated so I'm guessing the ConvertTo-Json cmdlet is to blame.

Can anyone else validate this behavior? Here's a snipet from my JSON config.

"cluster":  {
                    "drsmode":  1,
                    "drslevel":  3,
                    "haenable":  true

/cc @brianbunke @equelin

I've submitted a PR (#34) for testing HA configuration but it needs to be improved to be usable. HA as a lot of options and we'll have to check the most of them.

Here are a list of functionality that should be tested:

• HA Admission Control
• Host Monitoring
• Datastore Heartbeats
• VM component protection
• Advanced Settings (streched cluster, etc...)

Is something missing ? Do you have any ideas/advices about how to test all of this ?

Now that we have a full fledged module for Vester it's time to get the README.MD file updated. Most of this has already been done by @brianbunke in the about_help file and within the code itself. Just need it to all match on the project page + code itself.

Would also really like to get the help put into Markdown files so that we can start using ReadTheDocs to auto generate docs.

As @boyler05 pointed out in #91, we never got around to porting all of the old tests into the new format. I'd prefer others work on this, so I'm labeling it up for grabs. If you're interested, please speak up and I'll assign. (Random GitHub question: Can people w/o write access to the project self-assign issues?)

Happy to answer any questions and provide guidance if needed.

EDIT: This needs more info.

Here's a snapshot of the repo in the old test format: https://github.com/WahlNetwork/Vester/tree/47f400a76738d28308881f0573526ebc14f3885e

I believe all the tests pointed out existed at that time. You would need to:

• Find a test file that hasn't been ported yet
• Determine how many tests exist inside that file
  • Right now, test files are meant to test just one thing, but they could perform multiple checks at that point
• Review Config.ps1 (for at least the $Desired value) and Config.Tests.ps1 (for at least the $Description) to grab related values
• Copy an existing .Vester.ps1 file and update the variables
• Drop it in the test folder matching the test's scope

At this point, we generally name tests "Category-Specific.Vester.ps1". We haven't implemented anything that cares about that right now, but it's a decent way to group tests together, especially during a Get-ChildItem.

First off, thanks Chris for this great project. Kudos!
Second, the current vester-master appears to have two typos in the "Config.Tests.ps1" file. I get this when I test the config file:

Describing Config file validation
 [+] Is reading a valid config file 48ms
 [+] Properly supplies variable $config 25ms
 [+] Contains proper settings for .vcenter 46ms
 [+] Contains proper settings for .scope 481ms
 [-] Contains proper settings for .cluster 22ms
   Expected: {3}
   But was:  {4}
   52:             $config.cluster.Keys.Count | Should Be 3
   at <ScriptBlock>, E:\vester\Configs\Config.Tests.ps1: line 52
 [+] Contains proper settings for .host 3.1s
 [-] Contains proper settings for .vm 29ms
   Expected: {6}
   But was:  {8}
   93:             $config.vm.Keys.Count | Should Be 6
   at <ScriptBlock>, E:\vester\Configs\Config.Tests.ps1: line 93
 [+] Contains proper settings for .nfs 51ms
 [+] Contains proper settings for .vds 27ms

Currently when you run the New-VesterConfig cmdlet, the config file it generates is not sorted. I am just proposing that we sort it (so it is easier to read and maintain).

I tested this and we should just be able to add the following lines to the New-VesterConfig.ps1 file to fix the issue.

$Sorted = $config.$Scope.GetEnumerator() | Sort Name
$Sorted
$config.$Scope = [ordered]@{}
$Sorted | Foreach-Object { $config.$Scope.Add($_.Name, $_.Value) }

This would replace line 249 and add the 3 additional lines.

Using [ordered] is the key as it makes your arrays be sorted in the order they are added.

Currently, filtering each scope consists of Get-VM -Name YOURINPUTHERE & similar for other scopes.

That's ok, but leaves room for improvement. For example, Get-VM -Tag YOURTAGHERE would be a pretty sweet option.

Is that enough? Should we allow people to fully craft their own queries? That would allow Get-VM | Where Name -notlike 'abc'. Get-VM -Datastore YOURDATASTOREHERE is also a thing. Get-VM may have another useful parameter, or more useful parameters in the future. Who knows how many useful filtering parameters exist on the rest of the scopes right now.

However, currently, we're actually doing:

Get-Datacenter -Name $cfg.scope.datacenter -Server $cfg.vcenter.vc |
    Get-Cluster -Name $cfg.scope.cluster |
    Get-VM -Name $cfg.scope.vm

Will people know to replicate that, or realize that parent scopes may not apply? We could display, on-screen, the results of your query (like after you set it in New-VesterConfig), but some people have thousands of VMs, so I've held off on doing that.

I personally think inventory tags are sweet, and enough to solve this problem. But I'm interested in your feedback before I get started!

Defining globally-scoped variables always makes me feel a little dirty. I dug into the code and think a quick change in each file would remove the current need.

Delete-Snapshot, for example:

Process {
    # Variables
    Invoke-Expression -Command (Get-Item -Path $Config)
    [int]$snapretention = $global:config.vm.snapretention

    # Tests
    Describe -Name 'VM Configuration: Snaphot(s)' -Fixture {
        foreach ($VM in (Get-VM -Name $global:config.scope.vm))

...would become:

Process {
    # Tests
    Describe -Name 'VM Configuration: Snaphot(s)' -Fixture {
        # Variables
        . $Config
        [int]$snapretention = $config.vm.snapretention

        foreach ($VM in (Get-VM -Name $config.scope.vm))

Of note:

1. The Invoke-Expression/Get-Item can just be dot-sourcing to grab its $config variable
2. Variables defined outside of the Describe block aren't allowed to sit at Pester's lunch table

I'm happy to submit the proposal as a pull request if there's agreement.

Vester currently only has tests for distributed switches in the Network folder. Standard switches should be supported.

Expected Behavior

Additional scope for VSS, to live side-by-side with VDS, ESXi, etc. Additional subfolder below .\Tests to house all VSS-specific tests.

Current Behavior

No current VSS framework. VDS tests live in the "Network" folder; VDS/VSS should not live in the same folder.

Possible Solution

• Add vss to $config.scope in New-VesterConfig
• Add a VSS folder for Vester test files
• Rename .\Tests\Network to .\Tests\VDS, and then add VSS
  • Keeping in mind that this is a minor breaking change

Context

VSS support was requested in chat today. The framework should allow for simple contribution of new Vester tests for VSS objects.

Any comments are welcome; with no feedback, I expect this to be implemented as described above.