leeej84 / -deprecated---powerscale Goto Github PK

Hi Lee...I was trying to work out why it wasnt scaling up for me and I think you'll find this should be
($machineScaling -eq "Sessions") not Session

Apart from that it is working brilliantly.

Steve

https://github.com/leeej84/PowerScale/blob/4b20f7c1730504ff5ff9cff5ac0ca1921f04f35a/Decision%20Making.ps1#L1219

Adding the ability for admins to list holidays by date; this would allow the system to be setup to treat the identified holidays like non-business days. Thereby reducing mistakes when an admin forgets to update the config just prior to the holiday. In larger systems, this kind of an error can be thousands of dollars/pounds.

Hi,
Our admin Users don’t have idle timeout policy set so if a adminuser doesn’t log out powerscale would not shutdown the machine.
Would be nice to handle it with the script, maybe shutting down machine if the idle time of users exceed x hours.
what do you think about?

It would be very useful if we could specify the path and filename of configuration file (config.xml) as parameter in these scripts :

• Create Config File.ps1
• and Decision Making.ps1

In addition, this would allow to have several configurations depending on the delivery groups, machine catalog, tags,...

Regards

Hi, the current Decision Making.ps1 contains diff information that shouldn't be there. For instance on line 1171:

<<<<<<< HEAD
    $machinesToPowerOff = $machinesOnAndNotMaintenance | Sort-Object -Property SessionCount | Select-Object -First $($numberMachines)
=======

I guess this should not be in the master branche? Can you please provide the correct ps1?

Reviewing the dashboard and the "In Hours Machines:" shows the current host count rather than the parameter value. So if set to 35 with additional scaling so it can grow to 40, 45 or higher. When it scales up then "In Hours Machines:" shows current running count rather than 35 as saved in the config file.

It would be amazing if your script could ensure that a single machine doesn't run for more than a specified time. There is a chance that a machine may go into maintenance mode and come out of maintenance mode without shutting down, this could happen for quite a while without a full shutdown. Maybe add a tag based on uptime that would drain and shutdown before removing the tag.
I like to have my machines reboot at least once a week to make sure they are updated to the latest PVS image and this feature would allow me to do that. Without PowerScale, I use a script to put the machines in maintenance mode and then reboot them after the users have drained off.

Hi @leeej84

I was trying to work out why it kept booting my disconnected users, even though I had respectDisconnected set to true, after some trouble shooting, I realised that Decision Making.ps1 was missing the line
$respectDisconnected = $configInfo.respectDisconnected

Hope this helps some others.

Thanks for such a brilliant tool.
Steve

Thanks for the work on this tool! We are liking it. Graph help/ideas:

1. The data seems to be growing and growing in the graphs. Is there an easy way to truncate the data set in a setting someplace that we are missing? As more data points are in the same graph it's very difficult to see what's really happening with numbers. Getting to that "one" data point is challenging. :)
2. Or better yet, give us a way to zoom-in on a section of the graph by applying a filter on the days/hours we would like to review.

Create a variable that is set in the Create Config File.ps1. The variable is Yes/No for running additional scaling based on business days.

$AdditionalScaleNonBusDay = Yes/No

Then a small bit of code to implement. (Small may be relative ... :) )

When viewing the dashboard in v2.9 with a 30 day dashboard retention parameter, the individual links go out to the right beyond the rest of the layout on the page. They should wrap if they hit the end of the space.

Looks like there is an issue with getting load index and session numbers during the performance measuring jobs.

Lines 934, 935 and 1017, 1018
LoadIndex = (Get-BrokerMachine -AdminAddress $controller | Where-Object {$.DNSName -eq $computer}) | Select-Object -expand LoadIndex
Sessions = (Get-BrokerMachine -AdminAddress $controller | Where-Object {$.DNSName -eq $computer}) | Select-Object -expand SessionCount

These seem to need -MaxRecordCount added to the Get-BrokerMachine call to ensure you pull all records. Without it we saw improper calculations for index and sessions.

HI!
If I wanted to reach out..there is a mention to slack...slack says "please contact admin"
So my q is : who and how to access?

Circular logging is the method used by PowerScale and it helps isolate the daily files and allows for clean-up by setting a retention time. We've noted that there is some very good information in the log files that we'd like to monitor for and alert on. However, many log monitoring tools don't work well (if at all) with circular logging. Would you consider updating this so circular logging is not required? With large drives these files can grow fairly large without running out of drive space.

Examples of great log info we'd like to monitor:

1. CPU, Memory, Index and Session averages - Can help by alerting when our parameters aren't configured to handle a huge rush of users coming into the system which can create rapidly increasing loads without enough hosts resulting in a poor user experience.
2. Showing errors when communications fail - This is great since if communications between PowerScale and the control plane are not working none of the stats and scaling features will function. It just maintains status quo which can leave the farm undersized.
3. Suggested as part of this: If circular logging is off the log location could start running out of disk space at some point in the future. The script could also check on disk usage parameters: Disk size, available space, % available Then, with that written into the log file the same monitoring and alerting process could alert on this. (Of course, normally disk utilization is available from other sources so this isn't as helpful as #1 and #2.

400 machines with a target of 60. Where the load index creeps below the threshold PowerScale will attempt to shutdown 340 machines!

Request to have a scaling down bu a specific number to control the drop and its not like a "cliff"

Hi Lee,

I'm having trouble solving an issue, which I think could be solved. It is not occurring very often, but once or twice a week.

When the script needs to scale down, because of decreasing number of users (I use the average sessions for scaling). The script puts a machine in Maintanance to drain those users. So far so good.

If right after I need to scale up because of an increase in users, the script turns on a machine.

What I would rather have is that the machine that is in maintanance mode, is to get out of maintanance mode. This in stead of a new machine being turned on. For this the machine with the most sessions still alive should be get out of maintanance first, etc.

Using this procedure we could save even more on Azure consumption.

Something to think of with a new Realease, hopefully you do know how to solve this.

Jordy

I put this in prod last night and looks great! One thing I was thinking about and probably the way I am using it but I have the following configs. (Note: I have 17 servers total)
$inHoursMachines = 17
$outOfHoursMachines = 1

So, when the businessCloseTime hits, all but one shuts off if session count is 0. Or puts in maintenance until drained. Looks good. But I would like to ensure it is a different set of 17 each day. Meaning if all 17 servers are at 0 sessions at the EOD then the same machine looks like it would be staying on. I would like to make it a random 16 that get shut off to hopefully cycle through them if that happens.

I was looking to add something like a | Sort-Object {Get-random} | Select -first $($numberMachines) and believe it would be somewhere around this location of the script (Around line 1356 and 1378

If ($machinesOnAndNotMaintenance.DNSName.Count -eq $outOfHoursMachines) {
$machinesToPowerOff = $machinesOnAndMaintenance | Sort-Object -Property DNSName
} else {
$machinesToPowerOff = $machinesOnAndNotMaintenance | Sort-Object -Property SessionCount | Select-Object -First $($numberMachines)
}

Or right after it. What are your thoughts? Am I jumping the gun and it would cycle through all of them to make sure one server doesn’t stay on indefinitely?

Now that I am using the latest version of the scripts, I have the issue that all machines will become available when only 1 machine has to be shut down because it can be scaled down. See log:
PowerScaleLog.txt

Is this a setting that is configured and needs to be changed or a bug in the script?

Hi Leee,

Your script is very helpfull for while implementing Citrix on Azure. I do have a few suggestions you might want to take up in next releases.

I have been struggeling why the machines didn't start in time. you have to alter -le in -lt and -gt in -ge to make it work

Function TimeCheck($timeObj) { If (($timesObj.timeNow.Hour -le $timesObj.startTime.Hour) -or ($timesObj.timeNow.Hour -ge $timesObj.endTime.Hour)) { Return "OutOfHours" #OutOfHours as we are outside of working hours } ElseIf (($timesObj.timeNow.Hour -gt $timesObj.startTime.Hour) -and ($timesObj.timeNow.Hour -lt $timesObj.endTime.Hour)) { Return "InsideOfHours" #Dont OutOfHours as we are inside working hours } Else { Return "Error" #Dont do anything if the time calculation is not conclusive } }

What I have done as well is in stead of leaving the machine running in the Function LogoffShutdown, is setting them in Maintenancemode. this way you have the possibility to drain your environment when decreasing the number of machines.

Also I have set the sorting of the machines before selecting the top x, this way you sort all machines in stead of sorting the top x. in the comments what I have set using exisiting functions.

`Function LogoffShutdown () {
[CmdletBinding()]

param(
[Parameter(Mandatory=$true, HelpMessage = "Number of machines to power off")]
[ValidateNotNullOrEmpty()]
[int]$numberMachines

)

WriteLog -Message "User logoff mode is not set to force, waiting for sessions to gracefully disconnect before powering machines down" -Level Info
$machinesToPowerOff = $machinesOnAndNotMaintenance | Sort-Object -Property SessionCount | Select-Object -First $($numberMachines) #Jordy, To make sure the sorting takes place before the selecting. And ascending in stead of descending
foreach ($machine in $machinesToPowerOff) {
    #Check for active sessions on each machine before shutting down
    $sessions = $(brokerUserSessions -machineName $($machine.MachineName) | Where-Object {$_.SessionState -eq "Active"} | Select-Object *)
    If ($null -eq $sessions) {
        WriteLog -Message "No active session found on $($machine.DNSName), performing shutdown" -Level Info
        #Shutdown the machines as there are no active sessions (this will include disconnected sessions)
        If (!$testingOnly) { brokerAction -machineName $($machine.MachineName) -machineAction Shutdown }
    } else {
        WriteLog -Message "Active session(s) found on $($machine.DNSName), this machine cannot be gracefully shutdown yet, but will be placed in maintenance mode" -Level Info
        maintenance -machine $machine.MachineName -maintenanceMode On #Jordy, for Draining!
        foreach ($session in $sessions) {
            WriteLog -Message "Sessions active on $($machine.DNSName), $($session.BrokeringUsername) - session length and state $($(New-TimeSpan -Start $($session.EstablishmentTime)).Minutes) Minutes - State $($session.SessionState) " -Level Info
        }
    }
}

}
`

There is an issue where machines are being left in maintenance mode until the following morning. This means machines are not being put back into service until the exact time that the business starts.

Hi Leee, thanks for taking the time.

It seems the script is not taking into account the average Index but only the index of VDA02 that is Powered ON and out of maintenance mode.

If I manually take VDA03 out of maintenance mode, VDA02 AND VDA03 metrics are calculated, then the script correctly shut down VDA03:

VERBOSE: -
VERBOSE: #######PowerScale script starting - Test mode value is False#######
VERBOSE: Getting a list of machine from citrix.lab.local based on prefix - VDA
VERBOSE: There are 2 scaled on machines, these will be shutdown as soon as they are empty of users in business hours.
VERBOSE: Performance scaling is enabled - attempting performance metrics capture
VERBOSE: The authentication account provided is in the correct format, continuing
VERBOSE: Starting performance measurement job for VDA02.lab.local using specified credentials
VERBOSE: CIM Connection test for VDA02.lab.local successful
VERBOSE: ThreadID: Beginning 18456
VERBOSE: Starting performance measurement job for VDA03.lab.local using specified credentials
VERBOSE: CIM Connection test for VDA03.lab.local successful
VERBOSE: ThreadID: Beginning 18456
VERBOSE: 1 - Overall CPU Average
VERBOSE: 44.5 - Overall Memory Average
VERBOSE: 3000 - Overall Session Index Average
VERBOSE: 1.5 - Overall Session Count Average
VERBOSE: Start Circular Log Management
VERBOSE: The current number of powered on machines is 2 and the target is 1 - resulting action is to Shutdown 1
machines
VERBOSE: It is currently inside working hours - performing machine analysis
VERBOSE: User logoff mode is not set to force, waiting for sessions to gracefully disconnect before powering machines
down
VERBOSE: No active session found on VDA03.lab.local, performing shutdown
VERBOSE: Machine action for lab\VDA03 - Shutdown immediately

Originally posted by @alexis-roger in #27 (comment)

In the dashboard:

A. History section: there is a section of URLs for listing the historical dashboards from other days. Suggestions:

1. Update the height of this section in your template file; then when users select longer retention windows the URLs/Dates will all fit in the space properly. We've found that 120px seems to work nicely.
2. Update the date format to be mm-dd-yyyy. Current format where it can go to m-d-yyyy breaks the sorting of dates and you end up with 8-10-2020 showing up before 8-2-2020 in the listing.

B. Titles in the dashboard - with a number of farms it's not so easy to remember which dashboard you are reviewing; adding elements to the Create Config File.ps1

1. Ask for the user's preferred name/title for the specific instantiation of PowerScale. Use that name in both the <Title> section and the H1 section of the dashboard template. Then the tab has the name and the displayed dashboard has the name on it.

Hi, I have an issue which is causing a machine stuck in maintenance mode and shutting down the wrong machine.
Machine XA-Worker-05 was powered up via Powerscale during the day and was set to maintenance mode automatically when the number of sessions dropped during the day.

Powerscale does not shutdown XA-Worker-05 during the day but keeps the machine in maintenance mode (I prefer to have it shut down if possible). When business ours end and a new run is performed it seems Powerscale does see XA-Worker-05 powered up but chooses to shutdown the active machine with active users (signalling them to logoff) in stead of powering off XA-Worker-05 which has been in maintenance mode all the time and has no sessions active.

I do scaling based on session and business hours. This is in the log:

2021-01-12 18:00:02 INFO: -
2021-01-12 18:00:02 INFO: #######PowerScale script starting - Test mode value is False#######
2021-01-12 18:00:02 INFO: Getting a list of machine from dc01 based on prefix - XA-Worker-0
2021-01-12 18:00:02 INFO: Performance scaling is enabled - attempting performance metrics capture
2021-01-12 18:00:02 INFO: The authentication account provided is in the correct format, continuing
2021-01-12 18:00:02 INFO: Starting performance measurement job for XA-Worker-01 using specified credentials
2021-01-12 18:00:03 INFO: CIM Connection test for XA-Worker-01 successful
2021-01-12 18:00:03 INFO: Starting performance measurement job for XA-Worker-02 using specified credentials
2021-01-12 18:00:03 INFO: CIM Connection test for XA-Worker-02 successful
2021-01-12 18:00:03 INFO: Starting performance measurement job for XA-Worker-03 using specified credentials
2021-01-12 18:00:03 INFO: CIM Connection test for XA-Worker-03 successful
2021-01-12 18:00:03 INFO: Starting performance measurement job for XA-Worker-04 using specified credentials
2021-01-12 18:00:03 INFO: CIM Connection test for XA-Worker-04 successful
2021-01-12 18:00:08 INFO: 4 - Overall CPU Average
2021-01-12 18:00:08 INFO: 28.5 - Overall Memory Average
2021-01-12 18:00:08 INFO: 240 - Overall Session Index Average
2021-01-12 18:00:08 INFO: 5.75 - Overall Session Count Average
2021-01-12 18:00:08 INFO: Start Circular Log Management
2021-01-12 18:00:08 INFO: The current number of powered on machines is 4 and the target is 4 - resulting action is to perform Scaling calculations
2021-01-12 18:00:08 INFO: It is currently inside working hours - performing machine analysis
2021-01-12 18:00:08 INFO: Additional scaling is not enabled, the number of machines to scale at each run will be set to the default of 1
2021-01-12 18:00:08 INFO: The current running machines matches the target machines number, performing scaling analysis
2021-01-12 18:00:08 INFO: Scaling has been selected, the current scaling metric is Session and there are 3 machines currently powered off and there are 1 in maintenance mode available.
2021-01-12 18:00:08 INFO: Working with maintenanceMachines this round - maintenance machines will always be selected first
2021-01-12 18:00:08 INFO: Machine selected to be placed into service is XA-Worker-05 the machine is powered On and maintename mode is True
2021-01-12 18:00:09 INFO: #######PowerScale script finishing#######

Next run:

2021-01-12 19:30:02 INFO: #######PowerScale script starting - Test mode value is False#######
2021-01-12 19:30:02 INFO: Getting a list of machine from dc01 based on prefix - XA-Worker-0
2021-01-12 19:30:02 INFO: Performance scaling is enabled - attempting performance metrics capture
2021-01-12 19:30:02 INFO: The authentication account provided is in the correct format, continuing
2021-01-12 19:30:02 INFO: Starting performance measurement job for XA-Worker-01 using specified credentials
2021-01-12 19:30:02 INFO: CIM Connection test for XA-Worker-01 successful
2021-01-12 19:30:03 INFO: Starting performance measurement job for XA-Worker-02 using specified credentials
2021-01-12 19:30:03 INFO: CIM Connection test for XA-Worker-02 successful
2021-01-12 19:30:03 INFO: Starting performance measurement job for XA-Worker-03 using specified credentials
2021-01-12 19:30:03 INFO: CIM Connection test for XA-Worker-03 successful
2021-01-12 19:30:03 INFO: Starting performance measurement job for XA-Worker-04 using specified credentials
2021-01-12 19:30:03 INFO: CIM Connection test for XA-Worker-04 successful
2021-01-12 19:30:08 INFO: 1.5 - Overall CPU Average
2021-01-12 19:30:08 INFO: 17.5 - Overall Memory Average
2021-01-12 19:30:08 INFO: 120 - Overall Session Index Average
2021-01-12 19:30:08 INFO: 3 - Overall Session Count Average
2021-01-12 19:30:08 INFO: Start Circular Log Management
2021-01-12 19:30:08 INFO: It is currently outside working hours - performing machine analysis
2021-01-12 19:30:08 INFO: The current number of powered on machines is 4 and the target is 1 - resulting action is to Shutdown 3 machines
2021-01-12 19:30:08 INFO: Performance scaling is set to True and scaling outside of business hours is set to True
2021-01-12 19:30:08 INFO: Logging off all disconnected sessions
2021-01-12 19:30:08 INFO: Logging off all disconnected sessions in one hit
2021-01-12 19:30:08 INFO: Logging off userA
2021-01-12 19:30:08 INFO: User logoff mode is set to force, logging all users off of machines that are required to be shutdown
2021-01-12 19:30:08 INFO: Setting XA-Worker-02 maintenance mode On
2021-01-12 19:30:09 INFO: Found 1 user sessions on XA-Worker-02
2021-01-12 19:30:09 INFO: Messaging all users on XA-Worker-02 to logoff
2021-01-12 19:30:09 INFO: Setting XA-Worker-03 maintenance mode On
2021-01-12 19:30:09 INFO: Found 3 user sessions on XA-Worker-03
2021-01-12 19:30:09 INFO: Messaging all users on XA-Worker-03 to logoff
2021-01-12 19:30:09 INFO: Setting XA-Worker-04 maintenance mode On
2021-01-12 19:30:10 INFO: Found 4 user sessions on XA-Worker-04
2021-01-12 19:30:10 INFO: Messaging all users on XA-Worker-04 to logoff
2021-01-12 19:30:11 INFO: Sending message to users to log off - 5 minute warning
2021-01-12 19:30:13 INFO: Sending message to users to log off - 5 minute warning
2021-01-12 19:30:16 INFO: Sending message to users to log off - 5 minute warning
2021-01-12 19:35:12 INFO: Sending message to users to log off - 5 minute warning
2021-01-12 19:35:14 INFO: Sending message to users to log off - 5 minute warning
2021-01-12 19:35:17 INFO: Sending message to users to log off - 5 minute warning
2021-01-12 19:40:12 INFO: Logging off all active user sessions after sending messages at 5 minutes and then 5 minutes
2021-01-12 19:40:13 INFO: Machine action for XA-Worker-02 - Shutdown immediately
2021-01-12 19:40:14 INFO: Logging off all active user sessions after sending messages at 5 minutes and then 5 minutes
2021-01-12 19:40:15 INFO: Machine action for XA-Worker-03 - Shutdown immediately
2021-01-12 19:40:17 INFO: Logging off all active user sessions after sending messages at 5 minutes and then 5 minutes
2021-01-12 19:40:17 INFO: Machine action for XA-Worker-04 - Shutdown immediately
2021-01-12 19:40:33 INFO: Setting XA-Worker-02 maintenance mode Off
2021-01-12 19:40:33 INFO: Setting XA-Worker-03 maintenance mode Off
2021-01-12 19:40:33 INFO: Setting XA-Worker-04 maintenance mode Off
2021-01-12 19:40:34 INFO: #######PowerScale script finishing#######
2021-01-12 19:40:34 INFO: -

Next run:

2021-01-12 19:46:28 INFO: -
2021-01-12 19:46:28 INFO: #######PowerScale script starting - Test mode value is False#######
2021-01-12 19:46:28 INFO: Getting a list of machine from dc01 based on prefix - XA-Worker-0
2021-01-12 19:46:29 INFO: Performance scaling is enabled - attempting performance metrics capture
2021-01-12 19:46:29 INFO: The authentication account provided is in the correct format, continuing
2021-01-12 19:46:29 INFO: Starting performance measurement job for XA-Worker-01 using specified credentials
2021-01-12 19:46:29 INFO: CIM Connection test for XA-Worker-01 successful
2021-01-12 19:46:34 INFO: 0 - Overall CPU Average
2021-01-12 19:46:34 INFO: 21 - Overall Memory Average
2021-01-12 19:46:34 INFO: 200 - Overall Session Index Average
2021-01-12 19:46:34 INFO: 5 - Overall Session Count Average
2021-01-12 19:46:34 INFO: Start Circular Log Management
2021-01-12 19:46:35 INFO: It is currently outside working hours - performing machine analysis
2021-01-12 19:46:35 INFO: The current number of powered on machines is 2 and the target is 1 - resulting action is to Shutdown 1 machines
2021-01-12 19:46:35 INFO: Performance scaling is set to True and scaling outside of business hours is set to True
2021-01-12 19:46:35 INFO: User logoff mode is set to force, logging all users off of machines that are required to be shutdown
2021-01-12 19:46:35 INFO: Setting XA-Worker-01 maintenance mode On
2021-01-12 19:46:35 INFO: Found 5 user sessions on XA-Worker-01
2021-01-12 19:46:35 INFO: Messaging all users on XA-Worker-01 to logoff
2021-01-12 19:46:37 INFO: Sending message to users to log off - 5 minute warning
2021-01-12 20:00:02 INFO: -
2021-01-12 20:00:02 INFO: #######PowerScale script starting - Test mode value is False#######
2021-01-12 20:00:02 INFO: Getting a list of machine from dc01 based on prefix - XA-Worker-0
2021-01-12 20:00:02 INFO: Performance scaling is enabled - attempting performance metrics capture
2021-01-12 20:00:02 INFO: The authentication account provided is in the correct format, continuing
2021-01-12 20:00:02 INFO: Starting performance measurement job for XA-Worker-01 using specified credentials
2021-01-12 20:00:03 INFO: CIM Connection test for XA-Worker-01 successful
2021-01-12 20:00:08 INFO: 5 - Overall CPU Average
2021-01-12 20:00:08 INFO: 22 - Overall Memory Average
2021-01-12 20:00:08 INFO: 200 - Overall Session Index Average
2021-01-12 20:00:08 INFO: 5 - Overall Session Count Average
2021-01-12 20:00:08 INFO: Start Circular Log Management
2021-01-12 20:00:08 INFO: It is currently outside working hours - performing machine analysis
2021-01-12 20:00:08 INFO: The current number of powered on machines is 1 and the target is 1 - resulting action is to perform Scaling calculations
2021-01-12 20:00:08 INFO: Performance scaling is set to True and scaling outside of business hours is set to True
2021-01-12 20:00:08 INFO: Additional scaling is not enabled, the number of machines to scale at each run will be set to the default of 1
2021-01-12 20:00:08 INFO: The current running machines matches the target machines number, performing scaling analysis
2021-01-12 20:00:08 INFO: Scaling has been selected, the current scaling metric is Session and there are 7 machines currently powered off and there are 0 in maintenance mode available.
2021-01-12 20:00:08 INFO: Working with poweredOffMachines this round - maintenance machines will always be selected first
2021-01-12 20:00:08 INFO: Machine selected to be placed into service is XA-Worker-02 the machine is powered Off and maintename mode is False
2021-01-12 20:00:08 INFO: We're out of hours with the correct number of machines - removing scaling tag from XA-Worker-05
2021-01-12 20:00:09 INFO: #######PowerScale script finishing#######
2021-01-12 20:00:09 INFO: -

Can you help? Thanks!

I would love to see the functionality of upscaling based on VM usage.
To save resources it would be best to do this by creating new VM's.
Upscaling: When max capacity is reached new VM's should be created and added to the delivery group.
Downscaling: Can either be used with off business hours or at a predefined number of VM's.