The EC2 Right Sizing solution has reached the end of its useful life. Right-sizing functionality is available as a native feature of AWS Compute Optimizer. Details here: https://aws.amazon.com/compute-optimizer/. The solution will be removed the AWS Solutions library and archived on GitHub. Archived solutions will continue to be available on GitHub; however, the AWS Solutions Team has no further plans to update or provide technical support for the solution.

License: Apache License 2.0

Python 93.45% Shell 6.55%

cost-optimization-ec2-right-sizing's Introduction

AWS Cost Optimization: EC2 Right Sizing

Source code for the AWS solution "Cost Optimization: EC2 Right Sizing". Please see the main solution for the Cost Optimization: EC2 Right Sizing.

Cloudformation template

/deployment

cost-optimization-ec2-right-sizing.template

You will need to replace %%BUCKET_NAME%% & %%VERSION%% in the template to point to the bucket where you put your own copies of the Python source code below.

Python source code

/source

callgcw.py
deleteandterminate.py
getcloudwatchmetrics.py
run-rightsizing-redshift.py

Troubleshooting

Log files are exported to the CloudWatch Logs log group cost-optimization-ec2-right-sizing, log streams: {instance_id}/cfn-init.log {instance_id}/run-rightsizing-redshift.log {instance_id}/deleteandterminate.log

Log files are also located locally on the solution created EC2 instance (if you chose to not terminate resources): /var/log/cfn-init.log /tmp/run-rightsizing-redshift.log /tmp/deleteandterminate.log

Licensed under the Apache License Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at

http://www.apache.org/licenses/

or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions and limitations under the License.

cost-optimization-ec2-right-sizing's People

Contributors

Stargazers

Watchers

cost-optimization-ec2-right-sizing's Issues

Accurate in an A/S environment?

Instances in an A/S environment are ephemeral, will this account for that or will the results always be the current usage?

Wrong code for EUC1 (Frankfurt)

This line in the code is wrong:

source/scripts/run-rightsizing-redshift.py
ls_update_pricelist_sql += " when location='EU (Frankfurt)' then 'EU' "

It should be
ls_update_pricelist_sql += " when location='EU (Frankfurt)' then 'EUC1' "

This error causes that servers in eu-central-1 are not recognised.

How can I change this in the actual installation? The scripts are not usable for us with this error.

got a ValueError when running the script

INFO:root:Finish to download EC2 pricelist file and upload it to S3 bucket: ec2pricelist.csv
INFO:root:Connected to the database
INFO:root:Importing the CloudWatch files to Redshift table
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 169.254.169.254
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 169.254.169.254
INFO:root:Finish to import the CloudWatch files to the Redshift table: cwdatapvbnrmgw
INFO:root:Importing the EC2 pricelist to Redshift
INFO:root:Generate Redshift table structure
INFO:root:Importing the pricelist files to Redshift table: pricelistcrxnwpqt
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 169.254.169.254
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 169.254.169.254
INFO:root:Finish to import the EC2 pricelist to Redshift table: pricelistcrxnwpqt
INFO:root:Analyzing the instances need to be resized
Complete percent: 1.0%Traceback (most recent call last):
  File "/tmp/run-rightsizing-redshift.py", line 434, in <module>
    ls_temp_table = right_sizing(conn, ls_pricelist_tabname, ls_cw_tabname)
  File "/tmp/run-rightsizing-redshift.py", line 376, in right_sizing
    determine_right_type(db_conn, ls_resizetype_sql, ls_temp_table, ls_instanceid, ln_iops_usage, ln_ssd_total_size, ln_cpu_nbr, ln_network_level_usage, ln_rate, ln_mem)
  File "/tmp/run-rightsizing-redshift.py", line 251, in determine_right_type
    ln_min_ssd_size = int(ls_min_storage1[ls_min_storage1.find('x')+1:])
ValueError: invalid literal for int() with base 10: '0.475 NVMe'

CW metric lookup fails when there are lots of instances in a region

Need retry logic and session logic for handling when the EC2 role rotates keys. Additionally, log files do not provide timestamps, making it harder to identify when something failed in relation to other events (e.g. how long it takes to process the instances in a region).

Getting empty S3 bucket after running the solution

There appears to be some 'bug' recently introduced where currently, the right sizing solution runs 'fine' but creates empty s3 bucket (with no recommendation csv file). Raised a support case with aws and they said they were able to repro but can't help as it's not 'officially supported'.

Are more people running into this and/or can some one help fix this?

FYI - I was able to run it successfully last on March 26th.

stuck at EC2Instance creation

template has created most resources and got stuck at EC2Instance "resource creation initiated". When I locate the instance, it is created and at "running" status with green dot. What could be the issue? how long does it take it to complete? it has been about little over 1 hour and 40 minutes...

Unnecessary calls to CW metrics

Threading logic combines Account ID with region, resulting in multiple CW metric look-ups per region.

Stack is going to Create_Complete but there is no output in the built csv files in the s3 bucket

Can you please point out what could be failing and how to troubleshoot these scripts?

Right sizing only gives results of the right-sizing EC2

I am using the Cloud formation template in AWS to do right sizing. I create the cloud formation stack, it runs and completes. When I go to the S3 bucket and review the results, I only get results for the instance the template created. It does not give me any information on the other instances I have running. Any help would be much appreciated.

Can I specify CPU model for recommendation?

I got the result that recommends using t3a instance types. However we are not ready to use any non-Intel CPU. Is that possible to provide an option to exclude CPU model?

Thank you.

Empty result file

We have 15 EC2 instances in the same region i deplyoed the CF stack. The before336hour-with60min.csv file shows many entries but the results file is completly empty. i checked the cloudwatch logfiles, but i couldnt find any errors.

No output when using region ap-southeast-1

Hi!

I am trying to run this script to check my instances right sizing in the region ap-southeast-1.
However, the script seems to skip those regions. I believe it was caused by callgw.py.

if ls_single_region <> 'ap-southeast-1':

I tried to remove this condition, and it is working as expected.

May I know why is that condition necessary in the first place?

Thanks!

RDS right sizing?

Great solution. Are there any plans for an RDS right-sizing solution? If not, any tips on how that could be implemented?

cross-account support

How can i deploy this solution from organization account to other child accounts?

Multiple recommendations for each instance

When the price list file was updated to provide multiple line items for Linux for Microsoft SQL included licenses, the right sizing logic provides multiple recommendations for each EC2 instance.