Comments (10)
Hi Larry,
Thanks for reporting this. We are trying to reproduce the issue. I'll post an update once we know what is going on. Sorry about the delay.
Thanks,
Surya.
from aws-codedeploy-agent.
I tried with a 200 MB bundle and I'm not able to reproduce this in CentOS 7. But I'd like to isolate what causes the spike. I've got a few questions and suggestions,
- What is the exact metric in which you observe a spike? Is it memfree or memusable?
- Is your memusable keep decreasing when you do more deployments?
- I'd like you to remove all the hooks in your appspec and also don't copy any files as part of install step and do a deployment. This should only download your bundle and do nothing else. Do you see the same spike when you do this?
- Now same as above but keep your files section in appspec and do a deployment.
3 and 4 would let us know if the problem is with downloading and extracting your tarball or installing files or executing hooks scripts.
Thanks,
Surya.
from aws-codedeploy-agent.
Hi Surya,
Please see the following for the information you requested above.
- CodeDeploy Memory Usage - 10 Deployments DownLoadBundle Only
https://gist.github.com/laltomar-cn/330edfb9d9e956d29500 - CodeDeploy Memory Usage - 5 Deployments DownLoadBundle and Copy Files (Uses 40% of Memory)
https://gist.github.com/laltomar-cn/a02c659629ac6c0f8fe6 - CodeDeploy Memory Usage - 3 Deployments DownLoadBundle, Copy Files, and Hooks (3rd deployment fails due to OOM)
https://gist.github.com/laltomar-cn/2981f347cd1fc33b1f8c
I am also seeing high iNode usage, it does not look like tmp directories are getting cleaned up. I did submit close to 20 deploys which is not practical. But I am wondering how is /tmp/codedeploy-agent cleaned up?
from aws-codedeploy-agent.
What files do you see in /tmp/codedeploy-agent? With the default configuration there shouldn't be any /tmp/codedeploy-agent. Here are the files the agent keeps (along with their rotation policy),
- File /tmp/codedeploy-agent.update.log (Rotates at a max file size of 2MB)
This file is used by the agent installer and does not have anything to do with deployments. - Directory /var/log/aws/codedeploy-agent contains the agent run logs. It keeps the past 7 days worth of logs. Rotates everyday at midnight.
- Directory /opt/codedeploy-agent/deployment-root contains a single folder for each deployment group you've deployed to in this host and each deployment group folder at most keeps the last 5 deployments details (Including the last deployment with a successful install event). For e.g, 6th deployment will wipe off the 1st deployment folder unless 1st deployment is not the last deployment with a successful install. If so, it will wipe off the 2nd deployment.
- /opt/codedeploy-agent/state/.pid should contain a couple of pid files.
These are the only log/metadata files the codedeploy agent should be keeping. Are you noticing any other files accumulating other than these? Also does your bundle contain a lot of small files? What's the expanded size of your bundle?
from aws-codedeploy-agent.
Looks like all tmp deployment files are stored in.
/tmp/codedeploy-agent/deployment-instructions
/tmp/codedeploy-agent/e48521d4-3a85-456a-b428-f7776187479f
/opt/codedeploy-agent/ does not exist.
I installed codedeploy agent using instructions.
https://rubygems.org/gems/aws-codedeploy-agent
What are the install instructions for rpm?
from aws-codedeploy-agent.
Okay. The CodeDeploy agent gem hosted in RubyGems is not yet reviewed and not officially approved by AWS. It is a community contribution. Here's the pull request #4
CentOS is not officially supported yet. But if you are just playing around with CodeDeploy you can follow what I did to install the agent in CentOS 7. Though I don't recommend this for production since the updater cannot update the agent.
- Pick up the agent rpm from https://s3.amazonaws.com/aws-codedeploy-us-east-1/releases/codedeploy-agent-1.0-1.663.noarch.rpm
- In CentOS host run 'sudo yum install ruby' (Make sure it installs ruby2.0)
- Do 'ln -s /usr/bin/ruby /usr/bin/ruby2.0'
- Run 'rpm -i --nodeps codedeploy-agent-1.0-1.663.noarch.rpm
That should get the agent running. But the updater will fail to update the agent so you'd be missed out on bug fixes (including emergent security patches), new features, etc. So I wouldn't recommend for production use until there's one available officially.
Thanks,
Surya.
from aws-codedeploy-agent.
Ok, I will install this using the RPM. I will redo the use cases you requested (I.e DownLoadBundle only first, DownLoadBundle and Copy files. I will upload the results sometime this weekend.
We are very interested in using codedeploy in a CentOS production environment. What is the anticipated date that this OS will be supported?
from aws-codedeploy-agent.
Hi Surya,
Please see the following for the information you requested above.
- Use Case 1 DownLoadBundle Only, 3 Deployments.
https://gist.github.com/laltomar-cn/330edfb9d9e956d29500 - Use Case 2: DownloadBundle/Copy Files:
https://gist.github.com/laltomar-cn/a02c659629ac6c0f8fe6 - Use Case 3: DownloadBundle/Copy Files/Hooks, 3 deployments (Deployment Fails OOM).
https://gist.github.com/laltomar-cn/2981f347cd1fc33b1f8c
from aws-codedeploy-agent.
Hi,
Any update on this?
Thanks,
Larry
from aws-codedeploy-agent.
Hi Larry,
I did dive into this and profiled the codedeploy agent for memory leaks. I tried deployments with a reasonably large gzipped archive (529 Mb deflated, 90 Mb compressed with 20k+ files). I ran into the same issue you reported. I did a few memory optimizations to the agent before collecting the following metrics.
Garbage collector statistics at the end of each deployment
https://gist.github.com/suryanarayanan/cd1035afaa01ac5bc78a
Process stats at the end of each deployment (except for the first entry)
https://gist.github.com/suryanarayanan/a2adfd2f84485752db80
The process RSS stabilized after multiple runs and I do not see any sharp rise. As you can see the number of active objects in the heap do not considerably increase to indicate any object leaks. One of the reasons for increasing process RSS is the huge number of temporary strings and hashes the agent creates during the Install step if the archive has considerably large number of files. These strings contain file names and directory names present in your archive. These strings are eventually garbage collected but the freed pages are still part of agent's heap. Also partial GC (garbage collection) does not sweep all of these pages at once. I tested this by explicitly doing full GC at the end of every install step and ended up with much lesser process RSS.
I'll try to get the optimizations out in our next agent release. Meanwhile you can purge the clean caches with the way you already do. You can also play around with Ruby 2.0's GC parameters to make the garbage collection more aggressive.
Thanks,
Surya.
from aws-codedeploy-agent.
Related Issues (20)
- Code deploy falling very frequently HOT 1
- CodeDeplot Access denied when calling "codedeploy-commands.eu-central-1.amazonaws.com:443" HOT 1
- [Feature Request] Debian (12 Bookworm) support HOT 1
- [problem] CodeDeploy Agent systemd service bash-completion error
- [Error] AL2023 - max_revisions - Error deleting directories HOT 3
- CodeDeploy 'Downloading failed bundle.zip from s3 bucket' at "DownloadBundle" step HOT 1
- Service is running but with errors HOT 1
- Codedeploy failing randomly at Install not finding appspec.yml
- Switch to use s3 multipart download when fetching artifacts
- DownloadBundle deployment lifecycle events for troubleshooting UnknownError: execution expired HOT 1
- Failed deployment DownloadBundle events with error "No such file or directory - getcwd" HOT 1
- Add the CodeDeploy agent to the PATH HOT 1
- CodeDeploy agent support for RHEL 9 HOT 1
- Announcement: Releasing Agent 1.7.0 to all regions
- [Feature request] Deployments to ASG Warm pool instances using CodeDeploy lifecycle hooks HOT 1
- [feature] allow codeploy agent service with manual start on windows
- [Issue] : cron job not running as expected after CodeDeploy changes the code in EC2 using CodePipeline
- Install fails using SSM AWS-ConfigureAWSPackage on ARM
- Some request missing SNI field HOT 2
- Permission denied @ rb_sysopen
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-codedeploy-agent.