Application not scaling up/ down about app-autoscaler HOT 56 CLOSED

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/158903736

The labels on this github issue will be updated when the story is started.

from app-autoscaler.

@ashwithahn , please refer to docs :
`memoryutil

The metric "memoryutil", a short name of "memory utilization", is the percentage of used memory for total memory allocated to an application instance. If the memory usage of the application is "100MB of 200MB", the value of "memoryutil" is 50%.
`
If you want to scale out with 44.9M memory, you need to use type "memoryused"

from app-autoscaler.

@cdlliuy Thanks for the reply, as i understand metric "memoryutil" is the percentage of the memory used / memory limit. So according to Policy i have defined, it has breached 30% of overall Memory. Instance should have scaled up. But it has not

from app-autoscaler.

@ashwithahn as I can see from the screenshot, the memory usage is 44.9 MBs of 1 GB which is roughly 5% and the policy you've defined is for 30%.

from app-autoscaler.

Hi @ashwithahn
two things I noticed from your policy

1. breach_duration_secs and cool_down_secs should have minimum value of 60
   -> We'll check if there's a bug, it should not have been accepted.
2. there's a third parameter stat_window_secs which is removed from the main code but not yet reached the release code ( though it is optional, it's default value is 360).
   -> Scaling will take more than 5 minutes

try using a policy like this

{
    "instance_min_count":1,
    "instance_max_count":4,
    "scaling_rules":[
       {
          "metric_type":"memoryutil",
          "stat_window_secs":60,
          "breach_duration_secs":60,
          "threshold":3,
          "operator":">=",
          "cool_down_secs":60,
          "adjustment":"+1"
       },
       {
        "metric_type":"memoryutil",
        "stat_window_secs":60,
        "breach_duration_secs":60,
        "threshold":1,
        "operator":"<",
        "cool_down_secs":60,
        "adjustment":"-1"
     }
    ]
}

from app-autoscaler.

@rohitsharma04 I have used the same policy that you provided and been waiting for about 10mins now. App have not Scaled out.
Is there a way to check which application is bound to policy.json?

from app-autoscaler.

I'm not sure what is going wrong. Maybe you can provide more data using the app-autoscaler cf-cli plugin.
You can use find the plugin here
Using the plugin you can check the policy bound and metrics collected by autoscaler for your application.

from app-autoscaler.

@ashwithahn, besides @rohitsharma04 's suggestion of using app-autoscaler cf-cli plugin, please also paste all your commands you used to create/bind service and update policy as well for investigation.

from app-autoscaler.

@rohitsharma04 Thank u for suggestion. I have installed the plugin and trying to set AutoScaler service API endpoint. getting below error
Is there Anything wrong in what am trying?
@cdlliuy For creating service :
cf create-service-broker autoscaler username password https://autoscalerservicebroker.app-cloudfoundry.com
Binding service : cf bind-service bigapp2 autoscaler -c /root/policy_new.json autoscale_bind
The policy is same as @rohitsharma04 suggested
I have been following https://github.com/cloudfoundry-incubator/app-autoscaler-release document

from app-autoscaler.

Hi @ashwithahn,
the url you're trying to target is sevice broker url.
The public api endpoint is https://autoscaler.
I guess for your case it should be https://autoscaler.app-cloudfoundry.com

from app-autoscaler.

@ashwithahn , Also, why you bind and attach policy to bigapp2 while checking the status of bigapp ?

from app-autoscaler.

@ashwithahn , could you run with "CF_TRACE=true cf aasp appname policy file"? I think you can just copy the output text here , instead of the uploading a pic.

from app-autoscaler.

@ashwithahn , the latest result is OK ,right? so the policy is attached correctly now?

from app-autoscaler.

Hi @ashwithahn
your application will also not scale if your org/space quota is exhausted.
Can you please also provide the output for

cf autoscaling-history APP_NAME

cf autoscaling-metrics APP_NAME memoryutil

from app-autoscaler.

@cdlliuy Yes Policy is attached correctly. But service Autoscaler stopped working and deleted somehow. Trying below

cf create-service autoscaler autoscaler-free-plan autoscaler_service autoscaler_new
Creating service instance autoscaler_new in org dev / space dev as admin.. .
FAILED
Server error, status code: 504, error code: 0, message:

@rohitsharma04 : cf autoscaling-history bigapp2
Showing history for app bigapp2...
OK

No history found for app bigapp2.
[root# cf autoscaling-metrics bigapp2 memoryutil
Showing metrics for app bigapp2...
OK

No metrics found for app bigapp2.

from app-autoscaler.

@ashwithahn , could you check your bosh deployments status? i.e. bosh vms

Also, 504 is a timeout here. Running the create-service cmd with CF_TRACE=true ... will expose a little bit more information.

from app-autoscaler.

@cdlliuy : All Instances for app-autoscaler is in Running (Process state)
Only Service Autoscaler, got deleted

from app-autoscaler.

@ashwithahn , with Only Service Autoscaler, got deleted, do you mean you can't see app-autoscaler service offering anymore in "cf marketplace"? If not that , could you clarify a little bit?

from app-autoscaler.

@cdlliuy : I can see autoscaler in cf marketplace, But i do not see under cf services as running "cf create-service autoscaler autoscaler-free-plan autoscaler_service" is giving me below error:
Creating service instance autoscaler_service in org xyz / space dev as admin...
FAILED
Server error, status code: 504, error code: 0, message:

from app-autoscaler.

@ashwithahn , 504 is a timeout failure, but we can't determine what is wrong from the status code only. Could you do the following steps for further investigation?

1. curl -k -X GET -H "authorization:basic <auth>" https://autoscalerservicebroker.app-cloudfoundry.com/v2/catalog
   The here should be the dXNlcm5hbWU6cGFzc3dvcmQ= if you used "username:password" as auth info when create-service-broker

2. collect the service broker VM logs using “bosh log" cmd and upload the zipped file.
   i.e
   bosh -e vbox -d app-autoscaler logs servicebroker/5d7f36a5-45b6-42d5-9acb-99d148ce297b

from app-autoscaler.

@cdlliuy Also while Binding the service to application with policy file getting below error

cf bind-service bigapp2 autoscaler_service -c /root/policy_new.json autoscale_bind

Binding service autoscaler_service to app bigapp2 in org techm_dev / space dev as admin...
Unexpected Response
Response code: 502
CC code: 0
CC error code:
Request ID: cd1738ab-76f9-4b25-51e0-9968a899c36d::5c0f683c-25d4-4c55-8b34-bf2e3c401d59
Description: {
"description": "Service broker error: Target service instance 6937229e-c0f8-43c5-8246-23083ff68710 does not exist",
"error_code": "CF-ServiceBrokerRequestRejected",
"code": 10001,
"http": {
"uri": "https://autoscalerservicebroker.app-cloudfoundry.com/v2/service_instances/6937229e-c0f8-43c5-8246-23083ff68710/service_bindings/1b329750-e91e-4b75-bc4a-5644109be4f5",
"method": "PUT",
"status": 404
}
}

Also https://autoscalerservicebroker.app-cloudfoundry.com/ url in browser have the following message "Cannot GET /"

from app-autoscaler.

@ashwithahn ，

1. for cmd:
   bosh -e vbox -d app-autoscaler logs servicebroker/5d7f36a5-45b6-42d5-9acb-99d148ce297b, you need to replace the instance id to yours , the one you got when going through the output of "bosh vms". Please try again.

2. from the "curl" output, the service broker http server works correctly now.

3. the bind-service failure is expected, since the required service instances is not here when create-service cmd failed.

4. the https://autoscalerservicebroker.app-cloudfoundry.com/ 404 is expected well, since servicebroker don't expose any content under /

from app-autoscaler.

@ashwithahn , the log says:
Downloading resource 'd8f19b17-395c-4fc0-477f-a1d3dfa8a7e0' to '/root/CFAR/app-autoscaler.servicebroker.ff79ad49-0085-417d-a70f-a69d8e060f2f-20180711-041627-555328682.tgz'...
Could you post /root/CFAR/app-autoscaler.servicebroker.ff79ad49-0085-417d-a70f-a69d8e060f2f-20180711-041627-555328682.tgz here?

from app-autoscaler.

Also, could you provide the output of cf services and cf service autoscaler_service --guid ?

from app-autoscaler.

@ashwithahn , is that to drag & drop or just select the file to put it into comments? or maybe the file is too large to upload?

Anyway, the bind error here is quite weird for me. I can't understand why the service instance is created , but there is no record in database for it yet.
Could you help to do more tedious works as below？

1. delete the autoscaler_service , create it again, and bind to see what happened.

2. if still wrong, please run CF_TRACE=true cf service autoscaler_service to expose the detail information.

3. get the latest zipped log package with "bosh log" and upload. OR
   ssh into the service broker vm to check the logs tail -n 100 /var/vcap/sys/log/servicebroker/servicebroker.stdout.log or tail -n 100 /var/vcap/sys/log/servicebroker/servicebroker.stderr.log, just in case the zipped log file can't be uploaded.

4. ssh into the postgres vms, and do the following:

• find out the psql util by : ps -ef | grep "/bin/postgres" | grep -v "grep" | awk {'print $8'} | sed -e s/postgres$/psql/g
• login to the sql db to check the db records. Just put mine below as a reference:

>>> /var/vcap/data/packages/postgres-9.6.3/2373788fb835caeec4a643b311c7ec4b75a4ba10/bin/psql -U postgres
psql (9.6.3)
Type "help" for help.

postgres=> \c autoscaler
You are now connected to database "autoscaler" as user "postgres".
autoscaler=> select * from binding;
              binding_id              |         service_instance_id          |                app_id                |       created_at
--------------------------------------+--------------------------------------+--------------------------------------+-------------------------
 7d47fc0f-1dd1-4968-8ae5-2267f7d5e638 | 54140fa9-315a-4a55-9f30-51c7664ff4a7 | 825ebf5e-b551-4a09-8651-f306de734630 | 2018-07-10 06:28:49.346
(1 row)

autoscaler=> select * from service_instance;
         service_instance_id          |                org_id                |               space_id
--------------------------------------+--------------------------------------+--------------------------------------
 54140fa9-315a-4a55-9f30-51c7664ff4a7 | 5db8d8aa-6029-4c80-90fd-b088ed92a745 | c1fadc82-590f-4aa1-9d91-deb53d740345
 0d52f442-89ae-401b-b730-d20027e2971c | 5db8d8aa-6029-4c80-90fd-b088ed92a745 | c1fadc82-590f-4aa1-9d91-deb53d740345
(2 rows)

autoscaler=> \q

Let us know the database records on this.

from app-autoscaler.

@cdlliuy : Thank you very much for your Inputs.
I Deleted the service and Created again and Bound Application to it. Worked fine :)

Now again the Question goes back to 1st Comment, Even if the Memoryutil is more than 3 as specified in policy, Application is not Autoscaling out

from app-autoscaler.

@ashwithahn , ok, then please run cf asm app_name memoryutil to check the metric details.

from app-autoscaler.

@cdlliuy : cf autoscaling-metrics bigapp2 memoryutil
Showing metrics for app bigapp2...
OK

No metrics found for app bigapp2.
But

I have used Siege Load tester command to Add load to application, just to simulate Autoscaler Mechanism

siege -c80 -b http://bigapp2.app-cloudfoundry.com

from app-autoscaler.

@ashwithahn , so the metric collector didn't work correctly. Could you send the logs of the vm metricscollector ? still using "bosh logs" or just jump into the vms to fetch the /var/vcap/sys/log/metriccollector/*.log

from app-autoscaler.

@cdlliuy : metricscollector.stderr.log

from app-autoscaler.

@ashwithahn The message should be in metricscollector.stdout.log

from app-autoscaler.

@ashwithahn According to the log, there is issue in communication between metricscollector and cf doppler. So we can check two things:
1 Please check whether cf doppler works fine.
2 For metricscollector, if grant_type is password, please check the cf user and password are correct and the user has enough authority to access doppler, and if the grant_type is client_credentials, also please check the whether client and secret are correct, and whether the client has enough authority.

from app-autoscaler.

@qibobo : could you please help how to check on cf doppler

from app-autoscaler.

@ashwithahn ,

1. first of all, could you post your metriccollector config file? it is a yml file located in /var/vcap/jobs/metriccollector/config/ ... on the metriccollector VM
2. to check the dropper endpoint, you can run the following cmd:

1.  cf login ..
2.  token=`cf oauth-token`
3.  appId=`cf app app_name --guid`
4.  get the dropper url from cf api /v2/info, in my case, it is doppler.bosh-lite.com
5. curl -v -H "Authorization:$token" http://doppler.bosh-lite.com/apps/`cf app demoapp --guid`/containermetrics   .  
If you get 200 OK response, then it works fine.

from app-autoscaler.

@cdlliuy :
please find below metricscollector.yml
metriccollector.txt
Doppler endpoint result

GET /apps/769db0d6-ba2e-4967-aa17-1c179c23578c/containermetrics HTTP/1.1
Host: doppler.app-cloudfoundry.com
User-Agent: curl/7.53.1
Accept: /

< HTTP/1.1 401 Unauthorized
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin:
< Content-Type: text/plain; charset=utf-8
< Date: Thu, 12 Jul 2018 09:30:15 GMT
< Www-Authenticate: Basic
< X-Vcap-Request-Id: 3ebafead-e84f-44c0-6508-081ffe18d1a0
< Content-Length: 0
< Connection: keep-alive
<

• Connection #0 to host doppler.app-cloudfoundry.com left intact

from app-autoscaler.

@ashwithahn
The password seems is messy code.
username: admin
password: ##$$%^$^&
Given you got 401 as well with dropper API, I wonder whether the password you used is correct.

from app-autoscaler.

@qibobo : the password specified in the file is correct, Sorry I should have specified it. I have put characters while uploading here.

from app-autoscaler.

@ashwithahn , with the correct password, could you get containermetrics from dropper with cmd:
curl -v -H "Authorization:$token" http://doppler.bosh-lite.com/apps/cf app app_name --guid/containermetrics ? See my previous comments about how to get the $token.

from app-autoscaler.

@cdlliuy : Getting below error for same

• TCP_NODELAY set
• Connected to doppler.app-cloudfoundry.com (52.49.165.230) port 80 (#0)

GET /apps/769db0d6-ba2e-4967-aa17-1c179c23578c/containermetrics HTTP/1.1
Host: doppler.app-cloudfoundry.com
User-Agent: curl/7.53.1
Accept: /

< HTTP/1.1 401 Unauthorized
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin:
< Content-Type: text/plain; charset=utf-8
< Date: Fri, 13 Jul 2018 03:33:09 GMT
< Www-Authenticate: Basic
< X-Vcap-Request-Id: ee52d458-0f5f-4236-4d91-8f12b7896e4d
< Content-Length: 0
< Connection: keep-alive
<

• Connection #0 to host doppler.app-cloudfoundry.com left intact

from app-autoscaler.

Also i tried giving >>curl -v -H "Authorization:bearer (whole token) wss://doppler.app-cloudfoundry.com/apps/cf app app_name --guid/containermetrics
Got response ">"

from app-autoscaler.

@ashwithahn , the "websocket: bad handshake" msg thrown in "metriccollector" would be caused by different connection failures .. 401 Unauthorized is a possible cause for it.

1. Is the token generated by the "admin" user as configured in metriccollector?
2. Did you delete all "Authorization" from the "curl" output when post here?
3. When you say "Authorization:bearer (whole token)" , is the "(whole token)" started with "bearer" again or just the token string itself?
4. Is that a correct JWT token when you decode with "http://jwt.calebb.net/" ?

from app-autoscaler.

@cdlliuy

1. Is the token generated by the "admin" user as configured in metriccollector?--
   When deploying app-autoscaler as below:
   bosh -e EVN -d app-autoscaler
   deploy templates/app-autoscaler-deployment.yml
   --vars-store=<vars.yml>
   -v system_domain=
   -v cf_admin_password=
   -v skip_ssl_validation=true
   we give the path for vars file, so the metricscollector will fetch these token from it. Right?

2. Did you delete all "Authorization" from the "curl" output when post here?--
   Authorization is CF Password in this case right?
3. When i meant "Authorization:bearer (whole token)" whole token refers to token string
4. Checked in "http://jwt.calebb.net/" got below meassage
   Base64URL decode of JWT segment failed
   --

Also what is notice is, When i run cf nozzle --debug i get below error

Error dialing trafficcontroller server: websocket: bad handshake.
Please ask your Cloud Foundry Operator to check the platform configuration (trafficcontroller is wss://doppler.app-cloudfoundry.com:443).

from app-autoscaler.

@ashwithahn , Authorization is not the "CF password" directly.
you need to do "cf login with your user name and password first", then fetch the token with cmd cf oauth-token. Just to paste my comment again:

1.  cf login  -u  username -p password
2.  token=`cf oauth-token`
3.  appId=`cf app app_name --guid`
4.  get the dropper url from cf api /v2/info, in my case, it is doppler.bosh-lite.com
5. curl -v -H "Authorization:$token" http://doppler.bosh-lite.com/apps/`cf app demoapp --guid`/containermetrics   .  
If you get 200 OK response, then it works fine.

Also, the "cf nozzle" got the exact same output of metriccollector , something is wrong with your trafficcontroller .. but we don't know the exact error. That is why I suggests to have a try with "curl" .

from app-autoscaler.

@cdlliuy :

• TCP_NODELAY set
• Connected to doppler.app-cloudfoundry.com (52.208.128.0) port 443 (#0)

GET /apps/769db0d6-ba2e-4967-aa17-1c179c23578c/containermetrics HTTP/1.1
Host: doppler.app-cloudfoundry.com:443
User-Agent: curl/7.53.1
Accept: /

• Empty reply from server
• Connection #0 to host doppler.app-cloudfoundry.com left intact
  curl: (52) Empty reply from server

from app-autoscaler.

@ashwithahn , try https please, just notice you are using :433 port

from app-autoscaler.

@cdlliuy Please find below for https. And the port it is using is 443
NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER)

• Peer's certificate issuer has been marked as not trusted by the user.
• stopped the pause stream!
• Closing connection 0
  curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
  More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

from app-autoscaler.

@ashwithahn , you can try with -k option

from app-autoscaler.

@cdlliuy : I suspect CF is unable to generate the logs. As i see in loggregator_trafficcontroller, Please find below trafficcontroller.log below

Startup: Setting up the loggregator traffic controller
2018/07/17 03:10:09 health bound to: 127.0.0.1:14825
2018/07/17 03:10:09 Metrics endpoint is listening on 127.0.0.1:14825
2018/07/17 03:10:09 Event from finder: {GRPCDopplers:[10.0.16.16:8082]}
2018/07/17 03:10:09 adding doppler 10.0.16.16:8082
2018/07/17 03:10:09 pprof bound to: 127.0.0.1:41142
2018/07/17 03:10:09 ws bound to: 0.0.0.0:8081
2018/07/17 04:08:36 Connecting to 1 dopplers
2018/07/17 04:08:36 websocket handler: Not a websocket handshake: websocket: not a websocket handshake: 'upgrade' token not found in 'Connection' header
2018/07/17 04:08:36 Disconnecting from stream (10.0.16.16:8082) (doppler.disconnect=false) (ctx.disconnect=1)
2018/07/17 04:08:36 error receiving from doppler via gRPC context canceled
2018/07/17 04:08:37 Connecting to 1 dopplers
2018/07/17 04:08:37 websocket handler: Not a websocket handshake: websocket: not a websocket handshake: 'upgrade' token not found in 'Connection' header
2018/07/17 04:08:37 Disconnecting from stream (10.0.16.16:8082) (doppler.disconnect=false) (ctx.disconnect=1)

from app-autoscaler.

@ashwithahn , I can't determine the exact error in your loggregator_trafficcontroller deployment. But given you have problems when running "cf nozzle" as well, it seems not an app-autoscaler function issue up to now.
Could you seek help in loggrator team first?

from app-autoscaler.

@cdlliuy : I did post in Loggregator team and have not heard from them , meanwhile could you please confirm is wss used for doppler configuration correct or should it be https?? currently Doppler is "wss://doppler.app-cloudfoundry.com:443".

Have Starting the nozzle
Hit Ctrl+c to exit
WEBSOCKET REQUEST
GET /firehose/FirehosePlugin? HTTP/1.1
Host: wss://doppler.app-cloudfoundry.com:443
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: [HIDDEN]
Origin: wss://doppler.app-cloudfoundry.com:443
Authorization: deleted for posting here

WEBSOCKET RESPONSE
HTTP/1.1 400 Bad Request
X-Vcap-Request-Id: 7dcfc901-c3a1-42f9-5eb2-88e55434c6dd
Content-Length: 12
Connection: keep-alive
Content-Type: text/plain; charset=utf-8
Date: Wed, 18 Jul 2018 06:58:04 GMT
Sec-Websocket-Version: 13
X-Content-Type-Options: nosniff

Error dialing trafficcontroller server: websocket: bad handshake.
Please ask your Cloud Foundry Operator to check the platform configuration (trafficcontroller is wss://doppler.app-cloudfoundry.com:443).

Could you please guide me where i can find this configuration file for changing port to 4443 from 443 and rechecking?

from app-autoscaler.

All ,

Looks like the app-autoscaler code is not inserting the values in app_metrics table for two columns UNIT and VALUE . I have attached two screen shot of the table data .Old master branch and new master branch. Looks to me like a code issue .

from app-autoscaler.

@nsharmacovs : Hi Could you please elaborate on what i can do to fix this?

from app-autoscaler.

@ nsharmacovs, the problem you talked about seems different from @ashwithahn ones. Could you help to open a new issue to track it? Also, it would be helpful if you can specify the commit number when talking about "old" master and "new" master. Thanks.

from app-autoscaler.

@ashwithahn , so your traffic controller endpoint is configured to wss://doppler.app-cloudfoundry.com:4443??
What is the "cf curl /v2/info" output? In CC API /v2/info output, there is an entry of "doppler_logging_endpoint".

If you don't have time to fix the /v2/info , and just want your CLI "cf nozzle" running on wss://doppler.app-cloudfoundry.com:4443, then you may hack your ~/.cf/config.json to change the "DopplerEndPoint" entry in the configuration file. According to "cf nozzle" CLI code, it loads the endpoint from a configuration file.

from app-autoscaler.

@cdlliuy : Thank you for guiding us through this whole process. We redeployed aws.yml file which was somehow not updated in cf deployment. Now the app autoscaler is working :) but the problem is, it is not scaling down even if i have used memoryutil.
Please find attached policy file
policy_file.txt

Also current

cf autoscaling-metrics bigapp2 memoryutil

Metrics Name            Instance Index          Value           At
memoryutil              1                       2%              2018-07-30T06:45:23Z
memoryutil              2                       2%              2018-07-30T06:45:24Z
memoryutil              0                       2%              2018-07-30T06:45:24Z
memoryutil              1                       2%              2018-07-30T06:45:38Z
memoryutil              0                       2%              2018-07-30T06:45:39Z
memoryutil              2                       2%              2018-07-30T06:45:39Z
memoryutil              1                       2%              2018-07-30T06:45:53Z
memoryutil              0                       2%              2018-07-30T06:45:54Z
memoryutil              2                       2%              2018-07-30T06:45:54Z

from app-autoscaler.

Hi @cdlliuy : I would like to close this issue and open new for scaling down. Thank you very much for your timely response and help on resolving this problem.

from app-autoscaler.