cloudfoundry / app-autoscaler Goto Github PK

Auto Scaling for CF Applications

License: Apache License 2.0

Java 19.82% Shell 0.73% Go 79.12% Makefile 0.33%

app-autoscaler's Introduction

Archived repository - App-AutoScaler

⚠️ ⚠️The contents of this repo have been moved over to https://github.com/cloudfoundry/app-autoscaler-release, please submit all issues/pull requests on that repository.:warning: :warning:

The App-AutoScaler provides the capability to adjust the computation resources for Cloud Foundry applications through

Dynamic scaling based on application performance metrics
Scheduled scaling based on time

The App-AutoScaler has the following components:

api : provides public APIs to manage scaling policy
servicebroker: implements the Cloud Foundry service broker API
metricsgateway : collects and filters loggregator events via loggregator v2 API
metricsserver: transforms loggregator events to app-autoscaler performance metrics ( metricsgateway + metricsserver is a replacement of metricscollector)
metricsforwarder: receives and forwards custom metrics to loggreator via v2 ingress API
eventgenerator: aggreates memory metrics, evaluates scaling rules and triggers events for dynamic scaling
scheduler: manages the schedules in scaling policy and trigger events for scheduled scaling
scalingengine: takes the scaling actions based on dynamic scaling rules or schedules

You can follow the development progress on Pivotal Tracker.

Development

System requirements

Java 11 or above
Docker
Apache Maven 3
Cloud Foundry cf command line
Go 1.15 or above

Database requirement

The App-AutoScaler supports Postgres and MySQL. It uses Postgres as the default backend data store. These are run up locally with docker images so ensure that docker is working on your system before running up the tests.

Setup

To set up the development, firstly clone this project

$ git clone https://github.com/cloudfoundry/app-autoscaler.git

Generate scheduler test certs

Initialize the Database

Postgres

make init-db

MySQL

make init-db db_type=mysql

Generate TLS Certificates

create the certificates

Note: on macos it will install certstrap automatically but on other OS's it needs to be pre-installed

make test-certs

Install consul

To be able to run unit tests and integration tests, you'll need to install consul binary.

if uname -a | grep Darwin; then os=darwin; else os=linux; fi
curl -L -o $TMPDIR/consul-0.7.5.zip "https://releases.hashicorp.com/consul/0.7.5/consul_0.7.5_${os}_amd64.zip"
unzip $TMPDIR/consul-0.7.5.zip -d $GOPATH/bin
rm $TMPDIR/consul-0.7.5.zip

Unit tests

Postgres:

make test

MySQL:

make test db_type=mysql

Integration tests

Postgres

make integration

MySQL:

make test db_type=mysql

Build App-AutoScaler

make build

Clean up

You can use the make clean to remove:

database ( postgres or mysql)
autoscaler build artifacts

Coding Standards

Autoscaler uses Golangci and Checkstyle for its code base. Refer to style-guide

Deploy and offer Auto-Scaler as a service

Go to app-autoscaler-release project for how to BOSH deploy App-AutoScaler

Use Auto-Scaler service

Refer to user guide for the details of how to use the Auto-Scaler service, including policy definition, supported metrics, public API specification and command line tool.

License

This project is released under version 2.0 of the Apache License.

app-autoscaler's People

Contributors

Stargazers

Watchers

app-autoscaler's Issues

unit test failing with golang 1.7

I am trying to run the unit tests as suggested in the README.md, but when I try to execute

go install github.com/onsi/ginkgo/ginkgo
export DBURL=postgres://postgres@localhost/autoscaler?sslmode=disable
pushd src/autoscaler
ginkgo -r -race -randomizeAllSpecs
popd

I encounter following error

Failed to compile brokerserver:

# google.golang.org/grpc/naming
api/google.golang.org/grpc/naming/dns_resolver.go:41: undefined: net.DefaultResolver in net.DefaultResolver.LookupHost
api/google.golang.org/grpc/naming/dns_resolver.go:42: undefined: net.DefaultResolver in net.DefaultResolver.LookupSRV
# google.golang.org/grpc/credentials/internal
api/google.golang.org/grpc/credentials/internal/syscallconn.go:29: syntax error: unexpected = in type declaration
# google.golang.org/grpc/internal/syscall
api/google.golang.org/grpc/internal/syscall/syscall_linux.go:79: tcpconn.SyscallConn undefined (type *net.TCPConn has no field or method SyscallConn)
api/google.golang.org/grpc/internal/syscall/syscall_linux.go:100: tcpconn.SyscallConn undefined (type *net.TCPConn has no field or method SyscallConn)
# google.golang.org/grpc/resolver/dns
api/google.golang.org/grpc/resolver/dns/dns_resolver.go:66: undefined: net.DefaultResolver
api/google.golang.org/grpc/resolver/dns/dns_resolver.go:84: undefined: net.Resolver
# autoscaler/healthendpoint
api/brokerserver/healthendpoint/database_status_collector.go:97: dbMetrics.MaxOpenConnections undefined (type sql.DBStats has no field or method MaxOpenConnections)
api/brokerserver/healthendpoint/database_status_collector.go:101: dbMetrics.InUse undefined (type sql.DBStats has no field or method InUse)
api/brokerserver/healthendpoint/database_status_collector.go:103: dbMetrics.Idle undefined (type sql.DBStats has no field or method Idle)
api/brokerserver/healthendpoint/database_status_collector.go:105: dbMetrics.WaitCount undefined (type sql.DBStats has no field or method WaitCount)
api/brokerserver/healthendpoint/database_status_collector.go:107: dbMetrics.WaitDuration undefined (type sql.DBStats has no field or method WaitDuration)
api/brokerserver/healthendpoint/database_status_collector.go:109: dbMetrics.MaxIdleClosed undefined (type sql.DBStats has no field or method MaxIdleClosed)
api/brokerserver/healthendpoint/database_status_collector.go:111: dbMetrics.MaxLifetimeClosed undefined (type sql.DBStats has no field or method MaxLifetimeClosed)

What worked for me was to test this out with golang v1.13.
And the test cases passed

Is the cpu metric available?

Looking at the code, it appears that the relevant code for the cpu is still under development.
If i simply modify the policy registration (Constants.metrictype: public static String[] metrictype = {METRIC_TYPE_MEMORY, "CPU"};) part, auto scaling seems to work with the cpu metric.
Could it be used this way?
For some reason, is the cpu metric unusable?

500:Internal Server Error

Following the Documentation for app-autoscaler, When I am trying to make the API connection to the server by the following command-
curl -v -X 'PUT' -H "Content-Type: application/json" -H "Authorization: $ACCESS_TOKEN" app_url/v1/apps/app_id/policy -d '{ }'

I am facing the 500 : Internal Server Error

OUTPUT is as follows

upload completely sent off: 3 out of 3 bytes
HTTP/1.1 500 Internal Server Error
Content-Length: 0
Date: Tue, 07 Jun 2016 09:37:29 GMT
Server Apache-Coyote/1.1 is not blacklisted
Server: Apache-Coyote/1.1
X-Vcap-Request-Id: Vcap_ID
Connection #0 to host app_url left intact

while scaling - affects the running AI response - causing spikes and slowness

Description
when using app autoscaler the during the scaling the process .. the overall throughput of the app is going down .. the app instances(AI) that are already existing shows higher response times while the new app instances are coming /scaling.. this is hugely impacting our production systems while using autoscaler; appreciate your insights into this esp with large scale prod environments
Observations:
The are some queries around scaling_events is largely effecting performance esp some tables does not have indices

breach_duration_secs is "0" in trigger event, when it is not defined in trigger rule

when breach_duration_secs is not defined for a trigger rule:

{metric_type: "throughput", threshold: 30, operator: "<=", adjustment: "-1"}

the relevant scaling history record in database will show the breach_duration_secs as zero

{
  app_id: "77e14a0b-666a-40de-bf0f-e8af53257c0a",
  error: "",
  message: "",
  new_instances: 3,
  old_instances: 4,
  reason: "-1 instance(s) because throughput <= 30rps for 0 seconds",
  scaling_type: 0,
  status: 0,
  timestamp: 1543545701419043300
}

This is not issue, juts a query

Hi thanks for your opensource, i wounder this opensource has dashboard or it only supports API and allow the developers to build their customized dashboard ?, If available please provide the samples link. Thanks in advance

Regards
Abhilash S

integration test failed with "Error: API Server health check timed out"

I'm running integration test via "pushd test/integration/api; npm install; npm run integration; popd" on Mac OS 10.12 againt the latest code in develop branch:

npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No license field.

[email protected] integration /Users/huh/code/cloudfoundry/app-autoscaler/test/integration/api
cross-env DB_URI=postgres://postgres:[email protected]:5432/autoscaler cross-env SCHEDULER_URI=http://127.0.0.1:8080 cross-env APISERVER_URI=http://127.0.0.1:3002 node_modules/.bin/_mocha --timeout=40000 api.test.js

API Server-Scheduler Integration tests
Starting the Scheduler..................
Scheduler Health check succeeded. Server is up and running !
1) "before all" hook
Scheduler stopped successfully
API Server stopped successfully

0 passing (31s)
1 failing

API Server-Scheduler Integration tests "before all" hook:
Error: API Server health check timed out
at Request._callback (api.test.js:47:15)
at self.callback (node_modules/request/request.js:186:22)
at Request.onRequestError (node_modules/request/request.js:845:8)
at Socket.socketErrorListener (_http_client.js:267:9)
at emitErrorNT (net.js:1253:8)

npm ERR! Darwin 16.0.0
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "integration"
npm ERR! node v4.4.6
npm ERR! npm v2.15.5
npm ERR! code ELIFECYCLE
npm ERR! [email protected] integration: cross-env DB_URI=postgres://postgres:[email protected]:5432/autoscaler cross-env SCHEDULER_URI=http://127.0.0.1:8080 cross-env APISERVER_URI=http://127.0.0.1:3002 node_modules/.bin/_mocha --timeout=40000 api.test.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] integration script 'cross-env DB_URI=postgres://postgres:[email protected]:5432/autoscaler cross-env SCHEDULER_URI=http://127.0.0.1:8080 cross-env APISERVER_URI=http://127.0.0.1:3002 node_modules/.bin/_mocha --timeout=40000 api.test.js'.
npm ERR! This is most likely a problem with the autoscaler-api-integration package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! cross-env DB_URI=postgres://postgres:[email protected]:5432/autoscaler cross-env SCHEDULER_URI=http://127.0.0.1:8080 cross-env APISERVER_URI=http://127.0.0.1:3002 node_modules/.bin/_mocha --timeout=40000 api.test.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs autoscaler-api-integration
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!
npm ERR! npm owner ls autoscaler-api-integration
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! /Users/huh/code/cloudfoundry/app-autoscaler/test/integration/api/npm-debug.log

I want to scale my app more rapidly when there is a request explosion

Currently autoscaler evaluate the metrics in a breach duration, and it may be slow to do scale action when there are suddenly many requests come to the app.

So we can set 2 breach durations, the stale one and the panic one. For example, stable duration is 60s and panic duration is 6s. The autoscaler does evaluation for both stable and panic duration, and if the one for panic has been breached twice, then it use the panic duration. And if there is no scale action for some time, it then goes back to stable duration.

Support for MySQL

It would be great if mysql support were added so that those of us running with mysql as our primary CC database don't need to deploy and manage a postgres database.

Thanks!

Why do you use postgreSQL in this component?

I want to know the reason.

Database uri has to lower

This causes problems if your password has captial letters which is no fun

https://github.com/cloudfoundry-incubator/app-autoscaler/blob/develop/api/lib/config/setting.js#L7

Failed DB Lock causes process exit

In case a DB Lock can't be obtained, e.g. because the database is in maintenance, the dblock.go utility can cause the process to exit. The process should not exit, but rather obtaining the lock should be retried.

See here for further details: https://github.com/cloudfoundry/app-autoscaler/blob/master/src/autoscaler/sync/dblock.go#L68

[threat-modle] CEW352: CSRF test for GUI

ServiceBroker requires appId as request body for unbinding.

ServiceBroker requires appId as request body for unbinding, but cf never passes such data.
It seems that I bound autosclaer service once, I will never unbind that.

Liquibase leaves dangling lock on startup

Hi there!

We're getting an error on startup, where autoscaler-metrics can't start because liquibase attempts to acquire a lock, but its locking implementation means there's a chance of it dangling with no owner:

consul agent is not needed
Starting Liquibase at Fri, 16 Aug 2019 07:06:19 UTC (version 3.6.3 built at 2019-01-29 11:34:48)
Unexpected error running Liquibase: Could not acquire change log lock.  Currently locked by autoscaler-metrics-1.autoscaler-metrics-set.scf.svc.cluster.local (10.244.2.10) since 8/16/19, 5:26 AM
liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by autoscaler-metrics-1.autoscaler-metrics-set.scf.svc.cluster.local (10.244.2.10) since 8/16/19, 5:26 AM
	at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:230)
	at liquibase.Liquibase.update(Liquibase.java:184)
	at liquibase.Liquibase.update(Liquibase.java:179)
	at liquibase.integration.commandline.Main.doMigration(Main.java:1220)
	at liquibase.integration.commandline.Main.run(Main.java:199)
	at liquibase.integration.commandline.Main.main(Main.java:137)

Steps to reproduce:

(This was on SCF, but as far as I can tell this is an issue with liquibase, not how you run it.)

Start a CF cluster, and have autoscaler installed.
Restart the metricscollector job, and watch its output. (In my case, the autoscaler-metrics pod.)
Once it shows Starting Liquibase, forcibly terminate the process.
Let it restart, and see that it gets stuck.

Details

It appears that liquibase implements locking by inserting a row in a database, instead of taking a database lock (row-level or table-level). See this stackoverflow question for a similar situation, and recovery where manual intervention in the DB is required. I think it might actually be trying to use transactions, though; just removing the "lock" got it to recover.

I'm not sure what you could do to fix this (other than trying to fix it upstream in liquibase — unfortunately my Java is terrible). But I figured I should file it at the minimum.

Thanks!

"bin/deploy.sh myenv" failed due to servicebroker.war not found

$ bin/deploy.sh myenv
Updating app AutoScaling in org abc / space abc as abc...
OK

Using route autoscaling.apps.abc.local
FAILED
Error processing app files: lstat /opt/cf/app-autoscaler/server: no such file or directory
Updating app AutoScalingAPI in org abc / space abc as abc...
OK

Using route autoscalingapi.apps.abc.local
FAILED
Error processing app files: lstat /opt/cf/app-autoscaler/api/target: no such file or directory
Updating app AutoScalingServiceBroker in org abc / space abc as abc...
OK

Using route autoscalingservicebroker.apps.abc.local
FAILED
Error processing app files: lstat /opt/cf/app-autoscaler/servicebroker/target: no such file or directory

I found that 'mvn package' doesn't generate servicebroker.war and api.war

$ mvn package -Denv=myenv -DskipTests
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] db
[INFO] app-autoscaler
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building db 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
......
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building app-autoscaler 1.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] db ................................................. SUCCESS [ 0.835 s]
[INFO] app-autoscaler ..................................... SUCCESS [ 0.001 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.906 s
[INFO] Finished at: 2016-11-28T16:21:17+08:00
[INFO] Final Memory: 15M/309M
[INFO] ------------------------------------------------------------------------

Question about the public API

Hello!

I am wondering if the public API has been exposed or not.
If it is available, how can I use the public API from my local machine?

minjeong@ubuntu:~$ curl https://api_public.bosh-lite.com:6106/v1/apps/9df61bc3-4bdb-484c-b4b8-21a147178fd1/policy --insecure
curl: (7) Failed to connect to api_public.bosh-lite.com port 6106: Connection refused

Related Commits

Thank you in advance!

handling for "start-time" and "end-time" is wrong with golang apiserver

$ cf ash autoscaler-TestApp-4 --start "2019-09-10T10:25:26+08:00" --asc
Retrieving scaling event history for app autoscaler-TestApp-4...
Scaling Type     	Status        	Instance Changes     	Time                          	Action                                                                	Error
dynamic          	succeeded     	1->2                 	2019-09-09T16:23:26+08:00     	+1 instance(s) because cpu >= 2% for 60 seconds
dynamic          	succeeded     	1->2                 	2019-09-09T16:28:26+08:00     	+1 instance(s) because asmon_custom >= 10test-unit for 60 seconds
dynamic          	succeeded     	1->2                 	2019-09-09T16:33:26+08:00     	+1 instance(s) because memoryutil >= 3% for 60 seconds
dynamic          	succeeded     	1->2                 	2019-09-09T16:38:26+08:00     	+1 instance(s) because memoryused >= 10MB for 60 seconds

There is a CSS injection vulnerability in the dependency springfox referenced by scheduler

The CF security team reported CSS injection vulnerability in scheduler:

Hello folks, we received a report about a swagger-ui sequential import chaining vulnerability. It was fixed in v3.23.11. Swagger-UI 2.x is out of support, and is very likely vulnerable to this same attack.
SpringFox is another product which depends on an older (vulnerable) version of swagger-ui. We have filed an issue and submitted a pull-request to get that bumped. When that gets merged and a release is made, we will contact more teams about that dependency.
To be clear, swagger itself (for APIs) is okay, it's just swagger-ui that's an issue. If you depend directly on swagger-ui you can bump it to v3.23.11 or remove it.
If you depend on springfox you need to remove it entirely, since it hasn't bumped its own swagger-ui dependency
this vulnerability is scored at a 9.8 in NVD (https://nvd.nist.gov/vuln/detail/CVE-2019-17495)

Replace the autoscaler.network.subnet with individual independent addresses

[Carried over from cfibmers/open-Autoscaler]

We'd like to use this plugin, but its assumption that the four component servers reside on contiguous network addresses on one subnet doesn't fit with our structure.

I understand that the bosh-release is being completely rewritten from the old open-Autoscaler,
and will most likely be in its own app-autoscaler-release repo, but we still have feedback on how to describe a generic network that work for most users. The current bosh-release in open-Autoscaler was too restrictive.

Question on Autoscaling .

I configured autoscaling in my dev version of cloudfoundry cf release 256

I have applied the below policy to the application .

scenario:-

My application is having one instance only .
At the start start_date_time my application scales from 1 to 3 successfully .
But at the end_time it doesn't scale back to original count which is 1

Per sample catalog definition it should scale down .(I have a feeling that i am unable to understand specific_date schedule ) Any advice is great .
instance_min_count:
type: integer
minimum: 1
description: The number of instances to scale down to once recurrence period
instance_max_count:
type: integer
minimum: 1
description: Maximum number of instances to scale up during recurrence period
initial_min_instance_count:
type: integer
minimum: 1
description: The number of instances to scale up to as soon as the recurrence period starts

  "instance_min_count": 1,
  "instance_max_count": 4,
  "schedules": {
    "timezone": "America/New_York",
    "specific_date": [{
      "start_date_time": "2018-05-30T17:45",
      "end_date_time": "2018-05-30T17:50",
      "instance_min_count": 2,
      "instance_max_count": 4,
      "initial_min_instance_count": 3
    }]
  }
}```

Application not scaling up/ down

Hi, Following the document, i have created a service and attached application to it with policy. I have given load to the application as below

The policy file looks like

Please help where am i doing wrong

As an enduser, I expect the throughput not to be zero if its upper than 0 rps and less than 1 rps

Received complaint from customer about the ZERO throughput:

In order to SEE requests/min they need to have more than 420 request or otherwise it's going to say "ZERO": is this correct?

Is it possible to measure response time?

It is possible to measure something like average response time in two minutes? Sometimes customers might want to have SLA like this. And it would be great to auto-scale instances according to the SLA items.

And not sure whether streaming technology can help here. A typical streaming setup might include components like Metric Agent, Kafka, Spark Streaming, InfluxDB and Grafana. Streaming infrastructure can be used to compute aggregation (max, min, median etc.) for the given time window, trigger actions to scale up or down according to rules and save the metrics (or aggregates) to time series database. In the end Grafana can be used to view the graphs.

Schedules can not be removed in synchronization when an application is deleted directly

Description:
When an application is deleted without detaching its policy:
1 operator will delete the policy from policydb
2 scheduler will try to delete the schedules and notify scalingengine. Now scalingengine returns an error when the app is not found, then scheduler fails to delete the schedules.

Is develop branch available?

I was trying to use 'develop branch' by using the guide(README.md), but it did not working.

I think develop branch's guide is not updated.

Is this develop branch available now??

How to test API?

Context

I have deployed 'app-autoscaler-release' on CF(Open Source).
I am currently developing an autoscaler dashboard for users.

Question

I was test the API in the previous version (master branch) as follows:

$ curl -H "Authorization: Bearer [ACCESS-TOKEN]" "[API-APP-URL]/v1/apps/[APP-ID]/policy/"

{"instanceMinCount":1,"instanceMaxCount":2,"policyTriggers":[{"metricType":"Memory","statWindow":120,"breachDuration":120,"lowerThreshold":1,"lowerThreshold_set":true,"upperThreshold":1,"upperThreshold_set":true,"instanceStepCountDown":1,"instanceStepCountUp":1,"stepDownCoolDownSecs":120,"stepUpCoolDownSecs":120}],"schedules":null,"policyState":"ENABLED"}

In the current version, however, this way is not work.

curl: (56) Recv failure: Connection reset by peer

I am wondering how to test the API.

metricsgateway ut panics because of testhelpers/websocket_handler

-----------------------------
{"timestamp":"1581312962.330476284","source":"ws_helper","message":"ws_helper.WSHelper.setup-new-ws-connection","log_level":1,"data":{"session":"1"}}
•! Panic in Spec Setup (BeforeEach) [0.003 seconds]
WsHelper CloseConn when maximum number of close retries reached [BeforeEach] fails to close websocket connection 
/home/travis/build/cloudfoundry/app-autoscaler/src/autoscaler/metricsgateway/helpers/ws_helper_test.go:119
  Test Panicked
  runtime error: invalid memory address or nil pointer dereference
  /home/travis/.gimme/versions/go1.13.3.linux.amd64/src/runtime/panic.go:199
  Full Stack Trace
  	/home/travis/.gimme/versions/go1.13.3.linux.amd64/src/runtime/panic.go:679 +0x1b2
  github.com/gorilla/websocket.(*Conn).Close(...)
  	/home/travis/build/cloudfoundry/app-autoscaler/src/github.com/gorilla/websocket/conn.go:294
  autoscaler/testhelpers.(*WebsocketHandler).CloseWSConnection(0xc000257170)
  	/home/travis/build/cloudfoundry/app-autoscaler/src/autoscaler/testhelpers/websocket_handler.go:33 +0xbc
  autoscaler/metricsgateway/helpers_test.glob..func1.4.3.1()
  	/home/travis/build/cloudfoundry/app-autoscaler/src/autoscaler/metricsgateway/helpers/ws_helper_test.go:117 +0x84
  github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc0001fc8a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
  	/home/travis/build/cloudfoundry/app-autoscaler/src/autoscaler/metricsgateway/helpers/helpers_suite_test.go:12 +0x86
  testing.tRunner(0xc00046e200, 0xce5500)
  	/home/travis/.gimme/versions/go1.13.3.linux.amd64/src/testing/testing.go:909 +0x19a
  created by testing.(*T).Run
  	/home/travis/.gimme/versions/go1.13.3.linux.amd64/src/testing/testing.go:960 +0x652
  
------------------------------

Question: Enable autoscaler via api?

So we bind new apps to an autoscaler service instance via the manifest. This insures that new versions of the same app get bound to the autoscale service. However, at startup, the autoscaler is not enabled. So its bound, but turned off. We then have to go into the gui, and enable the autoscaler for a specific app. Surely there is a way to do this outside of the gui. We are currently on PCF1.9x. Do we have any options?

This is not issue just query

Is this autoscaler scales only memory or scale cpu & disk
Regards
Abhilash S

Clarification/Question :- Schedule policy

Gurus ,

I have defined below policy and bonded this policy to an application.my application is running with 1 instance .

Please let me know if my assumption is correct .

This policy will kick off at 2018-08-01T13:00 time. At this time the instance count will become 2 .
initial_min_instance_count=2 .

My questions are

At what time will my instance count will become 5 ?Will it ever become 5 ?
I have binded the policy with an application,but still i am able to manually scale the instance above 5 Is this the expected behavior ?
What is the purpose of instance_min_count and instance_max_count at the start of the policy and instance_min_count and instance_max_count in the schedule section of the policy ? Which takes precedence ?

  "instance_min_count": 1,
  "instance_max_count": 4,
  "schedules": {
      "timezone": "America/New_York",
      "specific_date": [{
      "start_date_time": "2018-08-01T13:00",
      "end_date_time": "2018-08-01T13:30",
      "instance_min_count": 1,
      "instance_max_count": 5,
      "initial_min_instance_count": 2
    }, {
      "start_date_time": "2018-08-01T13:35",
      "end_date_time": "2018-08-01T13:45",
      "instance_min_count": 1,
      "instance_max_count": 3,
      "initial_min_instance_count": 1
    }]
 }
}```

Couch and Postgres DBs

Hi,

Looking at the readme, both couch and postgres databases seem to be necessary to run the autoscaller. Could you elaborate on the need for/use of both of them? Is it a valid idea to use cf services?

Thanks

VCAP Services Credentials Null

I created an autoscaler service instance and bound my app to it.

When I execute cf env < MY APP> , I cannot find any the autoscaler service credentials.

 "VCAP_SERVICES": {
  "autoscaler": [
   {
    "credentials": null,
    "label": "autoscaler",
    "name": "portal-autoscaler",
    "plan": "autoscaler-free-plan",
    "provider": null,
    "syslog_drain_url": null,
    "tags": [],
    "volume_mounts": []
   }
  ]

Here is the link of the API document:
https://github.com/cloudfoundry-incubator/app-autoscaler/blob/develop/docs/API_usage.rst#api-endpoint

Any help would be appreciated!

Is memory based-autoscaling available? If so what are the features and limitations?

Support for updated multi process metrics

Howdy,

Wanted to make sure y'all are aware of some changes to how metrics are generated for multi process apps. This could theoretically cause scaling decisions to be made based on metrics that are not relevant to the web process that is being scaled.

More info here: https://lists.cloudfoundry.org/g/cf-dev/message/8483

App not scaling/"Error when polling app"

I get the following log entry on the AutoScaling app:

2016-07-21T12:42:21.06-0400 [APP/0] OUT 2016-07-21 16:42:21,061 ERROR org.cloudfoundry.autoscaler.metric.poller.CFAppStatsPoller:156 [CFAppStatsPoller-Thread-24] [run] Error when polling app a0026054-1cda-416a-9d7c-591dd7ea1424(it might be stopped): null

The app in question is not stopped or missing and has a defined policy, but doesn't scale up or down.

date parsing in server/.../metric/bean/CFInstance.java fails

Incoming times look like this:
2016-05-25T19:54:45.507700194Z

The date parser specifes this format:
"yyyy-MM-dd HH:mm:ss ZZZZZ"

It should be "yyyy-MM-ddTHH:mm:ss.SSS" and the incoming string gets cut off
three digits to the right of the ".". There's no need for time-zone data as the
incoming times are in UTC. Don't include more digits in the sub-second part
because parseDigit interprets this as milliseconds, and will overflow. For example,
the time string "2016-05-25T12:34:56.2000" is equivalent to "2016-05-25T12:34:58.000",
as 2000 msec = 1 sec.

App is only scaling up not scaling down once memory consumption is below the defined policy.

I am testing my autoscaler deployment. I created the service and bounded it with an app. After defining the policy and increase the traffic load, the app is scaling up and it's not scaling down when memory usage of each instance is below the threshold defined in the policy.
The app is now running with the maximum number of instances defined in the policy.

Autoscaler service-binding fails with the error - The service broker returned an invalid response for the request to http://autoscalerservicebroker.xxxxx.com/v2/service_instances/xxxxxxxxxxxxxxxxxx/service_bindings/xxxxxxxxxxxxxxxxxxxxxx?accepts_incomplete=true. Status Code: 500 Internal Server Error, Body: {}

Hello,
We have deployed the autoscaler service with bosh DNS on one of our environments.
The deployment works good, but binding a aervice-instance to an application with the policy results in the error - The service broker returned an invalid response for the request to http://autoscalerservicebroker.xxxxx.com/v2/service_instances/xxxxxxxxxxxxxxxxxx/service_bindings/xxxxxxxxxxxxxxxxxxxxxx?accepts_incomplete=true. Status Code: 500 Internal Server Error, Body: {}

on checking the apiserver logs we could see the error - {"timestamp":"2019-07-29T14:13:06.006Z","source":"autoscaler:apiserver","text":"Input policy JSON schema structure is valid","log_level":"info","data":{"app id":"75f0f8c7-1ae3-4bbc-8f40-614e34388fe9"}}
{"timestamp":"2019-07-29T14:13:06.007Z","source":"autoscaler:apiserver","text":"Input policy JSON validated successfully. Creating policy..","log_level":"info","data":{"app id":"75f0f8c7-1ae3-4bbc-8f40-614e34388fe9"}}
{"timestamp":"2019-07-29T14:13:06.008Z","source":"autoscaler:apiserver","text":"Policy creation request received","log_level":"info","data":{"app id":"75f0f8c7-1ae3-4bbc-8f40-614e34388fe9"}}
{"timestamp":"2019-07-29T14:13:06.008Z","source":"autoscaler:apiserver","text":"Policy guid 0da95b11-4514-46e5-845e-ddf40c08ac0c","log_level":"info"}
{"timestamp":"2019-07-29T14:13:06.030Z","source":"autoscaler:apiserver","text":"Error occurred during schedule creation/update ","log_level":"error","data":{"app id":"75f0f8c7-1ae3-4bbc-8f40-614e34388fe9","error":"getaddrinfo ENOTFOUND autoscalerscheduler.service.cf.internal autoscalerscheduler.service.cf.internal:6102"}}

Something similar was seen in the service broker logs too - {"timestamp":"2019-07-29T13:31:05.035Z","source":"autoscaler:servicebroker","text":"Api Server response","log_level":"info","data":{"status_code":500,"response":{"error":"getaddrinfo ENOTFOUND autoscalerscheduler.service.cf.internal autoscalerscheduler.service.cf.internal:6102"}}}

The logs are generated every time we try binding it to an instance with the -c parameter.

any ideas as to why the address is not getting resolved to its ip ?

Any suggestions or inputs will be helpful

Thanks

Application is Scaling up but not Scaling down

Hi Team,

I am unable to scale down the Instance, even if it has reached the minimum memory utilization.

Please find attached policy file
policy_file.txt

Also current

cf autoscaling-metrics bigapp2 memoryutil

Metrics Name            Instance Index          Value           At
memoryutil              0                       5%              2018-07-30T10:31:09Z
memoryutil              1                       3%              2018-07-30T10:31:23Z
memoryutil              2                       2%              2018-07-30T10:31:23Z
memoryutil              0                       5%              2018-07-30T10:31:24Z
memoryutil              3                       2%              2018-07-30T10:31:24Z
memoryutil              1                       3%              2018-07-30T10:31:38Z
memoryutil              2                       2%              2018-07-30T10:31:38Z
memoryutil              0                       5%              2018-07-30T10:31:39Z
memoryutil              3                       2%              2018-07-30T10:31:39Z
memoryutil              1                       3%              2018-07-30T10:31:53Z
memoryutil              2                       2%              2018-07-30T10:31:53Z
memoryutil              0                       5%              2018-07-30T10:31:54Z
memoryutil              3                       2%              2018-07-30T10:31:54Z

The Instance seems to be still in count of 3 and not Scaling down.

Compatible to older Cloud Foundry Version?

Hello,
Probably silly questions, but does this latest autoscaler service compatible some older CF-release, for example https://github.com/cloudfoundry-attic/cf-release/tree/v251
Thanks and Regards
HAO

different error format for go-api and node-api server

for the same invalid policy:

{"a":"123"}

new go-api responses 400 with error:

[{"context":"(root)","description":"Must validate at least one schema (anyOf)"},{"context":"(root)","description":"scaling_rules is required"},{"context":"(root)","description":"instance_min_count is required"},{"context":"(root)","description":"instance_max_count is required"}]

while old node-api responses 400 with error:

{"error":[{"property":"instance","message":"requires property \"instance_min_count\"","schema":"/policySchema","instance":{"a":"123"},"name":"required","argument":"instance_min_count","stack":"instance requires property \"instance_min_count\""},{"property":"instance","message":"requires property \"instance_max_count\"","schema":"/policySchema","instance":{"a":"123"},"name":"required","argument":"instance_max_count","stack":"instance requires property \"instance_max_count\""},{"property":"instance","message":"is not any of [subschema 0],[subschema 1]","schema":"/policySchema","instance":{"a":"123"},"name":"anyOf","argument":["[subschema 0]","[subschema 1]"],"stack":"instance is not any of [subschema 0],[subschema 1]"}]}

The different formats introduced conversion failure in app-autoscaler-cli-plugin, refer to cloudfoundry/app-autoscaler-cli-plugin#45.

Would memory based autoscaling be available for production?

Couchdb keeps old metrics data

It seems that couchdb has been keeping old metrics which is no more used.

apiserver not able to resolve service broker internal address

After the successful deployment, when I try to update policy, I am seeing this error in apiserver logs

source":"autoscaler:apiserver","text":"Error occurred during check binding ","log_level":"error","data":{"appId":"f898e6fd-94f5-4eae-825a-ddbb73cd2a4a","error":{"code":"ENOTFOUND","errno":"ENOTFOUND","syscall":"getaddrinfo","hostname":"servicebroker.service.cf.internal","host":"servicebroker.service.cf.internal","port":"6107"}}}

I have checked everything but I am not getting why this url servicebroker.service.cf.internal not getting resolved to service broker ip address. Please give me some suggestions to resolve this.

Which component does scale apps by trigger of memory usage?

I could find the schduler and metrics-collector component.

Update Service Instance Failed

Whenever I try to update a service instance with the "-c" option, I get the following error.

minjeong@ubuntu:~/workspace/shell/autoscaler$ cf update-service cf-spring-scheduler -c all_policy.json
Updating service instance cf-spring-scheduler as admin...
FAILED
Server error, status code: 502, error code: 10001, message: The service broker rejected the request to https://servicebroker.service.cf.internal:6101/v2/service_instances/e14a4f2d-6974-417e-b485-42d4b09f08ef?accepts_incomplete=true. Status Code: 404 Not Found, Body: Cannot PATCH /v2/service_instances/e14a4f2d-6974-417e-b485-42d4b09f08ef?accepts_incomplete=true

There are only "put" and "delete" options.
(servicebroker / lib / routes / serviceinstance.js)

  app.put('/v2/service_instances/:instanceId', function(req, res) {
    var serviceInstanceId = req.params.instanceId;
    var orgId = req.body.organization_guid;
    var spaceId = req.body.space_guid;
    
    models.service_instance.findOrCreate({
      serviceInstanceId: serviceInstanceId,
      orgId: orgId,
      spaceId: spaceId,
      where: {
        serviceInstanceId: serviceInstanceId,
        orgId: orgId,
        spaceId: spaceId
      }
    }).then(function(result) {
      var isNew = result[1];
      if (isNew === true) {
        res.status(201);
      } else {
        res.status(200);
      }
      res.json({ "dashboard_url": getDashboardUrl(serviceInstanceId) });
    }).catch(function(err) {
      if (err instanceof models.sequelize.UniqueConstraintError) {
        res.status(409);
      } else {
        logger.error("Fail to handle request: ", {req: req, err: err} );
        res.status(500);
      }
      res.end();
    });

  });

  app.delete('/v2/service_instances/:instanceId', function(req, res) {
    var serviceInstanceId = req.params.instanceId;

    models.service_instance.findById(serviceInstanceId)
      .then(function(instance) {
        if (instance != null) {
          models.service_instance.destroy({
            where: {
              serviceInstanceId: serviceInstanceId
            }
          }).then(function(count) {
            res.status(200);
          });
        } else {
          res.status(410);
        }
        res.json({});
      }).catch(function(err) {
        logger.error("Fail to handle request: ", {req: req, err: err});
        res.status(500).end();
      });

  });

I think there should be an "update" option too.
https://github.com/openservicebrokerapi/servicebroker/blob/v2.12/spec.md#updating-a-service-instance

Cf autoscaling issue

The application is unable to scale up/down when i tried to deploy CF autoscaler release

https://github.com/cloudfoundry-incubator/app-autoscaler-release/tree/master

Location for app-autoscaler tree is mentioned below
https://github.com/cloudfoundry-incubator/app-autoscaler/tree/cef9090c0b2b1d18dd1c9fac9ce9f2896abd5d6b

Issue description.

In past release (unfortunately i don't have the location of the code which i deployed ).We used scale up/down based on memoryutil .

With latest master branch for app-autoscaler-release we are unable to scale-up/down .It work perfectly fine with requests/persec configuration .Please find my policy attached .

{
  "instance_min_count": 1,
  "instance_max_count": 4,
  "scaling_rules": [{
    "metric_type": "memoryutil",
    "stat_window_secs": 60,
    "breach_duration_secs": 100,
    "threshold": 49,
    "operator": "<",
    "cool_down_secs": 60,
    "adjustment": "-1"
  }, {
    "metric_type": "memoryutil",
    "stat_window_secs": 60,
    "breach_duration_secs": 100,
    "threshold": 50,
    "operator": ">=",
    "cool_down_secs": 60,
    "adjustment": "+1"
  }]
}

covladmins-MacBook-Pro-6:~ nsharma$ cf app hello-world
Showing health and status for app hello-world in org paas / space properties as nsharma...
OK

requested state: started
instances: 1/1
usage: 1G x 1 instances
urls: hello-world-new.run.us2.covapp.io
last uploaded: Thu Jul 26 19:27:31 UTC 2018
stack: covs-internal-stack
buildpack: covs-java-III

     state     since                    cpu    memory         disk           details
#0   running   2018-07-27 04:14:41 PM   0.1%   760.4M of 1G   317.7M of 1G ```

![app-autoscaler- cef9090c0b2b1d18dd1c9fac9ce9f2896abd5d6b](https://user-images.githubusercontent.com/11034994/43407355-868e3604-93ec-11e8-8f7d-40c282794404.png)
![old_master](https://user-images.githubusercontent.com/11034994/43407356-869774a8-93ec-11e8-97ae-706cb8ee3ef9.png)

AutoScaling app failed to start saying 'org.ektorp.DbAccessException: Connection refused'

[Note: the original subject and description was removed]

I'm deploying app-autoscaler to my latest Open Source Cloud Foundry. AutoScalingAPI started but AutoScaling and AutoScalingServiceBroker failed to start.

The CF app logs says "org.cloudfoundry.autoscaler.data.couchdb.connection.manager.CouchDbConnectionManager:27 [localhost-startStop-1] [] java.net.ConnectException: Connection refused (Connection refused)". However, I can connect to the CouchDB on 192.168.101.208 using a ruby couchdb client script.
Verified that the deployed AutoScaling App reads 192.168.101.208 as the CouchDB host (by changed 192.168.101.208 to a192.168.101.208 which throws error '... a192.168.101.208 ...')
All CF VMs are in subnet 192.168.101.0.

Gurus, please give some advise. Thanks in advance.

------- commands and config file --------
mvn clean package -Denv=myenv -DskipTests
bin/deploy.sh myenv

$ cat profiles/myenv.properties

couchdbUsername=autoscaler
couchdbPassword=password
couchdbHost=192.168.101.208
couchdbPort=5984
couchdbServerDBName=couchdb-scaling
couchdbMetricDBPrefix=couchdb-scalingmetric
couchdbBrokerDBName=couchdb-scalingbroker

$ cf logs AutoScaling

Connected, tailing logs for app AutoScaling in org xxx / space xxx as xxx ...
2016-12-07T05:58:44.31-0500 [API/0]      OUT Updated app with guid 37aeb982-ca36-4f18-93ac-862abd2480da ({"state"=>"STOPPED"})
2016-12-07T05:58:44.80-0500 [DEA/0]      OUT Got staging request for app with id 37aeb982-ca36-4f18-93ac-862abd2480da
2016-12-07T05:58:46.54-0500 [API/0]      OUT Updated app with guid 37aeb982-ca36-4f18-93ac-862abd2480da ({"state"=>"STARTED"})
2016-12-07T05:58:46.75-0500 [STG/0]      OUT -----> Downloaded app package (5.6M)
2016-12-07T05:58:46.82-0500 [STG/0]      OUT -----> Downloaded app buildpack cache (4.0K)
2016-12-07T05:58:51.08-0500 [STG/0]      OUT -----> Java Buildpack Version: v3.10 (offline) | https://github.com/cloudfoundry/java-buildpack.git#193d6b7
2016-12-07T05:58:56.15-0500 [STG/0]      OUT -----> Downloading Open Jdk JRE 1.8.0_111 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_111.tar.gz (found in cache)
2016-12-07T05:58:57.63-0500 [STG/0]      OUT        Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (6.2s)
2016-12-07T05:58:57.63-0500 [STG/0]      OUT -----> Downloading Open JDK Like Memory Calculator 2.0.2_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-2.0.2_RELEASE.tar.gz (found in cache)
2016-12-07T05:58:57.67-0500 [STG/0]      OUT        Memory Settings: -Xms681574K -XX:MetaspaceSize=104857K -Xss349K -Xmx681574K -XX:MaxMetaspaceSize=104857K
2016-12-07T05:58:57.67-0500 [STG/0]      OUT -----> Downloading Tomcat Instance 8.0.38 from https://java-buildpack.cloudfoundry.org/tomcat/tomcat-8.0.38.tar.gz (found in cache)
2016-12-07T05:58:57.82-0500 [STG/0]      OUT        Expanding Tomcat Instance to .java-buildpack/tomcat (0.1s)
2016-12-07T05:58:57.83-0500 [STG/0]      OUT -----> Downloading Tomcat Lifecycle Support 2.5.0_RELEASE from https://java-buildpack.cloudfoundry.org/tomcat-lifecycle-support/tomcat-lifecycle-support-2.5.0_RELEASE.jar (found in cache)
2016-12-07T05:58:57.83-0500 [STG/0]      OUT -----> Downloading Tomcat Logging Support 2.5.0_RELEASE from https://java-buildpack.cloudfoundry.org/tomcat-logging-support/tomcat-logging-support-2.5.0_RELEASE.jar (found in cache)
2016-12-07T05:58:57.83-0500 [STG/0]      OUT -----> Downloading Tomcat Access Logging Support 2.5.0_RELEASE from https://java-buildpack.cloudfoundry.org/tomcat-access-logging-support/tomcat-access-logging-support-2.5.0_RELEASE.jar (found in cache)
2016-12-07T05:59:00.00-0500 [STG/0]      ERR 
2016-12-07T05:59:11.04-0500 [STG/0]      OUT -----> Uploading droplet (57M)
2016-12-07T05:59:20.35-0500 [DEA/0]      OUT Starting app instance (index 0) with guid 37aeb982-ca36-4f18-93ac-862abd2480da
2016-12-07T05:59:28.16-0500 [App/0]      OUT [CONTAINER] org.apache.coyote.http11.Http11NioProtocol         INFO    Initializing ProtocolHandler ["http-nio-61099"]
2016-12-07T05:59:28.19-0500 [App/0]      OUT [CONTAINER] org.apache.catalina.startup.Catalina               INFO    Initialization processed in 872 ms
2016-12-07T05:59:28.20-0500 [App/0]      OUT [CONTAINER] org.apache.catalina.core.StandardService           INFO    Starting service Catalina
2016-12-07T05:59:28.20-0500 [App/0]      OUT [CONTAINER] org.apache.catalina.core.StandardEngine            INFO    Starting Servlet Engine: Apache Tomcat/8.0.38
2016-12-07T05:59:28.24-0500 [App/0]      OUT [CONTAINER] org.apache.catalina.startup.HostConfig             INFO    Deploying web application directory /home/vcap/app/.java-buildpack/tomcat/webapps/ROOT
2016-12-07T05:59:29.69-0500 [App/0]      OUT [CONTAINER] org.apache.jasper.servlet.TldScanner               INFO    At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
2016-12-07T05:59:29.85-0500 [App/0]      OUT 2016-12-07 10:59:29,852 INFO  org.cloudfoundry.autoscaler.core.listener.LifecycleListener:31 [localhost-startStop-1] [<init>] LifecycleListener initialized.
2016-12-07T05:59:31.26-0500 [App/0]      OUT 2016-12-07 10:59:31,260 ERROR org.cloudfoundry.autoscaler.data.couchdb.connection.manager.CouchDbConnectionManager:27 [localhost-startStop-1] [<init>] java.net.ConnectException: Connection refused (Connection refused)
2016-12-07T05:59:31.26-0500 [App/0]      OUT org.ektorp.DbAccessException: java.net.ConnectException: Connection refused (Connection refused)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.ektorp.util.Exceptions.propagate(Exceptions.java:19)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.ektorp.http.StdHttpClient.executeRequest(StdHttpClient.java:183)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.ektorp.http.StdHttpClient.executeRequest(StdHttpClient.java:192)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.ektorp.http.StdHttpClient.head(StdHttpClient.java:146)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.ektorp.http.RestTemplate.head(RestTemplate.java:105)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.ektorp.impl.StdCouchDbInstance.checkIfDbExists(StdCouchDbInstance.java:72)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.ektorp.impl.StdCouchDbInstance.checkIfDbExists(StdCouchDbInstance.java:67)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.cloudfoundry.autoscaler.data.couchdb.connection.manager.CouchDbConnectionManager.initConnection(CouchDbConnectionManager.java:55)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.cloudfoundry.autoscaler.data.couchdb.connection.manager.CouchDbConnectionManager.<init>(CouchDbConnectionManager.java:23)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.cloudfoundry.autoscaler.data.couchdb.connection.manager.ServerDAOManager.<init>(ServerDAOManager.java:40)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.cloudfoundry.autoscaler.data.couchdb.CouchdbStorageService.<init>(CouchdbStorageService.java:99)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.cloudfoundry.autoscaler.data.couchdb.CouchdbStorageService.Initialize(CouchdbStorageService.java:78)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.cloudfoundry.autoscaler.data.couchdb.AutoScalingDataStoreFactory.InitializeAutoScalingDataStore(AutoScalingDataStoreFactory.java:8)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.cloudfoundry.autoscaler.core.listener.LifecycleListener.contextInitialized(LifecycleListener.java:41)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1092)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1834)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at java.lang.Thread.run(Thread.java:745)
2016-12-07T05:59:31.26-0500 [App/0]      OUT Caused by: java.net.ConnectException: Connection refused (Connection refused)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at java.net.PlainSocketImpl.socketConnect(Native Method)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at java.net.Socket.connect(Socket.java:589)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.http.impl.client.cache.CachingHttpClient.callBackend(CachingHttpClient.java:747)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.http.impl.client.cache.CachingHttpClient.execute(CachingHttpClient.java:446)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.apache.http.impl.client.cache.CachingHttpClient.execute(CachingHttpClient.java:339)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	at org.ektorp.http.StdHttpClient.executeRequest(StdHttpClient.java:177)
2016-12-07T05:59:31.26-0500 [App/0]      OUT 	... 25 more
2016-12-07T05:59:31.28-0500 [App/0]      OUT 2016-12-07 10:59:31,281 ERROR org.cloudfoundry.autoscaler.core.listener.LifecycleListener:44 [localhost-startStop-1] [contextInitialized] CouchDbConnector may not be null
2016-12-07T05:59:31.28-0500 [App/0]      OUT java.lang.NullPointerException: CouchDbConnector may not be null
2016-12-07T05:59:31.28-0500 [App/0]      OUT 	at org.ektorp.util.Assert.notNull(Assert.java:21)
2016-12-07T05:59:31.28-0500 [App/0]      OUT 	at org.ektorp.support.CouchDbRepositorySupport.<init>(CouchDbRepositorySupport.java:73)

Support for sending health metrics over Loggregator

copied from app-autoscaler-release project. cloudfoundry/app-autoscaler-release#162

As a CF component it would be nice if auto scalar published health and metrics over loggregator. CF Operators have all had to integrate loggregator into their monitoring systems to monitor all of the CF Core Components. It would be nice if auto scalar supported the same metrics system as all the other Cf core components. This would allow operators who wished to add auto scalar to their system to not have to bother with monitor auto scalar different than any other CF component.

Today auto scalar provides a Prometheus endpoint which is great for those using Prometheus to monitor their Cloud Foundry Deployments but not so great for those who don't.

With the rewrite to golang that might simplify integration with loggregator since loggregator provides a common golang library for this purpose.