Giter VIP home page Giter VIP logo

noaa's Introduction

NOAA

Coverage Status GoDoc

noaa is a client library to consume metric and log messages from Doppler.

If you have any questions, or want to get attention for a PR or issue please reach out on the #logging-and-metrics channel in the cloudfoundry slack

Get the Code

This Go project is designed to be imported into $GOPATH, rather than being cloned into any working directory. There are two ways to do this.

  • The easiest way with with go get. This will import the project, along with all dependencies, into your $ GOPATH.

    $ echo $GOPATH
    /Users/myuser/go
    
    $ go get github.com/cloudfoundry/noaa
    
    $ ls ~/go/src/github.com/cloudfoundry/
    noaa/         sonde-go/
    
  • You can also manually clone the repo into your $GOPATH, but you then have to manually import dependencies.

    $ echo $GOPATH
    /Users/myuser/go
    
    $ cd /Users/myuser/go/src/github.com/cloudfoundry
    $ git clone [email protected]:cloudfoundry/noaa.git
    $ cd noaa
    $ go get ./...
    

Updates

Reconnecting to Traffic Controller

noaa has recently updated its reconnect strategy from trying to reconnect five times in quick succession to a back-off strategy. The back-off strategy can be configured by setting the SetMinRetryDelay() and the SetMaxRetryDelay().

During reconnection, noaa will wait initially at the MinRetryDelay interval and double until it reaches MaxRetryDelay where it will try reconnecting indefinitely at the MaxRetryDelay interval.

This behavior will affect functions like consumer.Firehose(), consumer.Stream() and consumer.TailingLogs().

Sample Applications

Prerequisites

In order to use the sample applications below, you will have to export the following environment variables:

  • CF_ACCESS_TOKEN - You can get this value by executing ($ cf oauth-token). Example:
export CF_ACCESS_TOKEN="bearer eyJhbGciOiJSUzI1NiJ9.eyJqdGkiOiI3YmM2MzllOC0wZGM0LTQ4YzItYTAzYS0xYjkyYzRhMWFlZTIiLCJzdWIiOiI5YTc5MTVkOS04MDc1LTQ3OTUtOTBmOS02MGM0MTU0YTJlMDkiLCJzY29wZSI6WyJzY2ltLnJlYWQiLCJjbG91ZF9jb250cm9sbGVyLmFkbWluIiwicGFzc3dvcmQud3JpdGUiLCJzY2ltLndyaXRlIiwib3BlbmlkIiwiY2xvdWRfY29udHJvbGxlci53cml0ZSIsImNsb3VkX2NvbnRyb2xsZXIucmVhZCJdLCJjbGllbnRfaWQiOiJjZiIsImNpZCI6ImNmIiwiZ3JhbnRfdHlwZSI6InBhc3N3b3JkIiwidXNlcl9pZCI6IjlhNzkxNWQ5LTgwNzUtNDc5NS05MGY5LTYwYzQxNTRhMmUwOSIsInVzZXJfbmFtZSI6ImFkbWluIiwiZW1haWwiOiJhZG1pbiIsImlhdCI6MTQwNDg0NzU3NywiZXhwIjoxNDA0ODQ4MTc3LCJpc3MiOiJodHRwczovL3VhYS4xMC4yNDQuMC4zNC54aXAuaW8vb2F1dGgvdG9rZW4iLCJhdWQiOlsic2NpbSIsIm9wZW5pZCIsImNsb3VkX2NvbnRyb2xsZXIiLCJwYXNzd29yZCJdfQ.mAaOJthCotW763lf9fysygqdES_Mz1KFQ3HneKbwY4VJx-ARuxxiLh8l_8Srx7NJBwGlyEtfYOCBcIdvyeDCiQ0wT78Zw7ZJYFjnJ5-ZkDy5NbMqHbImDFkHRnPzKFjJHip39jyjAZpkFcrZ8_pUD8XxZraqJ4zEf6LFdAHKFBM"
  • DOPPLER_ADDR - It is based on your environment. Example:
export DOPPLER_ADDR="wss://doppler.10.244.0.34.xip.io:4443"

Application logs

The samples/app_logs/main.go application streams logs for a particular app. The following environment variable needs to be set:

  • APP_GUID - You can get this value from running $ cf app APP --guid. Example:
export APP_GUID=55fdb274-d6c9-4b8c-9b1f-9b7e7f3a346c

Then you can run the sample app like this:

go build -o bin/app_logs samples/app_logs/main.go
bin/app_logs

Logs and metrics firehose

The samples/firehose/main.go application streams metrics data and logs for all apps.

You can run the firehose sample app like this:

go build -o bin/firehose samples/firehose/main.go
bin/firehose

Multiple subscribers may connect to the firehose endpoint, each with a unique subscription_id (configurable in main.go). Each subscriber (in practice, a pool of clients with a common subscription_id) receives the entire stream. For each subscription_id, all data will be distributed evenly among that subscriber's client pool.

Container metrics

The samples/container_metrics/consumer/main.go application streams container metrics for the specified appId.

You can run the container metrics sample app like this:

go build -o bin/container_metrics samples/container_metrics/consumer/main.go
bin/container_metrics

For more information to setup a test environment in order to pull container metrics look at the README.md in the container_metrics sample.

Development

Use go get -d -v -t ./... && ginkgo --race --randomizeAllSpecs --failOnPending --skipMeasurements --cover to run the tests.

noaa's People

Contributors

acrmp avatar ajackson avatar bradylove avatar ctlong avatar dependabot[bot] avatar dsabeti avatar enocom avatar frodenas avatar geofffranks avatar glyn avatar ihcsim avatar iplay88keys avatar jasonkeene avatar jmtuley avatar johannasmith avatar jtarchie avatar mkocher avatar mrosecrance avatar nelsam avatar poy avatar roxtar avatar rroberts2222 avatar shalako avatar shinji62 avatar tcnksm avatar theone20001 avatar thepeterstone avatar voelzmo avatar wfernandes avatar xenophex avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

noaa's Issues

SOCKS5 proxy support by way of setting proxy env var

Golang 1.9 added support for SOCKS5 proxies. It works out of the box for the cf CLI, except for cf logs.

We figured out how to get socks5 proxy to work with cf logs. It requires some changes to the websocket library the loggregator team vendors (https://github.com/gorilla/websocket).

gorilla needs to use a SOCKS dialer to establish a connection with the proxy, similar to the http transport in the second link.
https://github.com/gorilla/websocket/blob/master/client.go#L249-L279
https://github.com/golang/go/blob/master/src/net/http/transport.go#L1149-L1165

Please add out-of-the-box SOCKS5 proxy support?

TailingLogs[WithoutReconnect] leaks goroutines

Take a look at https://github.com/cloudfoundry/noaa/blob/master/consumer.go#L85

The goroutine ranges over the channel allEvents until it's closed, so some other goroutine would have to initiate closing that channel. As written, the deferred close will never be reached.

Here's some code to demonstrate. Fire up and then visit http://localhost:6060/debug/pprof/ to see:

package main

import (
    _ "net/http/pprof"

    "crypto/tls"
    "fmt"
    "net/http"
    "os"
    "time"

    "github.com/cloudfoundry/noaa"
    "github.com/cloudfoundry/sonde-go/events"
)

func main() {
    for i := 0; i < 99; i++ {
        consumer := noaa.NewConsumer(
            getEnvironmentVariable("traffic_controller_url"), &tls.Config{InsecureSkipVerify: true}, nil,
        )
        eventChannel := make(chan *events.LogMessage)
        go rangeEvents(i, eventChannel)
        go tailLogs(i, consumer, eventChannel)
        go closeAfter(i, consumer, eventChannel)
    }

    fmt.Printf("Listening on :6060\n")
    http.ListenAndServe(":6060", nil)
}

// Just a sanity check to make sure we're listening properly
func rangeEvents(i int, events chan *events.LogMessage) {
    for range events {
        fmt.Printf("[%02d] received message\n", i)
    }
    fmt.Printf("[%02d] Stopped ranging channel\n", i)
}

func tailLogs(i int, consumer *noaa.Consumer, eventChannel chan *events.LogMessage) {
    consumer.TailingLogsWithoutReconnect(
        getEnvironmentVariable("app_id"), getEnvironmentVariable("token"), eventChannel,
    )
    fmt.Printf("[%02d] Stopped tailing logs\n", i)
}

func closeAfter(i int, consumer *noaa.Consumer, eventChannel chan *events.LogMessage) {
    time.Sleep(15 * time.Second)
    consumer.Close()
    close(eventChannel)
    fmt.Printf("[%02d] Closed consumer and channel\n", i)
}

func getEnvironmentVariable(name string) string {
    if variable := os.Getenv(name); variable != "" {
        return variable
    } else {
        panic(fmt.Sprintf("Environment variable '%s' is required", name))
    }
}

And the shell script I was using to run it

#!/usr/bin/env bash

export traffic_controller_url=wss://doppler.<REPLACE_ME_WITH_INSTALL_DOMAIN>:443
export token=`cf oauth-token | tail -n 1 | tr -d '\n'`
export app_id=<REPLACE_ME_WITH_SOME_APP_ID>

go run main.go

Reconnection token failed

Hi,

As you know we provide an authToken in order to connect to the firehose.
If a problem occur we try to reconnect, but if the reconnection happen after the token get expire we could not reconnect.

Can be nice if the noaa lib could manage the auth and manage the token life.

@wfernandes

Import Cycle Issue when trying to build the application

I am running go build -o bin/app_logs samples/app_logs/main.go from inside the $GOPATH/src/github.com/cloudfoundry/noaa and getting the import cycle error.

import cycle not allowed
package main
	imports crypto/tls
	imports bytes
	imports errors
	imports internal/reflectlite
	imports runtime
	imports internal/bytealg
	imports internal/cpu
	imports runtime

Need help regarding this. I know why it is occurring and what it means, but from where should I run this command to build the code? Thanks.

http response body data is lost in error message

https://github.com/cloudfoundry/noaa/blob/master/consumer/async.go#L448

errMsg should include the HTTP error code and response bodyerrMsg

The error returned here does not include the HTTP status code or body, making it difficult to understand why an error is occurring.

The error message becomes something like this:

Error dialing trafficcontroller server: websocket: bad handshake.
Please ask your Cloud Foundry Operator to check the platform configuration (trafficcontroller is wss://doppler.my-env.my-domain.com:443).

Without the HTTP body or status code:

404 page not found

include timstamp when getting container metrics

when calling 'noaa.ContainerMetrics(appId, token)', the timestamp is lost. We will need to know when the metrics are collected. Otherwise we will not know whether it is up-to-date or not.

If we delete some instances of an app, we still get the container metrics of the instances for the following several minutes. With the timestamp info, we will at least know when the last metrics were collected.

building sample clients fails

$ go build -o bin/firehose_sample firehose_sample/main.go
firehose_sample/main.go:8:2: cannot find package "github.com/cloudfoundry/noaa" in any of:
    /usr/local/Cellar/go/1.4.2/libexec/src/github.com/cloudfoundry/noaa (from $GOROOT)
    /Users/scoen/go/src/github.com/cloudfoundry/noaa (from $GOPATH)
firehose_sample/main.go:9:2: cannot find package "github.com/cloudfoundry/sonde-go/events" in any of:
    /usr/local/Cellar/go/1.4.2/libexec/src/github.com/cloudfoundry/sonde-go/events (from $GOROOT)
    /Users/scoen/go/src/github.com/cloudfoundry/sonde-go/events (from $GOPATH)

There are probably some path management steps necessary, however these are not documented. I don't see a .envrc file I am accustomed to sourcing for go projects.

Still supported?

Hey everyone, is this project still supported - I believe some of the functionality like streaming logs still relies on websockets - has this project been deprecated in favor of go-loggregator?
Thanks!

Fail to close app streaming connection when something wrong happened during readmessage

Not sure whether the repo is still active when loggregator is moving to v2 API.
I met problem when running with app streaming with V1 API. I tried to close the stream when error detected, and reconnect. Here is the code sample:

eventChan, errorChan := noaaConsumer.Stream(  appId, accessToken)
for {
	select {
	case  err = <-errorChan:
            if err != nil {
                                logger.Error("stream data..", err, lager.Data{"appid": as.appId})
				closeErr := noaaConsumer.Close()
				if closeErr != nil {
				     logger.Error("close-noaa-connection", closeErr, lager.Data{"appid": as.appId})
				}
                                 eventChan, errorChan := noaaConsumer.Stream(  appId, accessToken)
			}
        case  
        ....
       }
}

But I found a lot of "CLOSE_WAIT" connection in the client side, further along, the client crashed with "too many open files" errors ..

I noticed the Close() function even got errors , and failed to recycle related fd.

Example 1:

{"data":{"appid":"c1c06ff5-760a-4d3b-82ba-58cdf2f609e8","error":"websocket: close 1008 (policy violation): Client did not respond to ping before keep-alive timeout expired.","session":"1478"},"log_level":2,"log
_time":"2018-12-10T08:29:04Z","message":"metricscollector.app-streamer.stream-metrics-error","source":"metricscollector","timestamp":"1544430544.125764608"}

{"data":{"appid":"c1c06ff5-760a-4d3b-82ba-58cdf2f609e8","error":"websocket: close sent","session":"1478"},"log_level":2,"log_time":"2018-12-10T08:29:04Z","message":"metricscollector.app-streamer.close-noaa-conn
ection","source":"metricscollector","timestamp":"1544430544.125951767"}

In this example, the close error is "close sent". If checking the default close handler, https://github.com/gorilla/websocket/blob/master/conn.go#L1075
we can find that a write control msg is sent by default. Then when we invoke noaa Close() , it failed when trying to write another writecontrol.

After ignored the writecontrol error in

noaa/consumer/async.go

Lines 492 to 496 in 9d59e2b

err := c.ws.WriteControl(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""), time.Time{})
if err != nil {
return err
}
return c.ws.Close()
, I got another series of error:
Example 2:

{"data":{"appid":"edc7c205-db63-4ab3-883c-410672d6e5d1","error":"websocket: close 1006 (abnormal closure): unexpected EOF","session":"87"},"log_level":2,"log_time":"2018-12-10T20:16:06Z","message":"metricscolle
ctor.app-streamer.stream-metrics-error","source":"metricscollector","timestamp":"1544472966.802547216"}

{"data":{"appid":"edc7c205-db63-4ab3-883c-410672d6e5d1","error":"tls: use of closed connection","session":"87"},"log_level":2,"log_time":"2018-12-10T20:16:06Z","message":"metricscollector.app-streamer.close-noa
a-connection-witherror","source":"metricscollector","timestamp":"1544472966.802864075"}

{"data":{"appid":"edc7c205-db63-4ab3-883c-410672d6e5d1","session":"87"},"log_level":1,"log_time":"2018-12-10T20:16:06Z","message":"metricscollector.app-streamer.noaa-reconnected-0","source":"metricscollector","
timestamp":"1544472966.802957058"}

In this example, I don't quite understand why the tls layer responded the connection is closed when calling ws.close(), but as a result, the invocation of conn.close() did fail in below code snippets.

noaa/consumer/async.go

Lines 164 to 169 in 9d59e2b

for len(c.conns) > 0 {
if err := c.conns[0].close(); err != nil {
return err
}
c.conns = c.conns[1:]
}

So, the problematic connection is still kept in the conn slice, and new connection is append after that. If the new connection becomes weird again and the conn[0].close() keeps failing, we don't have chance to close conn[1] at all.

Is there any special consideration here to break out the conns close with any error?

consumer has goroutine leak for http persistConn readLoop and writeLoop after consumer.Close() called

When I using noaa to consume firehose metrics data drain, I have to refresh the token as the token have to be expired for a period.
So the rough steps I used:
0.authToken = nozzle.authTokenFetcher.FetchAuthToken()

  1. noaa.NewConsumer for the first time
  2. get firehose metrics data via Consumer.FirehoseWithoutReconnect(authToken)
  3. Time to refresh token, I close the ws via Consumer.Close()
    At that time, I dumped the goroutines. I found the goroutine leaks on persistConn readLoop and writeLoop
    goroutine 66 [select, 7 minutes]:
    net/http.(_persistConn).readLoop(0xc8200cd1e0)
    /usr/local/go/src/net/http/transport.go:976 +0xac7
    created by net/http.(_Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 67 [select, 7 minutes]:
net/http.(_persistConn).writeLoop(0xc8200cd1e0)
/usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(_Transport).dialConn
/usr/local/go/src/net/http/transport.go:686 +0xc9d

Connection retry counter and backoff delay should be reset on successful connection.

Hi,

I come here from cloudfoundry-community/stackdriver-tools#151, where code is being added to the stackdriver nozzle to force it to commit suicide via lager.Fatal when the connection retry limit is reached.

For reasons as yet unknown (guess what I'm doing after I have submit here), the nozzle in our test installation of PCF gets disconnected from the firehose pretty regularly, but is always able to reconnect. It uses the consumer's inbuilt retry logic to do this. Because the counter and backoff are not reset, after 8 such disconnections (and with 992 to go until suicide) the nozzle is waiting 1 minute between reconnects, and effectively dropping firehose data.

I can see good reasons for having a limit on the number of consecutive unsuccessful retry attempts, but users of noaa/consumer should not have to Fatal to reset these counters. We could manage our own retries, but then why bother having the functionality in noaa/consumer at all?

Simply setting the retry limit to 2^63-1 is not enough because then consecutive retries are not limited and the backoff quickly plateaus at 1 minute, and setting the max backoff to 500ms breaks exponential backoff when retrying unsuccessful connections.

/cc @johnsonj @knyar

Missing events.

Howdie!

Im using https://github.com/cloudfoundry-samples/fib-cpu in a newly deployed empty CF 211 VMWare installation. For each request to the app we get two log events

$ curl fib.test.cf.domain.com/10
$ cf logs fib
...
2015-06-23T15:26:40.28+0200 [App/0]      OUT Request /10 took 3.3326e-05 seconds to complete
2015-06-23T15:26:40.29+0200 [RTR/0]      OUT fib.test.cf.domain.com - [23/06/2015:13:26:40 +0000] "GET /10 HTTP/1.1" 200 0 2 "-" "curl/7.43.0" 10.230.31.7:51182 x_forwarded_for:"10.230.31.7" vcap_request_id:ce6fb1e7-ff26-497b-6c5c-b11cd47fb70d response_time:0.009590970 app_id:082fea25-f40b-4bff-8c12-7586447cb6c4

Using the noaa-client I would expect that shooting off 10 requests would yield 20 events but I get between 10-18 for each test I do.

Setup

$ GOPATH=/tmp go get github.com/cloudfoundry/noaa
$ cd /tmp/src/github.com/cloudfoundry/noaa/
$ git --no-pager show --summary 
commit 7f3b03afe00e1bf449cf1651a8623d878e20c70d
Author: John Tuley <[email protected]>
Date:   Thu Jun 18 16:50:33 2015 -0600

    Revert "Set gogo/protobuf version to be the same as the one in loggregator for travis builds"

    This reverts commit 3c78d4555842c8a7396fa4fc3ef18376c08a3e31.

    [#97345078]

Consuming and comparing events

term1$ cf logs fib > /tmp/logs-from-cf-logs
term2$ CF_ACCESS_TOKEN=wryyyy DOPPLER_ADDR=wss://doppler.test.cf.domain.com:443 GOPATH=/tmp go run firehose_sample/main.go > /tmp/noaa-client
term3$ repeat 10 curl fib.test.cf.domain.com/10 
term3$ cat /tmp/logs-from-cf-logs | egrep "App|RTR" | wc -l
20
term3$ cat /tmp/noaa-client | egrep "LogMessage" | wc -l  
17
$ cat /tmp/noaa-client | egrep "LogMessage"        
origin:"dea_logging_agent" eventType:LogMessage timestamp:1435066555822822483 logMessage:<message:"Request /10 took 3.3829e-05 seconds to complete" message_type:OUT timestamp:1435066555822820553 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"App" source_instance:"0" >  
origin:"router__0" eventType:LogMessage timestamp:1435066555829558583 logMessage:<message:"fib.test.cf.domain.com - [23/06/2015:13:35:55 +0000] \"GET /10 HTTP/1.1\" 200 0 2 \"-\" \"curl/7.43.0\" 10.230.31.3:51264 x_forwarded_for:\"10.230.31.3\" vcap_request_id:37c3dd2f-49d4-49e5-4d85-52cf57baac84 response_time:0.007996051 app_id:082fea25-f40b-4bff-8c12-7586447cb6c4\n" message_type:OUT timestamp:1435066555829556378 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"RTR" source_instance:"0" >  
origin:"dea_logging_agent" eventType:LogMessage timestamp:1435066555891428615 logMessage:<message:"Request /10 took 3.2354e-05 seconds to complete" message_type:OUT timestamp:1435066555891427679 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"App" source_instance:"0" >  
origin:"dea_logging_agent" eventType:LogMessage timestamp:1435066555972095337 logMessage:<message:"Request /10 took 7.346e-05 seconds to complete" message_type:OUT timestamp:1435066555972093060 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"App" source_instance:"0" >  
origin:"router__0" eventType:LogMessage timestamp:1435066555974757126 logMessage:<message:"fib.test.cf.domain.com - [23/06/2015:13:35:55 +0000] \"GET /10 HTTP/1.1\" 200 0 2 \"-\" \"curl/7.43.0\" 10.230.31.1:51266 x_forwarded_for:\"10.230.31.1\" vcap_request_id:5efaea2d-aee3-4c30-4e6d-91b354d5bf92 response_time:0.008960204 app_id:082fea25-f40b-4bff-8c12-7586447cb6c4\n" message_type:OUT timestamp:1435066555974755399 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"RTR" source_instance:"0" >  
origin:"dea_logging_agent" eventType:LogMessage timestamp:1435066556038989559 logMessage:<message:"Request /10 took 5.659e-05 seconds to complete" message_type:OUT timestamp:1435066556038988623 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"App" source_instance:"0" >  
origin:"dea_logging_agent" eventType:LogMessage timestamp:1435066556105148981 logMessage:<message:"Request /10 took 3.25e-05 seconds to complete" message_type:OUT timestamp:1435066556105136936 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"App" source_instance:"0" >  
origin:"router__0" eventType:LogMessage timestamp:1435066556107187549 logMessage:<message:"fib.test.cf.domain.com - [23/06/2015:13:35:56 +0000] \"GET /10 HTTP/1.1\" 200 0 2 \"-\" \"curl/7.43.0\" 10.230.31.5:51268 x_forwarded_for:\"10.230.31.5\" vcap_request_id:06cf7d10-f3d7-47fb-6227-86eea2813bdd response_time:0.005772654 app_id:082fea25-f40b-4bff-8c12-7586447cb6c4\n" message_type:OUT timestamp:1435066556107185859 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"RTR" source_instance:"0" >  
origin:"dea_logging_agent" eventType:LogMessage timestamp:1435066556187180248 logMessage:<message:"Request /10 took 7.2833e-05 seconds to complete" message_type:OUT timestamp:1435066556187178939 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"App" source_instance:"0" >  
origin:"router__0" eventType:LogMessage timestamp:1435066556190834073 logMessage:<message:"fib.test.cf.domain.com - [23/06/2015:13:35:56 +0000] \"GET /10 HTTP/1.1\" 200 0 2 \"-\" \"curl/7.43.0\" 10.230.31.4:51269 x_forwarded_for:\"10.230.31.4\" vcap_request_id:fdf08d66-fe09-4578-7545-921207b5ca2e response_time:0.008445217 app_id:082fea25-f40b-4bff-8c12-7586447cb6c4\n" message_type:OUT timestamp:1435066556190832236 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"RTR" source_instance:"0" >  
origin:"dea_logging_agent" eventType:LogMessage timestamp:1435066556273007272 logMessage:<message:"Request /10 took 3.2422e-05 seconds to complete" message_type:OUT timestamp:1435066556273005906 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"App" source_instance:"0" >  
origin:"router__0" eventType:LogMessage timestamp:1435066556276260011 logMessage:<message:"fib.test.cf.domain.com - [23/06/2015:13:35:56 +0000] \"GET /10 HTTP/1.1\" 200 0 2 \"-\" \"curl/7.43.0\" 10.230.31.3:51270 x_forwarded_for:\"10.230.31.3\" vcap_request_id:86dfedaf-27aa-4333-6542-5f17171afe9b response_time:0.006513990 app_id:082fea25-f40b-4bff-8c12-7586447cb6c4\n" message_type:OUT timestamp:1435066556276258104 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"RTR" source_instance:"0" >  
origin:"dea_logging_agent" eventType:LogMessage timestamp:1435066556336575706 logMessage:<message:"Request /10 took 3.2241e-05 seconds to complete" message_type:OUT timestamp:1435066556336574740 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"App" source_instance:"0" >  
origin:"dea_logging_agent" eventType:LogMessage timestamp:1435066556441946000 logMessage:<message:"Request /10 took 7.332e-05 seconds to complete" message_type:OUT timestamp:1435066556441944975 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"App" source_instance:"0" >  
origin:"router__0" eventType:LogMessage timestamp:1435066556445317709 logMessage:<message:"fib.test.cf.domain.com - [23/06/2015:13:35:56 +0000] \"GET /10 HTTP/1.1\" 200 0 2 \"-\" \"curl/7.43.0\" 10.230.31.2:51272 x_forwarded_for:\"10.230.31.2\" vcap_request_id:8ee423bd-88d7-4b02-5605-290e9935d259 response_time:0.008849451 app_id:082fea25-f40b-4bff-8c12-7586447cb6c4\n" message_type:OUT timestamp:1435066556445315729 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"RTR" source_instance:"0" >  
origin:"dea_logging_agent" eventType:LogMessage timestamp:1435066556529838323 logMessage:<message:"Request /10 took 7.2413e-05 seconds to complete" message_type:OUT timestamp:1435066556529837317 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"App" source_instance:"0" >  
origin:"router__0" eventType:LogMessage timestamp:1435066556533252926 logMessage:<message:"fib.test.cf.domain.com - [23/06/2015:13:35:56 +0000] \"GET /10 HTTP/1.1\" 200 0 2 \"-\" \"curl/7.43.0\" 10.230.31.6:51273 x_forwarded_for:\"10.230.31.6\" vcap_request_id:ac3e6844-3b6f-4214-6995-c888113b930d response_time:0.008822410 app_id:082fea25-f40b-4bff-8c12-7586447cb6c4\n" message_type:OUT timestamp:1435066556533250929 app_id:"082fea25-f40b-4bff-8c12-7586447cb6c4" source_type:"RTR" source_instance:"0" >

Let me know if you need any other info!

Feature request: accept https_proxy

The cf CLI relies on noaa to talk to loggregator.
It appears cf logs myapp works when behind a proxy when http_proxy is set, not with https_proxy.
As https_proxy seems to be the standard environment variable to use to specify an http proxy to an https endpoint, in cf CLI help and documentation we suggest that (although by virtue of being implemented in golang it supports both, in lower and in upper case).

Can noaa also be made to accept https_proxy?

Allow more flexible recent logs URL structure

We are thinking of submitting a PR on noaa and would like to discuss the proposed change prior to doing that.

We have a service broker which needs to surface a special logs endpoint along the lines of a doppler endpoint, but we would like to get away from the hardcoded URL structure that noaa presumes for recent logs endpoints. The proposed change maintains the current noaa behaviour as default but allows the caller to specify their own function for constructing the recent logs endpoint URL from its constituents. The draft code, and tests, are here: https://github.com/glyn/noaa/commit/07aabc4509d8ea37a81f596d5918dc5094e5b90f

If this is acceptable, then -- after more testing -- we'll submit a PR. Any feedback gratefully received.

Modify consumer api to return detailed error responses

The consumer api currently returns only the message in case of any error that occurs during any of the api calls. Can we modify the api to also return some more detail about the error like the statusCode/errorType for callers who would like a little more information about what went wrong and react appropriately.

go get fails building noaa

scoen@magellan ~/go
$ go get github.com/cloudfoundry/noaa
# github.com/cloudfoundry/sonde-go/events
src/github.com/cloudfoundry/sonde-go/events/envelope.pb.go:237: undefined: proto.NewRequiredNotSetError
src/github.com/cloudfoundry/sonde-go/events/envelope.pb.go:245: undefined: proto.NewRequiredNotSetError
src/github.com/cloudfoundry/sonde-go/events/envelope.pb.go:931: undefined: proto.NewRequiredNotSetError
src/github.com/cloudfoundry/sonde-go/events/envelope.pb.go:934: undefined: proto.NewRequiredNotSetError
src/github.com/cloudfoundry/sonde-go/events/error.pb.go:70: undefined: proto.NewRequiredNotSetError
src/github.com/cloudfoundry/sonde-go/events/error.pb.go:78: undefined: proto.NewRequiredNotSetError
src/github.com/cloudfoundry/sonde-go/events/error.pb.go:85: undefined: proto.NewRequiredNotSetError
src/github.com/cloudfoundry/sonde-go/events/error.pb.go:277: undefined: proto.NewRequiredNotSetError
src/github.com/cloudfoundry/sonde-go/events/error.pb.go:280: undefined: proto.NewRequiredNotSetError
src/github.com/cloudfoundry/sonde-go/events/error.pb.go:283: undefined: proto.NewRequiredNotSetError
src/github.com/cloudfoundry/sonde-go/events/error.pb.go:283: too many errors

As noaa is included with cf-release, building the firehose sample works there as long as you set GOPATH:

$  cd ~/workspace/cf-release/src/loggregator/src/github.com/cloudfoundry/noaa/
$  export GOPATH=~/workspace/cf-release/src/loggregator
$  go build -o bin/firehose_sample firehose_sample/main.go

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.