Maybe TEST_CREDENTIAL_FILE?

Many of M-Lab's Dockerfiles have a line like the following to include the git commit of the repo in the software they build:

RUN go get -v -ldflags "-X github.com/m-lab/go/prometheusx.GitShortCommit=$(git log -1 --format=%h)"  .

This has the undesired side-effect of pulling in the prometheusx package for software that has nothing to do with Prometheus. Moreover, creating an application that uses any package that includes prometheusx results in polluting the flags when printing its help message. Below is an example:

$ cat main.go
package main

import (
	"flag"
	"fmt"
	"github.com/m-lab/traceroute-caller/parser"
)

var someFlag = flag.Bool("f", false, "Some flag")

func main() {
	flag.Parse()
	fmt.Printf("%+v\n", parser.Scamper1{})
}

$ go build -o main .
$ ./main -h
Usage of ./main:
  -f	Some flag
  -prometheusx.listen-address string
    	 (default ":9990")
$

logx fails when weird things happen with task scheduling on Travis.
bqext fails every now and then for reasons I have not investigated.

It's frustrating, and I wish the associated tests were more reliable.

Currently there are several possible implementations of a content.Provider, depending on the scheme used in the URL passed to FromURL. These implementations differ in how they use the passed-in context.

httpProvider.Get(ctx) should use the passed-in context to allow the caller to define a timeout, rather than adding its own (unconfigurable) one.

ListenAndServe*Async both may call log.Fatalf, which is buried down under the server shutdown. Though we don't expect to be shutting down servers, it is still worrisome that a server shutdown may crash the whole process, and is very hard to see when calling the top level functions.

Next() should be able to fill out a struct argument.

Perhaps this should take a server, and add just the prometheus handler. Of course, it then becomes just a single line, so maybe it isn't really useful.

If we also have a separate HandlePprof, then the code is better self documenting.

See: https://travis-ci.org/m-lab/go/builds/514878680

$ go test -covermode=count -coverprofile=_bqext.cov -v ./bqext -tags=$TEST_TAGS
=== RUN   TestGetTableStats
2019/04/02 20:50:47 http.go:55: Request:
&http.Request{Method:"GET", URL:(*url.URL)(0xc00013e580), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"X-Goog-Api-Client":[]string{"gl-go/1.11.6 gccl/20180226"}, "User-Agent":[]string{"google-api-go-client/0.5"}}, Body:io.ReadCloser(nil), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:0, TransferEncoding:[]string(nil), Close:false, Host:"www.googleapis.com", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"", RequestURI:"", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.valueCtx)(0xc0001ccb70)}
2019/04/02 20:50:47 dataset_integration_test.go:64: Get https://www.googleapis.com/bigquery/v2/projects/mlab-testing/datasets/go/tables/TestGetTableStats?alt=json&prettyPrint=false: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fbigquery" not defined
FAIL	github.com/m-lab/go/bqext	0.022s
The command "go test -covermode=count -coverprofile=_bqext.cov -v ./bqext -tags=$TEST_TAGS" exited with 1.

Also see #88

Migrate clients and then delete the deprecated code in go/cloud/bqfake.

(There may not be any clients)

The following packages should be consolidated:

• bqext
• bqx
• cloud/bqfake
• cloudtest
• cloudtest/dsfake
• dataset

This could be reduced to two (preferred) or three packages (at most). If some packages should remain as-is they should be moved to subdirectories of one or the other.

For example:

• bqx :: could include bqext, dataset, and bqxtest package for bqfake and dsfake.
• cloudtest :: could include bqfake an dsfake, and eliminate cloud.

Or, we could combine storagex and cloudtest GCS methods in a helpful way.

Other options are also possible.

Right now, we use the /proc filesystem as a proxy for boot time. This is not consistent between containers in a pod.

See m-lab/ndt-server#93 for the use case.

But the upshot is: We should have a single string variable, set to the output of git describe --long and then parse that output into its three component parts (tag, commit count, commit ID), use each of the component parts as a label for the metric, and then set the value of the metric to be the commit ID, interpreted as a base-16 integer, converted into a prometheus-compatible float64.

Issue #46 reported this in April, however recently CRON builds as well as ordinary merges to master are failing due to test failures in the bqext package:

For example:

=== RUN   TestGetTableStatsMock
--- FAIL: TestGetTableStatsMock (0.00s)
    dataset_test.go:104: [Location: US != ]
=== RUN   TestResultQuery
--- PASS: TestResultQuery (0.00s)
=== RUN   TestDestQuery
--- PASS: TestDestQuery (0.00s)
FAIL
coverage: 37.9% of statements
FAIL	github.com/m-lab/go/bqext	0.010s

M-Lab is using a colloquial sense of "anonymize" that means "de-identify" which differs from the technical definition of anonymization from https://www.rfc-editor.org/rfc/rfc6973#section-6.1.1 which depends on non-uniqueness (multiple events). This difference is confusing to some audiences, and may imply more than what this package actually does.

See also: #154 (review)

The anonymize package should be renamed to more clearly match its intention and application.

PR #1

For production daemons, logging the command line flags is an excellent feature.

For cli tools that use ArgsFromEnv, logging the command line flags is not excellent.

It would be helpful for there to be a way to make this logging conditional, or disabled for CLI use cases.

Do we want to unify them, so code would look like:

  datatypes := flagx.StringArray("datatype", "Help text")

instead of the two-stage init that they currently require?

Unit tests no longer pass on Travis.

We had to disable the other project "go vet" in travis in order to fix breakage from upstream library changes.

Once the other projects are working again, we should uncomment these.

• echo $TEST_CREDENTIALS_mlab_testing | base64 -d > travis-testing.key
• gcloud auth activate-service-account --key-file=travis-testing.key

Then replace this in m-lab/go/.travis.yml

github.com/m-lab/go/logx

github.com/m-lab/go/logx

../go/logx/logx.go:40:19: logger.Writer undefined (type *log.Logger has no field or method Writer)
github.com/m-lab/etl/etl
github.com/m-lab/go/bqx
The command "go get -v -t ./..." failed and exited with 2 during .

Also see #88

bqx should be grouped with the cloud packages. There is now a copy in go/cloud/bqx, and clients should migrate to that path.

./go/cloud/bqx/schema_test.go:21:
./etl-gardener/job-service/job-service_test.go:19:
./etl-gardener/tracker/tracker_race_test.go:16:
./etl-gardener/tracker/tracker.go:30:
./etl-gardener/tracker/tracker_test.go:14:
./etl-gardener/tracker/handler_test.go:16:
./etl-gardener/ops/ops_test.go:10:
./etl-gardener/ops/actions_test.go:13:
./etl/cmd/generate_schema_docs/main.go:32:
./etl/cmd/update-schema/update.go:25:
./etl/parser/ndt_result.go:50://
./etl/schema/schema_test.go:9:
./etl/schema/ndt_result_test.go:7:
./etl/schema/schema.go:10:
./etl/schema/tcpinfo.go:10:
./etl/schema/pt_schema.go:10:
./etl/schema/ndt_result.go:5: