knative / test-infra Goto Github PK
View Code? Open in Web Editor NEWTest infrastructure for the Knative project
License: Apache License 2.0
Test infrastructure for the Knative project
License: Apache License 2.0
Git issue doesn't support priority. Would be nice to support P0, P1, P2 for issues opened.
e.g., https://gubernator.knative.dev/pr/knative/serving/1833 shows no results.
Probably because GitHub access is not configured yet: https://gubernator.knative.dev/github_auth/pr
Sometimes you want multiple reviewers to weigh in before a change is merged, but Prow always merges as soon as the minimum set is satisfied (AFAIK).
Internally, we have the concept of WANT_LGTM
to enumerate those you want an /lgtm
from before merging a change. I feel like such a concept would be useful in Prow as well.
e.g. I'd like to write:
/want-lgtm @fejta
/want-lgtm @cjwagner
To have tide
block merges until both have given the /lgtm
, this should also implicitly /assign
them.
I think it would also be useful to have /want-lgtm all
that waits for /lgtm
from all assigned reviewers.
cc @cjwagner @fejta @grantr @adrcunha @jessiezcc WDYT?
/area test-and-release
/kind dev
Expected Behavior
When Prow job fails to run properly, we should get alert/notification automatically
Actual Behavior
We are finding out the failing job run manually.
This way, working group triage will get the visibility to flaky tests and assign proper owners.
The bug should provide data points about flakiness per test. Prefer one issue per flaky test.
This was brought up from scaling working group.
"What about automatically notifying if LGTM is lost because of a merge?"
For example, in https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_eventing/276/pull-knative-eventing-build-tests/1027295713966428160/ there's a single error:
W0808 20:50:44.825] pkg/sources/github/receive_adapter/receive_adapter.go:20:2: imported and not used: "context"
but it's hidden in a long list of files, which makes it hard to spot.
No bot issues. :)
We've seen PRs where the bot complains that the CLA was not signed (even though it was before), like in knative/serving#1435; or where a PR created with pair programming like in knative/serving#1390 or knative/serving#1352.
For knative/serving#1352, an adming was able to solve the problem by temporarily disabling and then re-enabling a setting about admins being able to edit PRs.
I hit an issue with the eventing e2e tests where the serving install fails because the Istio sidecar injector is not available to handle requests.
As we drive up code coverage numbers, # of tests (especially unit tests) increase significantly.
@evankanderson reported overall it's taking 30+min to run the tests, we should look at how much we can save if we run tests concurrently. go test has a parallel option that supports concurrency.
We can start with unit tests first.
none of the jobs can be rerun with the command suggested by the re-run button
for example
kubectl create -f "https://prow.knative.dev/rerun?prowjob=e2aeff0e-99c2-11e8-80e3-0a580a3c0043"
In knative/build#335 the e2e tests report (https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_build/335/pull-knative-build-integration-tests/1039211002157076480/):
W0910 17:56:49.696] Fetching server config for us-central1
I0910 17:56:49.811] Latest GKE is 1.10.7, from [1.10.7-gke.1, /1.10.6-gke.3, /1.10.6-gke.2, /1.9.7-gke.6, /1.9.7-gke.5, /1.9.6-gke.2]
However, cluster creation fails with:
W0910 17:57:41.815] 2018/09/10 17:57:41 process.go:152: Step 'gcloud container clusters create --quiet --enable-autoscaling --min-nodes=1 --max-nodes=3 --scopes=cloud-platform --project=knative-boskos-05 --zone=us-central1-a --machine-type=n1-standard-4 --image-type=cos --num-nodes=3 --network=kbuild-e2e-net1039211002157076480 --cluster-version=1.10.7 kbuild-e2e-cls1039211002157076480' finished in 681.070848ms
W0910 17:57:42.751] 2018/09/10 17:57:42 main.go:309: Something went wrong: starting e2e cluster: error creating cluster: error during gcloud container clusters create --quiet --enable-autoscaling --min-nodes=1 --max-nodes=3 --scopes=cloud-platform --project=knative-boskos-05 --zone=us-central1-a --machine-type=n1-standard-4 --image-type=cos --num-nodes=3 --network=kbuild-e2e-net1039211002157076480 --cluster-version=1.10.7 kbuild-e2e-cls1039211002157076480: exit status 1
Although this is not Knative-specific, would be nice to get it fixed.
Prow contains several places where kubernetes is hardcoded. For example, the default reply from Prow robot about an approval contains links to Kubernetes documentation like https://go.k8s.io/bot-commands and https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process. See #73 (comment) as an example.
This will simplify presubmit tests in the repos, unify features and increase coverage.
Build tests helper should:
vendor
present);Unit test helper should
W0905 20:12:53.535] # github.com/knative/pkg/test
W0905 20:12:53.535] test/e2e_flags.go:28:2: imported and not used: "github.com/knative/pkg/test/logging"
Integration test helper should
--emit-metrics
flag, likelocal options=""
(( EMIT_METRICS )) && options="--emit-metrics"
./test/e2e-tests.sh ${options}
kind (kubernetes in docker) might make e2e tests faster since there's no need to create an external cluster. However, it needs to be taken into consideration that we have to start a k8s cluster with a known, public version (e.g. 1.11.1).
e.g. knative/serving#1977 lists no changed files in the log
Since this will be our primary repo for productivity related projects, is it feasible to rename the repo to Productivity? "test-infra" is a bit limited in terms of scope.
When I open code coverage issues for each repo, I wish we can clone an existing issue, and just need to do minor updates to create a new issue.
Prow fails running any test on knative/build-templates#50
I0815 15:57:40.426] Call: /workspace/./test-infra/jenkins/../scenarios/kubernetes_execute_bazel.py -- ./test/presubmit-tests.sh --unit-tests
W0815 15:57:40.449] Run: ('./test/presubmit-tests.sh', '--unit-tests')
W0815 15:57:40.547] Traceback (most recent call last):
W0815 15:57:40.547] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_execute_bazel.py", line 66, in <module>
W0815 15:57:40.547] main(ARGS.cmd + ARGS.args)
W0815 15:57:40.548] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_execute_bazel.py", line 54, in main
W0815 15:57:40.548] return_code = call(*cmd)
W0815 15:57:40.548] File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_execute_bazel.py", line 40, in call
W0815 15:57:40.548] return subprocess.call(cmd)
W0815 15:57:40.548] File "/usr/lib/python2.7/subprocess.py", line 522, in call
W0815 15:57:40.548] return Popen(*popenargs, **kwargs).wait()
W0815 15:57:40.548] File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
W0815 15:57:40.549] errread, errwrite)
W0815 15:57:40.549] File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
W0815 15:57:40.549] raise child_exception
W0815 15:57:40.549] OSError: [Errno 2] No such file or directory
E0815 15:57:40.552] Command failed
I0815 15:57:40.552] process 648 exited with code 1 after 0.0m
E0815 15:57:40.553] FAIL: pull-knative-build-templates-unit-tests
Does the Prow config for the build-templates repo need to be updated?
/assign @adrcunha
Now we have check_links_in_markdown function, I suggest to make it part of automatic presubmit check for all .md changes, so that no individual PR need to worry about add such tests.
This came up from scaling working group with @josephburnett.
PR owners often need to wait for approve/lgtm to close on a PR. However, from the reviewer side, they also have difficulty knowing when something needs their attention. It's normal for folks to get like a bazillion emails about Knative each day. We have an internal chrome pluggin for our google code review tool that gives me a little red flag in the corner of my browser when a cr needs my attention. something like that would be useful for knative. so individuals don't have to ping each person on slack.
Perhaps git notification can help. Let's discuss options.
/area test-and-release
/kind dev
There is no reason folks should have to build and push the e2e images and conformance test images whenever they want to run the tests, the images hardly ever change and they don't do much. They should be available publicly so folks can get up and running with the tests faster, and they should be regularly updated.
Note this will have to play nicely with a person's $DOCKER_REPO_OVERRIDE
.
The tests expect anyone running the e2e or conformance tests to have built and pushed the images to their own repo.
n/a
Thanks @jonjohnsonjr !
e.g., in https://gubernator.knative.dev/build/knative-prow/logs/ci-knative-serving-continuous/1039348282343559168 the e2e tests failed but were not reported
If a contributor has an older version of dep there may be unnecessary edits to Gopkg.lock
when opening a PR. Future PRs from other contributors with a newer version of dep may change the lock file. This thrashing is noisy.
It'd be nice if we could enforce a minimal version of dep as part of the presubmit checks.
Latest release at this time is v0.5.0
https://github.com/golang/dep/releases
Y'know, in case I'm hit by a bus :)
Internal Prow switched to timestamp-indexed builds, which cause resource names to be too long:
W0803 18:42:07.634] ERROR: (gcloud.container.clusters.create) ResponseError: code=400, message=cluster.name must be less than 40 characters.
W0803 18:42:07.682] 2018/08/03 18:42:07 process.go:152: Step 'gcloud container clusters create --quiet --enable-autoscaling --min-nodes=1 --max-nodes=3 --scopes=cloud-platform --project=gke-gerrit-boskos-02 --zone=us-central1-a --machine-type=n1-standard-4 --image-type=cos --num-nodes=3 --network=kbuild-templates-e2e-net1025451389398028288 --cluster-version=1.10.5-gke.3 kbuild-templates-e2e-cls1025451389398028288' finished in 657.926618ms
W0803 18:42:09.506] 2018/08/03 18:42:09 main.go:309: Something went wrong: starting e2e cluster: error creating cluster: error during gcloud container clusters create --quiet --enable-autoscaling --min-nodes=1 --max-nodes=3 --scopes=cloud-platform --project=gke-gerrit-boskos-02 --zone=us-central1-a --machine-type=n1-standard-4 --image-type=cos --num-nodes=3 --network=kbuild-templates-e2e-net1025451389398028288 --cluster-version=1.10.5-gke.3 kbuild-templates-e2e-cls1025451389398028288: exit status 1
i have a kubernetes cluster that is bootstrapped by kubeadm
, but currently, there is no way to let knative components run e2e on it
In the prow logs, when creating a knative cluster with kubetest, we see:
util.go:132: Please use kubetest --gcp-service-account=/etc/service-account/service-account.json (instead of deprecated GOOGLE_APPLICATION_CREDENTIALS=/etc/service-account/service-account.json)
We should heed that warning and update our kubetest args
Currently the testgrid bucket in gubernator is hardcoded, thus the version deployed is locally modified. Proper bucket definition (flag? yaml?) and deploy command (Makefile
) needs to be done.
This came up from discussion with Matt.
Number of (/test, /retest) is a good measurement for test flakiness. Let's include it as part of project health metric.
FlakinessSignal: Number of (/test, /retest) / Number of PR
Ideally, as number of PR grows, we'd like above ratio not to go up.
https://testgrid.knative.dev/knative-pkg#coverage
The changes from #110 were pushed yesterday, and the other repos show coverage numbers for today. It seems that the target change caused the coverage report on knative/pkg to break.
The makefile only build is locally, but does not push it. we need to push it to the repo as well
Current approach using git
frequently produces wrong results.
Now that we have prow-tests
image, we can add a tool to it that would query GitHub about the changes and report the changes files in the current PR.
as title suggests
We need: google/go-containerregistry#253
This breaks some container analysis tooling, since some of the layer tarballs are malformed.
cc @imjasonh
Although this is not Knative-specific, would be nice to get it fixed.
Gubernator contains several places where kubernetes is hardcoded. For example, when clicking "Ongoing" or "Incoming" in https://gubernator.knative.dev/pr you'll be taken to the GitHub search page with user hardcoded to kubernetes.
@adrcunha, can you pls document steps on how to onboard a new Knative repo to Knative Prow?
The e2e tests currently fail for me attempting to download a GKE release.
Will download and extract kubernetes-test.tar.gz from https://storage.googleapis.com/kubernetes-release-gke/release/v1.10.5-gke.3
Right now I can only run the e2e tests via prow for a PR. The e2e tests need to be runable without special privileges or infrastructure.
For user trust reasons.
The go coverage report only looks at files that have *_test.go files. This provides wrong coverage outputs and also bumps up the coverage numbers. It is not reflective of the current state.
go test does specify that the package has no test files and we should capture this behaviour in our report.
For eg. in knative/pkg/test, we have some go files but no _test files.
knative/pkg/test$ go test
? github.com/knative/pkg/test [no test files]
Running the update-codegen.sh script from eventing fails because PROW_JOB_ID
is unbounded. This var should have a default automatically set when not running on prow.
$ ./hack/update-codegen.sh
./hack/../vendor/github.com/knative/test-infra/scripts/library.sh: line 32: PROW_JOB_ID: unbound variable
Not sure if this is something we configure to point to knative/test-infra
knative/pkg#97 (comment)
Now that test-infra
is stable, we should vendor the helper scripts it in the other repos and remove the fetching using docker from prow-tests image.
In https://gubernator.knative.dev/pr/knative_eventing/376, the links to the eventing repo are https://github.com/knative/knative_eventing, but they should be https://github.com/knative/eventing.
Root cause:
2018/08/07 08:28:18 error processing import paths in "config/webhook.yaml": no token in bearer response:
W0807 08:48:47.729] {"errors":[{"code":"DENIED","message":"Token exchange failed for project 'knative-releases'. Caller does not have permission 'storage.buckets.get'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control"}]}
When I visit: https://testgrid.knative.dev/knative-serving#continuous
The repo is now ready to be open, and this is a requirement for #30.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.