Repos with this problem:

• guest-agent - GoogleCloudPlatform/guest-agent#3
• compute-images-packages - GoogleCloudPlatform/compute-image-packages#859

{
 insertId: "3206wog42gdrsb"  
 jsonPayload: {
  author: "adjackura"   
  component: "hook"   
  event-GUID: "a5d1a300-ebb8-11e9-9868-406a0c1c7f7f"   
  event-type: "issue_comment"   
  file: "prow/plugins/approve/approvers/owners.go:164"   
  func: "k8s.io/test-infra/prow/plugins/approve/approvers.Owners.GetSuggestedApprovers"   
  level: "warning"   
  msg: "Couldn't find/suggest approvers for each files. Unapproved: [""]"   
  org: "GoogleCloudPlatform"   
  plugin: "approve"   
  pr: 2   
  repo: "guest-agent"   
  url: "https://github.com/GoogleCloudPlatform/guest-agent/pull/2#issuecomment-540843760"   
 }
 labels: {…}  
 logName: "projects/oss-prow/logs/hook"  
 receiveTimestamp: "2019-10-10T23:49:58.731720740Z"  
 resource: {…}  
 severity: "ERROR"  
 timestamp: "2019-10-10T23:49:53Z"  
}

#270 shows X

https://prow.gflocks.com/view/gcs/oss-prow/pr-logs/pull/GoogleCloudPlatform_oss-test-infra/270/pull-oss-test-infra-check-testgrid-config/1228047829478412290 shows green

https://prow.gflocks.com/prowjob?prowjob=488e4009-4e9c-11ea-86af-b23cf998d5f2 shows job failed

/assign @chases2

What is going on here?

we return a 200 You don't have permission to rerun that job when I click rerun.

Starting today, google-oss-robot account is posting incorrect links in 'test failed' comments. The links previously pointed to prow.gflocks.com.... and now point back to the PR itself.

See working: GoogleCloudPlatform/guest-oslogin#4
And broken: GoogleCloudPlatform/guest-oslogin#5

/assign @cjwagner

https://prow.gflocks.com/view/gcs/oss-prow/logs/ci-test-infra-autobump-prow/1182074783416717312

Bumping: 'gcr.io/k8s-prow/' images to v20191009-0492f1d6e ...
Attempting to bump the following files: prow/cluster/cert-manager.yaml prow/cluster/cluster.yaml prow/cluster/grandmatriarch_default.yaml prow/cluster/grandmatriarch_test-pods.yaml prow/config.yaml prow/prowjobs/GoogleCloudPlatform/oss-test-infra/gcp-oss-test-infra-config.yaml
bump.sh completed successfully!
[master 7ac2309] Bump prow from v20191008-bbd30bc09 to v20191009-0492f1d6e
 5 files changed, 15 insertions(+), 15 deletions(-)
Pushing commit to google-oss-robot/oss-test-infra:autobump...
remote: Repository not found.
fatal: repository 'https://google-oss-robot:@github.com/google-oss-robot/oss-test-infra/' not found

cc @cjwagner

https://prow.gflocks.com/tide is pretty blank, wonder if some config was missing? (reported by @hopkiw)

cc @cjwagner

As a collaborator, clarketm can lgtm pull requests.

/assign @fejta

Otherwise things will explode next week-ish.

It would be great to expose this feature to existing prow users. The infrastructure is in place - it just needs to be configured.

Tasks:

• A GoogleCloudPlatform owner to create a Github Oauth App (step 1 and 2)
• Define which users or teams will be delegated rerun permission.
• Update the deck Kubernetes deployment and secrets (github-oauth-config and cookie)., #100

Create a dedicate build cluster

This allows for isolation between the Prow cluster and the jobs running on it. Furthermore, this enables the introduction of "trusted" jobs for general deployments, image builds, and prow bumps in order to automate repeatable, mundane tasks without exposing the cluster (or secrets) to unprivileged users.

• Create a GCP Prow build cluster.
• Migrate jobs to the build cluster.
• Set the build cluster as the default cluster.

/assign

Enable ghproxy for Github API calls to limit token usage.

#100

@cjwagner - We'll probably want to set up ghproxy for this cluster since listing the member of this team currently costs 25 API ratelimit tokens. That being said, the cache entry would become invalid pretty quickly as I suspect that team has high churn and this bot token isn't heavily utilized.

/assign

{
 insertId: "1e8iuueg2rjyho6"  
 jsonPayload: {
  component: "plank"   
  error: "errors syncing: [error listing pods in cluster "kubeflow": Get https://35.196.213.148/api/v1/namespaces/test-pods/pods?labelSelector=created-by-prow%3Dtrue: dial tcp 35.196.213.148:443: connect: connection refused], errors reporting: []"   
  file: "prow/cmd/plank/main.go:149"   
  func: "main.main.func2"   
  level: "error"   
  msg: "Error syncing."   
 }
 labels: {…}  
 logName: "projects/oss-prow/logs/plank"  
 receiveTimestamp: "2020-01-21T19:02:28.929499401Z"  
 resource: {…}  
 severity: "ERROR"  
 timestamp: "2020-01-21T19:02:24Z"  
}

/kind oncall-hotlist

{
 insertId: "wqxj9cg6l82b79"  
 jsonPayload: {
  component: "tide"   
  file: "prow/config/config.go:1258"   
  func: "k8s.io/test-infra/prow/config.parseProwConfig"   
  level: "warning"   
  msg: "The `plank.allow_cancellations` setting is deprecated. It will be removed and set to always true in March 2020"   
 }
 labels: {…}  
 logName: "projects/oss-prow/logs/tide"  
 receiveTimestamp: "2019-12-04T22:56:59.879872048Z"  
 resource: {…}  
 severity: "ERROR"  
 timestamp: "2019-12-04T22:56:55Z"  
}

/kind oncall-hotlist

We should stop sending this flag:

We already define a service account:

We should annotate it with GCP SA rights: https://github.com/kubernetes/test-infra/tree/master/experiment/workload-identity

If the flag is unset then it will use default creds:
https://github.com/kubernetes/test-infra/blob/master/prow/cmd/gerrit/main.go#L240

https://github.com/kubernetes/test-infra/blob/164c5e85105f85bb8fd8c181a085911a3d1010fd/pkg/io/opener.go#L57-L60

https://godoc.org/cloud.google.com/go/storage#hdr-Creating_a_Client

Work:

• Migrate gerrit - #210
• Migrate grandmatriach - #224 #226
• Migrate pod utils
• Migrate testgrid image upload - #219
• Migrate prow upgrade - #221
• Migrate gcp-guest - (issue #223)
• Migrate espv2 - issue #222 - #227

Starting today, users trying to view prow jobs via the 'gubernator' interface report seeing only the main interface elements (logo, job history, artifacts link etc.) followed by a blank black page, where previously the output of build-log.txt was present. The build log is still discoverable through the artifacts link.

This is blocking a PR for esp-v2: #317

==================== Test output for //config/tests/testgrids:go_default_test:
--- FAIL: TestKubernetesProwInstanceJobsMustHaveMatchingTestgridEntries (3.20s)
    config_test.go:436: Job ci-test-infra-resultstore-upload does not have a matching testgrid testgroup

This was working earlier in the day. I don't know how it started failing, as there were no other PRs (other than esp-v2).

where can I find the repo for the google cloud cli?

https://github.com/istio/test-infra/tree/master/prow/cluster/gcsweb
https://github.com/istio/test-infra/blob/ed60c0a8b05037ce92a5337788f96d6d6b26d987/prow/config.yaml#L44

/assign

What would you like to be added:
Add knative build controller to oss-test-infra prow instance.
Why is this needed:
We want to use knative build step functionality

ref #19 (comment)

#159 (comment)

textPayload: "error retrieving resource lock default/prow-sinker-leaderlock: configmaps "prow-sinker-leaderlock" is forbidden: User "system:serviceaccount:default:sinker" cannot get resource "configmaps" in API group "" in the namespace "default"

textPayload: "external/io_k8s_client_go/tools/cache/reflector.go:98: Failed to watch *v1.ProwJob: unknown (get prowjobs.prow.k8s.io)"

https://prow.gflocks.com/view/gcs/oss-prow/pr-logs/pull/GoogleCloudPlatform_oss-test-infra/259/pull-oss-test-infra-check-testgrid-config/1227020925258436608

/assign @TAOXUY
/kind bug

Getting this error on startup:

jsonPayload: {
  component: "tide"   
  file: "prow/flagutil/github.go:90"   
  func: "k8s.io/test-infra/prow/flagutil.(*GitHubOptions).Validate"   
  level: "error"   
  msg: "It doesn't look like you are using ghproxy to cache API calls to GitHub! This has become a required component of Prow and other components will soon be allowed to add features that may rapidly consume API ratelimit without caching. Starting May 1, 2020 use Prow components without ghproxy at your own risk! https://github.com/kubernetes/test-infra/tree/master/ghproxy#ghproxy" 

https://github.com/GoogleCloudPlatform/oss-test-infra/blob/master/prow/config.yaml#L59 that's set to 48h

while we still have super old presubmit entries on deck? (https://prow.gflocks.com/?repo=kunit-review.googlesource.com%2Flinux&type=presubmit)

/assign @hopkiw
#202

Instructions:

• Run enable-workload-identity.sh on the build cluster
  • cluster: gcp-guest
• Run bind-service-accounts.sh for the service account pair
  • secretName: daisy-service-account
• Ensure job does not depend on the secret.json file existing
• Update job to use a serviceAccountName, stop using the secret

{
 insertId: "1io8wcxf6ejq1m"  
 jsonPayload: {
  component: "crier"   
  error: "error setting status: status code 404 not one of [201], body: {"message":"Not Found","documentation_url":"https://developer.github.com/v3/repos/statuses/#create-a-status"}"   
  file: "prow/crier/controller.go:279"   
  func: "k8s.io/test-infra/prow/crier.(*Controller).processNextItem"   
  jobName: "b3623ad8-c82f-11e9-b096-0a580a2c044d"   
  jobStatus: {
   build_id: "1166055611327057920"    
   completionTime: "2019-08-26T18:43:06Z"    
   description: "Job succeeded."    
   pod_name: "b3623ad8-c82f-11e9-b096-0a580a2c044d"    
   prev_report_states: {…}    
   startTime: "2019-08-26T18:31:21Z"    
   state: "success"    
   url: "https://prow.gflocks.com/view/gcs/oss-prow/pr-logs/pull/kunit-review.googlesource.com_linux/2370/kunit-test-presubmit/1166055611327057920"    
  }
  level: "error"   
  msg: "failed to report job"   
  prowjob: "default/b3623ad8-c82f-11e9-b096-0a580a2c044d"   
 }
 labels: {…}  
 logName: "projects/oss-prow/logs/crier"  
 receiveTimestamp: "2019-09-28T00:07:44.841801148Z"  
 resource: {…}  
 severity: "ERROR"  
 timestamp: "2019-09-28T00:07:41Z"  
}

That's an odd url

/kind oncall-hotlist

See #125

#155

Is autobumping on this repo a work-in-progress?

When I click on the "rerun" button on a Prow job in Deck, I get redirected to https://prow.gflocks.com/github-login/redirect, which results in a Domain Not Found error. The job does not rerun.

Rerun needs to be reconfigured somewhere, but I don't know exactly where.

/assign @clarketm

Example:

{
  component: "gerrit"   
  error: "failed to query gerrit changes: API call to https://kunit-review.googlesource.com/a/changes/?n=5&o=CURRENT_REVISION&o=CURRENT_COMMIT&o=CURRENT_FILES&o=MESSAGES&q=project:linux failed: 403 Forbidden"   
  file: "prow/gerrit/client/client.go:276"   
  func: "k8s.io/test-infra/prow/gerrit/client.(*gerritInstanceHandler).queryAllChanges"   
  level: "error"   
  msg: "fail to query changes for project linux"   
}

Interestingly, these errors happen every minute, but only between the 42nd and 49th minutes of each hour.
Outside of this window queries appear to succeed. Example:

{
  component: "gerrit"   
  file: "prow/gerrit/client/client.go:315"   
  func: "k8s.io/test-infra/prow/gerrit/client.(*gerritInstanceHandler).queryChangesForProject"   
  level: "info"   
  msg: "Find 5 changes from query [project:linux]"   
}

/kind bug

Currently, this repo uses Makefiles to perform build and deploy operations. We would like to replace each make target with its equivalent bazel command.

/help

https://prow.gflocks.com/view/gcs/oss-prow/logs/post-oss-test-infra-deploy-prow/1218256511885643776

/kind oncall-hotlist
/assign

Example:

{
  component: "tide"   
  controller: "status-update"   
  error: "status code 404 not one of [201], body: {"message":"Not Found","documentation_url":"https://developer.github.com/v3/repos/statuses/#create-a-status"}"   
  file: "prow/tide/status.go:393"   
  func: "k8s.io/test-infra/prow/tide.(*statusController).setStatuses.func1"   
  level: "error"   
  msg: "Failed to set status context from "PENDING" to "pending"."   
  org: "GoogleCloudPlatform"   
  pr: 1150   
  repo: "compute-image-tools"   
  sha: "1b3f5263ce6c6b6984dbc5539f157c837dabeae3"   
}

Since this is only happening on the compute-image-tools repo I suspect that the repo setting may have been accidentally changed to cause this. Perhaps the bot's permissions were demoted or removed?
cc @fejta @hopkiw

See #105

I think also plank is missing rbac rules... maybe other thigns.

Need to investigate further

Same problem as here: kubernetes/test-infra#14944

File for this instance: https://github.com/GoogleCloudPlatform/oss-test-infra/blob/master/prow/cluster/cert-manager.yaml

/kind oncall-hotlist
cc @Katharine

After running make -C prow deploy, the other pods updated, but gerrit is stuck in ContainerCreating mode. The new pod won't spin up, and the old pod won't terminate.

NAME                                  READY   STATUS              RESTARTS   AGE
pod/crier-75b645fbf9-lwcrv            1/1     Running             0          9m1s
pod/deck-795595cf75-skpfj             1/1     Running             0          9m2s
pod/deck-795595cf75-xlnv2             1/1     Running             0          9m2s
pod/gerrit-cf9cccc74-2xjr8            0/1     ContainerCreating   0          9m1s
pod/gerrit-dc4987ccb-r555v            1/1     Running             0          26h
pod/grandmatriarch-6c54cb845c-99fph   1/1     Running             0          8m59s
pod/hook-69548ff9bb-7p9kk             1/1     Running             0          9m
pod/hook-69548ff9bb-ddm9m             1/1     Running             0          9m1s
pod/horologium-558f7c4596-cdr6q       1/1     Running             0          9m2s
pod/plank-6d5f544979-m62qq            1/1     Running             0          9m3s
pod/sinker-7d44548f88-bd4jw           1/1     Running             0          9m3s
pod/tide-69ddd6c949-bw59j             1/1     Running             0          9m

/kind oncall-hotlist should work
/assign

/cc @fejta @BenTheElder

All of the Prow Jobs on the gflocks instance of Prow have been stuck in "triggered" state since Nov 7th. https://prow.gflocks.com/

Rerunning jobs seems to only get them also stuck in that state.

Can we get your utility set up on this repo to a) automatically create/update a testgrid.yaml based on the annotations on prowjobs in this repo and b) create kubernetes/test-infra PRs to get this added to testgrid.k8s.io?

/assign @chases2

/assign @hopkiw

ref #202

• Run enable-workload-identity.sh on the build cluster
• Run bind-service-accounts.sh for the service account pair
  • secretName: service-account
  • secretName: cloudesf-testing-github-prow-service-account
• Ensure job does not depend on the secret.json file existing, or the GOOGLE_APPLICATION_CREDENTIALS environment variable being set
• Update job to use a serviceAccountName, stop using the secret (example: #221)

https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity

I'm seeing

{
 insertId: "1fm21jlg7mcygnw"  
 jsonPayload: {
  component: "plank"   
  file: "prow/flagutil/github.go:107"   
  func: "k8s.io/test-infra/prow/flagutil.(*GitHubOptions).GitHubClientWithLogFields"   
  level: "warning"   
  msg: "empty -github-token-path, will use anonymous github client"   
 }
 labels: {…}  
 logName: "projects/oss-prow/logs/plank"  
 receiveTimestamp: "2019-12-05T22:09:39.247181734Z"  
 resource: {…}  
 severity: "ERROR"  
 timestamp: "2019-12-05T22:08:53Z"  
}

messages

See istio/test-infra#1976 for something similar

/kind oncall-hotlist

{
 insertId: "1l5t5s7fulrh9c"  
 jsonPayload: {
  component: "hook"   
  file: "prow/config/config.go:986"   
  func: "k8s.io/test-infra/prow/config.(*Config).finalizeJobConfig"   
  level: "warning"   
  msg: "default_decoration_config will be deprecated on April 2020, and it will be replaced with default_decoration_configs['*']."   
 }
 labels: {…}  
 logName: "projects/oss-prow/logs/hook"  
 receiveTimestamp: "2019-10-15T18:10:06.542949020Z"  
 resource: {…}  
 severity: "ERROR"  
 timestamp: "2019-10-15T18:10:05Z"  
}

/assign
/kind oncall-hotlist