Comments (15)
awels
commented on August 11, 2024
1
Your kubernetes version is very old, I believe populators which CDI uses are beta in 1.24 and enabled by default there. In 1.22 you probably have to enable a feature gate to get it to work.
from containerized-data-importer.
shusley244
commented on August 11, 2024
Thanks for your answer. I will upgrade the kubernetes version and try again.
from containerized-data-importer.
knkedlaya
commented on August 11, 2024
Hi,
Facing similar issue though my kubernetes(AWS EKS) setup is latest.
virtctl version
Client Version: version.Info{GitVersion:"1.2.0", GitCommit:"$Format:%H$", GitTreeState:"", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.22.1", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{GitVersion:"v1.2.0", GitCommit:"f26e45d99ac35743fc33d6a121b629e9a9af6b63", GitTreeState:"clean", BuildDate:"2024-03-05T21:32:21Z", GoVersion:"go1.21.5 X:nocoverageredesign", Compiler:"gc", Platform:"linux/amd64"}
kubectl version
Client Version: v1.29.0-eks-5e0fdde
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.1-eks-b9c9ed7
pv.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
name: models-1-0-0
labels:
name: models-1-0-0
spec:
capacity:
storage: 10Gi
storageClassName: local-storage
claimRef:
namespace: default
name: models-1-0-0-claim
accessModes:
- ReadWriteMany
gcePersistentDisk:
pdName: models-1-0-0
fsType: ext4
readOnly: true
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: models-1-0-0-claim
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
selector:
matchLabels:
name: models-1-0-0
Trying to upload image with below command
virtctl image-upload \
--pvc-name=models-1-0-0-claim --no-create \
--image-path=/image/path/image-name-disk1.qcow2 \
--uploadproxy-url=https://127.0.0.1:18443 --insecure --force-bind
which ends up with
Flag --pvc-name has been deprecated, specify the name as the second argument instead.
Using existing PVC default/models-1-0-0-claim
Waiting for PVC models-1-0-0-claim upload pod to be ready...
timed out waiting for the condition
Output of
kubectl logs -f -n cdi cdi-deployment-79b586b894-hxcnh
...
...
{"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.RouteReconcile","msg":"No route found, setting to blank","RouteURL":""}
{"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.StorageClassReconcile","msg":"Setting scratch space to default","storageClass.Name":"gp2"}
{"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller","msg":"reconciling CDIConfig","CDIConfig":"/config"}
{"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.IngressReconcile","msg":"No ingress found, setting to blank","IngressURL":""}
{"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.RouteReconcile","msg":"No route found, setting to blank","RouteURL":""}
{"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.StorageClassReconcile","msg":"Setting scratch space to default","storageClass.Name":"gp2"}
{"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/gp2"}
{"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
{"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
{"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"Set metric:gp2 complete:true default:true vdefault:false rwx:false smartclone:false (deleted 1)"}
{"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
{"level":"debug","ts":"2024-03-26T04:38:38Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"info","ts":"2024-03-26T04:38:38Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"info","ts":"2024-03-26T08:32:13Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
from containerized-data-importer.
awels
commented on August 11, 2024
Hi,
Facing similar issue though my kubernetes(AWS EKS) setup is latest.
virtctl version Client Version: version.Info{GitVersion:"1.2.0", GitCommit:"$Format:%H$", GitTreeState:"", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.22.1", Compiler:"gc", Platform:"darwin/arm64"} Server Version: version.Info{GitVersion:"v1.2.0", GitCommit:"f26e45d99ac35743fc33d6a121b629e9a9af6b63", GitTreeState:"clean", BuildDate:"2024-03-05T21:32:21Z", GoVersion:"go1.21.5 X:nocoverageredesign", Compiler:"gc", Platform:"linux/amd64"} kubectl version Client Version: v1.29.0-eks-5e0fdde Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.1-eks-b9c9ed7pv.yaml
kind: PersistentVolume apiVersion: v1 metadata: name: models-1-0-0 labels: name: models-1-0-0 spec: capacity: storage: 10Gi storageClassName: local-storage claimRef: namespace: default name: models-1-0-0-claim accessModes: - ReadWriteMany gcePersistentDisk: pdName: models-1-0-0 fsType: ext4 readOnly: true --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: models-1-0-0-claim spec: accessModes: - ReadWriteMany resources: requests: storage: 5Gi selector: matchLabels: name: models-1-0-0Trying to upload image with below command
virtctl image-upload \ --pvc-name=models-1-0-0-claim --no-create \ --image-path=/image/path/image-name-disk1.qcow2 \ --uploadproxy-url=https://127.0.0.1:18443 --insecure --force-bindwhich ends up with
Flag --pvc-name has been deprecated, specify the name as the second argument instead. Using existing PVC default/models-1-0-0-claim Waiting for PVC models-1-0-0-claim upload pod to be ready... timed out waiting for the conditionOutput of
kubectl logs -f -n cdi cdi-deployment-79b586b894-hxcnh ... ... {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.RouteReconcile","msg":"No route found, setting to blank","RouteURL":""} {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.StorageClassReconcile","msg":"Setting scratch space to default","storageClass.Name":"gp2"} {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller","msg":"reconciling CDIConfig","CDIConfig":"/config"} {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.IngressReconcile","msg":"No ingress found, setting to blank","IngressURL":""} {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.RouteReconcile","msg":"No route found, setting to blank","RouteURL":""} {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.StorageClassReconcile","msg":"Setting scratch space to default","storageClass.Name":"gp2"} {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/gp2"} {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"} {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}} {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"Set metric:gp2 complete:true default:true vdefault:false rwx:false smartclone:false (deleted 1)"} {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"} {"level":"debug","ts":"2024-03-26T04:38:38Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"} {"level":"info","ts":"2024-03-26T04:38:38Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"} {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"} {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"} {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"} {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"} {"level":"info","ts":"2024-03-26T08:32:13Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"} {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"} {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"} {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false} {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"} {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"} {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
Looks like you are using the local-storage storage class AND RWX volumes. I am fairly certain local-storage doesn't support RWX, try RWO.
from containerized-data-importer.
knkedlaya
commented on August 11, 2024
Thanks @awels . I changed local-storage to ReadWriteOnce and I see different logs
This is new pv.yaml after changing to RWO
kind: PersistentVolume
apiVersion: v1
metadata:
name: models-1-0-0
labels:
name: models-1-0-0
spec:
capacity:
storage: 10Gi
storageClassName: local-storage
claimRef:
namespace: default
name: models-1-0-0-claim
accessModes:
- ReadWriteOnce
gcePersistentDisk:
pdName: models-1-0-0
fsType: ext4
readOnly: true
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: models-1-0-0-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
selector:
matchLabels:
name: models-1-0-0
{"level":"info","ts":"2024-03-26T13:17:39Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
{"level":"info","ts":"2024-03-26T13:17:39Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
{"level":"info","ts":"2024-03-26T13:17:40Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim","isUpload":true,"isCloneTarget":false,"isBound":true,"podSucceededFromPVC":false,"deletionTimeStamp set?":true}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"info","ts":"2024-03-26T13:17:40Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
{"level":"info","ts":"2024-03-26T13:17:40Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"info","ts":"2024-03-26T13:17:40Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim-scratch"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim-scratch"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim-scratch","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim-scratch"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim-scratch","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":true}
{"level":"info","ts":"2024-03-26T13:17:40Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
{"level":"info","ts":"2024-03-26T13:17:40Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim-scratch"}
{"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim-scratch"}
{"level":"info","ts":"2024-03-26T13:17:40Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
{"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
{"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
{"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
{"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
{"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
{"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim","Phase":"Pending"}
{"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
{"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
{"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
{"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
{"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
{"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim","isUpload":false,"isCloneTarget":false,"isBound":true,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"info","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Updating PVC from pod","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Phase is now","pvc.anno.Phase":""}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Updated PVC","PVC":"default/models-1-0-0-claim","pvc.anno.AnnImportPod":"cdi-upload-models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"info","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"}
I0326 13:18:15.553328 1 util.go:2109] Applying PVC annotationNamecdi-upload-models-1-0-0-claimsidecar.istio.io/injectfalse
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"upload pod created\n","Namespace":"default","Name":"cdi-upload-models-1-0-0-claim","Image name":"quay.io/kubevirt/cdi-uploadserver:v1.58.3"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim-scratch"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim-scratch"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim-scratch","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim-scratch"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim-scratch","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"upload service created\n","Namespace":"default","Name":"cdi-upload-models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Phase is now","pvc.anno.Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"info","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Phase is now","pvc.anno.Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"info","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim-scratch"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim-scratch"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim-scratch"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim-scratch","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim-scratch","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim-scratch"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim-scratch","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim-scratch"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim-scratch"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim-scratch","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
from containerized-data-importer.
awels
commented on August 11, 2024
And the upload pod is still not starting? Do you have any pods in the namespace you are trying to upload to?
from containerized-data-importer.
awels
commented on August 11, 2024
Also I see no mention of the scratch space pvc. Does that get created? It also seems you are creating the PVC ahead of time, you don't really need to do that, if you specify the datavolume name, CDI should take care of creating all the required resources.
from containerized-data-importer.
knkedlaya
commented on August 11, 2024
You are right, I have a pod running. Never came to my mind that the pod is related to the image I was trying to upload. I will delete this pod, PV, PVC and retry with dv create option while uploading.
I think upload creates scratch pvc because I havent created it explicitly.
kubectl get po
NAME READY STATUS RESTARTS AGE
cdi-upload-models-1-0-0-claim 0/1 Pending 0 25m
from containerized-data-importer.
awels
commented on August 11, 2024
it doesn't look running, it looks pending, if it remains pending after retrying, do a kubectl describe pod <name> it should tell you why it is pending.
from containerized-data-importer.
knkedlaya
commented on August 11, 2024
These are the events I am seeing when I describe the pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 33m default-scheduler 0/2 nodes are available: persistentvolumeclaim "models-1-0-0-claim-scratch" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
Warning FailedScheduling 2m58s (x3 over 23m) default-scheduler running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded
from containerized-data-importer.
awels
commented on August 11, 2024
Is the scratch PVC created, and is it bound? The events tell me it can't mount the scratch space.
from containerized-data-importer.
knkedlaya
commented on August 11, 2024
yes, scratch pvc is created. When I describe I see following. Hinting at some permission related issue.
status code: 403, request id: e58dde0e-03e9-4167-9b16-dd18cfe80ca2
Warning ProvisioningFailed 18m (x6 over 38m) ebs.csi.aws.com_ebs-csi-controller-7db7cf79d5-5sq47_d48c1c09-e066-4697-aa55-c1064e3b5068 (combined from similar events): failed to provision volume with StorageClass "gp2": rpc error: code = Internal desc = Could not create volume "pvc-855bd68a-f81c-4e06-9406-e8dfb914459f": could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::<Accound-ID-masked>:assumed-role/eksctl-kubevirt-cluster-1-nodegrou-NodeInstanceRole-2dTUBsU8mLgv/i-023afa1afc95aedde is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:ap-south-1:<Accound-ID-masked>:volume/* because no identity-based policy allows the ec2:CreateVolume action. Encoded authorization failure message: 3PwEe0JdTGHWwrlf1c6ClLMyjwtHtCIpFNlKRq_uinU1OrKB4itwGSUh5eSS786CVBAhBK1tm38KGkKunGL1HM9sA5shKk-LcDm2nKI0xZMUmbuscSdYgwBBvdxpC-xpJpqUdKQ8WUDXCTdKjQGIoRdqmG-TyzBGgiPaNuuMr83dS1td17JU4dbWxBUgJppCL4Iio4J7zLMlutTeXE-H5iKQJK6Rk361E-FXnZBurJhDj5rmYioYFeoeWqRzBAFbJUB2Awd0_-J770LUnnSyARD-Ho56A4KWonOKaXx5KoOVG3gk41tGL1ggFt62V2syzKL204yzlpR4v4vFT5Ulk1PeJXvpNoQIUKzSR6m2IAQ-FtuUEuGPRoGxMsHWlRsteHsdNGmIThrxZ3YCPsXvIElwOv55cUk26z-Rhrr-SOWYKUy3a_7pRRsUXQeYyim3Y0s7rFVHNpzQeyeASR6Gq7-Lg5Rfs45kLHv52KaYbPNSSxm7VUp5aQQUY59BVQdSQxGBGx_dDGbawUe4NT4ZvfYOmASbj6ejJonodDdxyMbLOsRrSHkWH4MOcB7hJbWjW5lWSztKxV-d7kOhiMFP9YWlohagwyciMp6Ffw3cGGJkG9yiWffif81ps_4_CdjittFF4VdgoCdLrkl0wxs5oK9CWhq0p9E8
status code: 403, request id: 15649ff3-480e-4e28-882a-b84adac025ca
Normal ExternalProvisioning 101s (x182 over 46m) persistentvolume-controller Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
Normal Provisioning 86s (x20 over 46m) ebs.csi.aws.com_ebs-csi-controller-7db7cf79d5-5sq47_d48c1c09-e066-4697-aa55-c1064e3b5068 External provisioner is provisioning volume for claim "default/models-1-0-0-claim-scratch"
from containerized-data-importer.
awels
commented on August 11, 2024
Yeah, seems like it. That is your current problem.
from containerized-data-importer.
knkedlaya
commented on August 11, 2024
Created a new cluster, now the pod is in ContainerCreating state with the following event on the pod. Not sure if some config is gone missing/wrong.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 33m default-scheduler 0/1 nodes are available: persistentvolumeclaim "sipvc-dv-scratch" not found. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Normal Scheduled 27m default-scheduler Successfully assigned default/cdi-upload-sipvc-dv to ip-192-168-82-77.ap-south-1.compute.internal
Warning FailedAttachVolume 3m40s (x10 over 25m) attachdetach-controller AttachVolume.Attach failed for volume "sipvc-dv" : timed out waiting for external-attacher of pd.csi.storage.gke.io CSI driver to attach volume projects/UNSPECIFIED/zones/UNSPECIFIED/disks/models-1-0-0
Note that, I had to create scratch pv to bind with scratch pvc. After that, the upload container moved from pending to ContainerCreating state.
from containerized-data-importer.
awels
commented on August 11, 2024
Seems like you still have some storage issues. If the attach is failing it means the volume cannot be mounted on the node the pod is supposed to run on. Looking at the message you have some UNSPECIFIED fields somewhere.
from containerized-data-importer.
Related Issues (20)
- Slow import HOT 3
- Prlimit Unit Test fail on s390x with go1.21.5 HOT 4
- Missing clear explanation on how to pull an image from S3 HOT 6
- Parameterize hard coded environment variables in bazel-build-builder.sh
- Datavolume is failed occasionally when clone a datavolume from a pvc for concurrency HOT 12
- CDI importer pod shows nbdkit error , not able to create data-volume HOT 12
- DataVolume import - volume node affinity conflict HOT 2
- Add s390x cross-build libraries to builder container docker file HOT 4
- Add troubleshooting doc
- DataVolumes.md HOT 1
- Labels derived from env vars on containerdisks are not set on imported PVC when using populators HOT 1
- Out of date?
- 为啥指定的镜像源，比如qcow2格式的镜像文件，会默认转为img格式，能保持原格式不变，供kubevirt使用吗 HOT 2
- Unable to build docker image from source? HOT 1
- Include functional test focus capabilities
- Prlimit Unit Test fails on s390x HOT 12
- s390x enablement
- Error when requesting an UploadTokenRequest
- Not getting images from mirror-registry
- cdi Label matching bound pv does not take effect HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from containerized-data-importer.