andyzhangx / demo Goto Github PK

This is what I see in the controller manager logs, lots of these, where the azure storage controller tries to access a storage account in the wrong resource group:

I0425 13:47:20.952988       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"wekamedia", Name:"postgres-wekamedia-aurora-postgres", UID:"b4b1fc09-488e-11e8-8c3b-000d3ab7f8fa", APIVersion:"v1", ResourceVersion:"7511", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' Failed to provision volume with StorageClass "default": azureDisk - account ds6c822a4d484211eXXXXXX does not exist while trying to create/ensure default container
I0425 13:47:35.951122       1 pv_controller.go:1379] failed to provision volume for claim "somenamespace/aurora-postgres" with StorageClass "default": azureDisk - account ds6c822a4d484211eXXXXXX does not exist while trying to create/ensure default container

Feel free to just close this if I got you wrong. It does seem you know these issues very well. That using managed disks fixes this I was not aware of. I was even not aware you could switch those on.

References:

• Azure/acs-engine#2768
• kubernetes/kubernetes#46421

Could you explain how to use azure-secret.yaml (azure storage secret) in your process? I do not see how the azure-secret to be used? Thanks!

https://github.com/andyzhangx/demo/blob/master/issues/acr-issues.md#2-image-pull-error-from-acr-anonymous-repository said that image pull error for public ACR was fixed in v1.13.4. That looks not right -- https://github.com/kubernetes/kubernetes/blob/v1.13.4/pkg/credentialprovider/azure/azure_credentials.go doesn't include the fix. Instead, it is v1.13.5 including the fix: https://github.com/kubernetes/kubernetes/blob/v1.13.5/pkg/credentialprovider/azure/azure_credentials.go.

If we use the default storage classes, it creates a storage account in the shadow resource group(MC_*), this resource group get deleted when we recreate the AKS cluster.

What should be the steps to use an existing Storage Account with StatefulSet resource where ersistent Volume needs to be created dynamically using StatefulSet's volumeClaimTemplate. I have already created a storageAccount in the same RG as the AKS.

I tried to use secretkey and secretnamespae in storageClass, as mentioned in Kubernetes docs it ends up with error - invalid option "secretName" for volume plugin kubernetes.io/azure-file.

When I don't define the secret key in storageAccout resource, AKS tries to look for storage account in shadow resource group(MC_)

PoC spec file

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azurefile-standard-customtwo
provisioner: kubernetes.io/azure-file
reclaimPolicy: Retain
allowVolumeExpansion: True
parameters:
  skuName: Standard_LRS
  storageAccount: rahulpulsarpoc
  secretName: azuresecretname
---
apiVersion: v1
kind: Service
metadata:
  name: nginx2
  labels:
    app: nginx2
spec:
  ports:
    - port: 80
      name: web
  clusterIP: None
  selector:
    app: nginx2
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx2 # has to match .spec.template.metadata.labels
  serviceName: "nginx2"
  replicas: 2 # by default is 1
  template:
    metadata:
      labels:
        app: nginx2 # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: nginx
          image: k8s.gcr.io/nginx-slim:0.8
          ports:
            - containerPort: 80
              name: web
          volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: www
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: "azurefile-standard-customtwo"
        resources:
          requests:
            storage: 1Gi

I know there is posible to use Azure Files from AKS if you give permissions over the storage account to your sp or managed id.
But is it possible to create a StatefulSet if you are not allowed to give such access and only have storage key.

With normal deploys is possible since you only need to have the secret and to use secretName or even include shareName in order to have always the same name.
But when it comes to StatefulSet, wich uses volumeClaimTemplates, It seems imposible, unless yo have permissions over the storage as mentioned before.

Hi

In some of the Azure disks issues, you refer to using 1.12.4 as the stable version to solve the issue. Is this a typo? Because I think this a beta release. Shouldn't it be 1.12.3?

Thank you for preparing the examples and documentation. I believe I am following your steps correctly but I am getting the error "failed to find a matching storage account". My cluster was setup using aks.

RESOURCE_GROUP=staging
$ az storage account list -g $RESOURCE_GROUP  --query="[*].sku.name"
[
  "Standard_LRS"
]
$ az storage account list -g $RESOURCE_GROUP  --query="[*].location"
[
  "centralus"
]
$ az storage account list -g $RESOURCE_GROUP  --query="[*].name"
[
  "my_storage_account_name"
]
$ kubectl describe storageclass azurefile
Name:            azurefile
IsDefaultClass:  No
Annotations:     <none>
Provisioner:     kubernetes.io/azure-file
Parameters:      location=centralus,skuName=Standard_LRS,storageAccount=my_storage_account_name
Events:          <none>
$ cat pvc-azurefile.yaml
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-azurefile
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: azurefile
  resources:
    requests:
      storage: 5Gi
$ kubectl create -f pvc-azurefile.yaml
persistentvolumeclaim "pvc-azurefile" created
$ kubectl describe pvc pvc-azurefile
Name:          pvc-azurefile
Namespace:     default
StorageClass:  azurefile
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/azure-file
Capacity:
Access Modes:
Events:
  Type     Reason              Age              From                         Message
  ----     ------              ----             ----                         -------
  Warning  ProvisioningFailed  1m (x2 over 1m)  persistentvolume-controller  Failed to provision volume with StorageClass "azurefile": failed to find a matching storage account

Hi,

My projects involves azure fileshare integration.
It is working fine in AKS cluster. I would like to replicate the same in local environment for developers using docker-desktop with kubernetes enabled but fileshare is currently is not working because it is provisioned from azure.

I tried https://github.com/andyzhangx/demo/tree/master/linux/azurefile with static provisioning because I already have fileshare.
1.Created azure-secret with storage name and key.
2.Then edited nginx-pod-azurefile-static.yaml file with my sharename. This is throwing ERROR ".......bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount. helper program."

How would I link my local setup with azure fileshare for my pod deployment? I am feeling that I have to change something in mount options.

FYI: I am running docker-desktop on windows machine with WSL2 enabled

I have created a k8s cluster in Azure portal using Kubicorn. We used one VMSS for the master and a second VMSS for the nodes.

VMSS for the nodes is showing in failed status in Azure portal, with the following error message:

Resource /subscriptions/***/resourceGroups/***/providers/Microsoft.Network/loadBalancers/K8S-LOADBALANCER referenced by resource /subscriptions/***/resourceGroups/***/providers/Microsoft.Compute/virtualMachineScaleSets/k8sclustereastusdctest-node was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.

There is no with name K8S-LOADBALANCER in the RG (or in any RG that I use)

The instances are in status Failed (Running). But I cannot stop, upgrade the instance because of the error message above.

I never created a LB with the specific name and I cannot find a way to remove the reference from the VMSS.

In kubectl there are two LB that were created with Helm charts
`kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

lb-****-logging LoadBalancer 10.99.250.72 7171:30353/TCP,7272:32666/TCP,5140:31988/TCP 9d

****-loadbalancer LoadBalancer 10.104.217.232 80:31317/TCP 6d`

These two services show up under a LB with the name kuberetes.

Is there a way for me to remove the Reference from VMSS, so that it goes out from the failed status?