Comments (3)
danilo404
commented on July 26, 2024
1
Hi @mboersma, thank you so much for looking into this. Your assumption is correct, the ipConfigurations field is always present in our case. The virtualnetworkssubnets.network.azure.com object is now smaller, and it successfully reconciled, but we can still see many items in ipConfigurations, I believe it will break again as soon as the number of nodes scale up.
Our CR looks almost exactly like yours, except for the ipConfigurations field and we also have some serviceEndpoints, here are more complete versions:
kubectl get virtualnetworkssubnets.network.azure.com example-cluster-vnet-example-cluster-subnet -o yaml
---
apiVersion: network.azure.com/v1api20201101
kind: VirtualNetworksSubnet
metadata:
creationTimestamp: "2024-02-01T16:32:39Z"
finalizers:
- serviceoperator.azure.com/finalizer
generation: 1
labels:
serviceoperator.azure.com/owner-group-kind: VirtualNetwork.network.azure.com
serviceoperator.azure.com/owner-name: example-cluster-vnet
sigs.k8s.io_cluster-api-provider-azure_owned: example-cluster-aks
name: example-cluster-vnet-example-cluster-subnet
ownerReferences:
- apiVersion: network.azure.com/v1api20201101storage
kind: VirtualNetwork
name: example-cluster-vnet
uid: ...
resourceVersion: "1193221614"
uid: ...
spec:
addressPrefix: 10.0.0.0/16
addressPrefixes:
- 10.0.0.0/16
azureName: example-cluster-subnet
owner:
name: example-cluster-vnet
serviceEndpoints:
- locations:
- '*'
service: Microsoft.Sql
- locations:
- '*'
service: Microsoft.KeyVault
- locations:
- '*'
service: Microsoft.Storage
- locations:
- '*'
service: Microsoft.AzureCosmosDB
- locations:
- '*'
service: Microsoft.ServiceBus
- locations:
- '*'
service: Microsoft.EventHub
status:
addressPrefix: 10.0.0.0/16
conditions:
- lastTransitionTime: "2024-03-19T17:39:06Z"
observedGeneration: 1
reason: Succeeded
status: "True"
type: Ready
etag: ...
id: ....
ipConfigurations:
- id: /subscriptions/<subscription id>/resourceGroups/<resource group name>/PROVIDERS/MICROSOFT.COMPUTE/VIRTUALMACHINESCALESETS/AKS-AIMODELSCPU-17351824-VMSS/VIRTUALMACHINES/0/NETWORKINTERFACES/AKS-AIMODELSCPU-17351824-VMSS/ipConfigurations/IPCONFIG1
[... thousands of similar entries ...]
name: example-cluster-subnet
privateEndpointNetworkPolicies: Disabled
privateLinkServiceNetworkPolicies: Enabled
provisioningState: Succeeded
serviceEndpoints:
- locations:
- northeurope
provisioningState: Succeeded
service: Microsoft.Sql
- locations:
- '*'
provisioningState: Succeeded
service: Microsoft.KeyVault
- locations:
- northeurope
- westeurope
provisioningState: Succeeded
service: Microsoft.Storage
- locations:
- '*'
provisioningState: Succeeded
service: Microsoft.AzureCosmosDB
- locations:
- '*'
provisioningState: Succeeded
service: Microsoft.ServiceBus
- locations:
- '*'
provisioningState: Succeeded
service: Microsoft.EventHub
type: [Microsoft.Network/virtualNetworks/subnets](http://microsoft.network/virtualNetworks/subnets)
kubectl get amcp example-cluster -o yaml
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureManagedControlPlane
metadata:
annotations:
clusterctl.cluster.x-k8s.io/block-move: "true"
creationTimestamp: "2023-12-06T13:58:26Z"
finalizers:
- azuremanagedcontrolplane.infrastructure.cluster.x-k8s.io
generation: 6
name: example-cluster-aks
ownerReferences:
- apiVersion: cluster.x-k8s.io/v1beta1
blockOwnerDeletion: true
controller: true
kind: Cluster
name: example-cluster-aks
resourceVersion: "1193254100"
spec:
aadProfile:
adminGroupObjectIDs:
- ...
managed: true
addonProfiles:
- config:
logAnalyticsWorkspaceResourceID: ...
enabled: true
name: omsagent
apiServerAccessProfile:
authorizedIPRanges:
- [46 entries]
controlPlaneEndpoint:
host: ...
port: 443
dnsPrefix: example-cluster-aks
identity:
type: SystemAssigned
identityRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureClusterIdentity
name: example-cluster-aks-identity
kubeletUserAssignedIdentity: ...
loadBalancerSKU: Standard
location: northeurope
networkPlugin: azure
networkPolicy: azure
nodeResourceGroupName: MC_example-cluster-rg_example-cluster-aks_northeurope
oidcIssuerProfile:
enabled: true
resourceGroupName: example-cluster-rg
sku:
tier: Free
sshPublicKey: ....
version: v1.27.9
virtualNetwork:
cidrBlock: 10.0.0.0/16
name: example-cluster-vnet
resourceGroup: example-cluster-rg
subnet:
cidrBlock: 10.0.0.0/16
name: example-cluster-subnet
serviceEndpoints:
- locations:
- '*'
service: Microsoft.Sql
- locations:
- '*'
service: Microsoft.KeyVault
- locations:
- '*'
service: Microsoft.Storage
- locations:
- '*'
service: Microsoft.AzureCosmosDB
- locations:
- '*'
service: Microsoft.ServiceBus
- locations:
- '*'
service: Microsoft.EventHub
status:
conditions:
- lastTransitionTime: "2024-03-19T17:43:15Z"
status: "True"
type: Ready
- lastTransitionTime: "2024-03-19T17:43:15Z"
status: "True"
type: ManagedClusterRunning
- lastTransitionTime: "2024-03-14T14:23:51Z"
status: "True"
type: ResourceGroupReady
- lastTransitionTime: "2024-03-19T17:39:30Z"
status: "True"
type: SubnetsReady
- lastTransitionTime: "2024-03-14T14:24:04Z"
status: "True"
type: VNetReady
initialized: true
oidcIssuerProfile:
issuerURL: ...
ready: true
version: v1.27.9
from cluster-api-provider-azure.
mboersma
commented on July 26, 2024
@danilo404 I'm investigating this but haven't reproduced it so far.
I started with CAPZ v1.13.0 and created an AKS cluster with similar network config to what you posted, and 10 node pools with 1000 nginx pods. But I'm not seeing any ipConfigurations on the VirtualNetworksSubnet:
% kubectl get amcp -o yaml
apiVersion: v1
items:
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureManagedControlPlane
metadata:
name: aks-etcd-1234
spec:
networkPlugin: azure
networkPolicy: azure
version: v1.28.3
virtualNetwork:
cidrBlock: 10.0.0.0/16
name: aks-etcd-1234-vnet
resourceGroup: aks-etcd-1234
subnet:
cidrBlock: 10.0.0.0/16
name: aks-etcd-1234
% kubectl get VirtualNetworksSubnet -o yaml
apiVersion: v1
items:
- apiVersion: network.azure.com/v1api20201101
kind: VirtualNetworksSubnet
spec:
addressPrefix: 10.0.0.0/16
addressPrefixes:
- 10.0.0.0/16
azureName: aks-etcd-1234
owner:
name: aks-etcd-1234-vnet
status:
addressPrefix: 10.0.0.0/16
conditions:
id: /subscriptions/00000000-0000-0000-0000-00000000000/resourceGroups/aks-etcd-1234/providers/Microsoft.Network/virtualNetworks/aks-etcd-1234-vnet/subnets/aks-etcd-1234
name: aks-etcd-1234
privateEndpointNetworkPolicies: Disabled
privateLinkServiceNetworkPolicies: Enabled
provisioningState: Succeeded
type: Microsoft.Network/virtualNetworks/subnets
kind: ListSorry, I must be missing something. Any ideas on what other configuration might be relevant? I assume I wouldn't need to scale up as far as you have just to see the way ipConfigurations is being used.
from cluster-api-provider-azure.
mboersma
commented on July 26, 2024
@danilo404 I'm sorry I haven't added an update here. I tried again but haven't been able to light up the ipConfigurations. Have you found a workaround or do you have any other details I might be missing around the network configuration?
from cluster-api-provider-azure.
Related Issues (20)
- Add AzureMachinePoolTemplate CRD to support ClusterClass with MachinePools
- BYO VNet Deletion Issues: Route Tables & NSGs
- VMSS VM is in Deleting state, Owner Machine of AzureMachinePoolMachine is not deleted
- Downscaling with cluster-autoscaler (provider clusterapi) scales down wrong MachinePoolMachines
- Add westeurope to e2e test regions
- Hi,
- Management Cluster initiation failed while using User managed Identity for Cluster Provision into our Azure subscription HOT 3
- Deleting a MachinePool can result in stuck AzureMachinePoolMachine due to recreation
- Update ASO dep to 2.8.0 HOT 1
- VMSS reaches limit number of models (10) and can't be scaled anymore HOT 1
- Support AKS uninstall of NetworkPolicyManager as part of upgrading to CNI Overlay HOT 4
- Better UX to indicate the reason of WaitingForInfrastructure when using MachinePool/AzureMachinePool
- AMCP needs explicit IP CIDR block when referencing existing VNETS HOT 6
- Migrate Upstream tests to community infrastructure HOT 1
- MachinePool ready state leading to not processing providerIDs in CAPI
- Better error message for not setting identity field for UserAssignedIdentity
- CAPI v1.8.0-beta.0 has been released and is ready for testing
- Generate API version upgrade test templates with kustomize instead of hard-coding them
- CAPZ doesn't invalidate Azure client cache when only auth mechanism changes
- community infra migration: subscription and regional issues
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cluster-api-provider-azure.