Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KubernetesCluster/platform-ref-azure-cluster-aks composed resource status is flipping #122

Open
ytsarev opened this issue Oct 29, 2024 · 0 comments
Open

KubernetesCluster/platform-ref-azure-cluster-aks composed resource status is flipping #122

ytsarev opened this issue Oct 29, 2024 · 0 comments
Labels
bug Something isn't working needs-epic-link Needs a link to an epic needs-points-label Needs a story point label needs-project Needs a project

Comments

@ytsarev
Copy link
Member

ytsarev commented Oct 29, 2024

What happened?

Detected with manual setup ( uptest is green as it proceeds on the first successful result)

rossplane beta trace cluster.azure.platformref.upbound.io/platform-ref-azure-cluster
NAME                                                            SYNCED   READY   STATUS
Cluster/platform-ref-azure-cluster (default)                    True     True    Available
└─ XCluster/platform-ref-azure-cluster-f7dkm                    True     True    Available
   ├─ Usage/platform-ref-azure-cluster-f7dkm-hplr9              -        True    Available
   ├─ Usage/platform-ref-azure-cluster-f7dkm-lmfrz              -        True    Available
   ├─ Usage/platform-ref-azure-cluster-f7dkm-wbmgq              -        True    Available
   ├─ XAKS/platform-ref-azure-cluster-f7dkm-n9ht8               True     True    Available
   │  ├─ KubernetesCluster/platform-ref-azure-cluster-aks       False    True    ReconcileError: ...bject": spec.forProvider.defaultNodePool is a required parameter
   │  ├─ ProviderConfig/platform-ref-azure-cluster              -        -
   │  ├─ Object/platform-ref-azure-cluster-f7dkm-f9v4h          True     True    Available
   │  └─ ProviderConfig/platform-ref-azure-cluster              -        -
   ├─ XNetwork/platform-ref-azure-cluster-f7dkm-9t4qm           True     True    Available
   │  ├─ ResourceGroup/platform-ref-azure-cluster-f7dkm-glbsd   True     True    Available
   │  ├─ Subnet/platform-ref-azure-cluster-sn                   True     True    Available
   │  └─ VirtualNetwork/platform-ref-azure-cluster-vnet         True     True    Available
   ├─ XFlux/platform-ref-azure-cluster-f7dkm-f5pkg              True     True    Available
   │  ├─ Release/platform-ref-azure-cluster-f7dkm-296hc         True     True    Available
   │  └─ Release/platform-ref-azure-cluster-f7dkm-cqzz8         True     True    Available
   └─ XOss/platform-ref-azure-cluster-f7dkm-cjq49               True     True    Available
      ├─ Usage/platform-ref-azure-cluster-f7dkm-849qw           -        True    Available
      ├─ Usage/platform-ref-azure-cluster-f7dkm-j62n6           -        True    Available
      ├─ Usage/platform-ref-azure-cluster-f7dkm-k2x2p           -        True    Available
      ├─ Release/platform-ref-azure-cluster                     True     True    Available
      ├─ Object/platform-ref-azure-cluster-f7dkm-42m2l          True     True    Available
      ├─ Object/platform-ref-azure-cluster-f7dkm-g6xvz          True     True    Available
      └─ Object/platform-ref-azure-cluster-f7dkm-g8sgh          True     True    Available

k get KubernetesCluster/platform-ref-azure-cluster-aks -o yaml
apiVersion: containerservice.azure.upbound.io/v1beta2
kind: KubernetesCluster
metadata:
  annotations:
    crossplane.io/composition-resource-name: kubernetesCluster
    crossplane.io/external-create-pending: "2024-10-29T22:15:51Z"
    crossplane.io/external-create-succeeded: "2024-10-29T22:15:51Z"
    crossplane.io/external-name: platform-ref-azure-cluster-aks
  creationTimestamp: "2024-10-29T22:13:45Z"
  finalizers:
  - finalizer.managedresource.crossplane.io
  generateName: platform-ref-azure-cluster-f7dkm-
  generation: 75
  labels:
    crossplane.io/claim-name: platform-ref-azure-cluster
    crossplane.io/claim-namespace: default
    crossplane.io/composite: platform-ref-azure-cluster-f7dkm
  name: platform-ref-azure-cluster-aks
  ownerReferences:
  - apiVersion: azure.platform.upbound.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: XAKS
    name: platform-ref-azure-cluster-f7dkm-n9ht8
    uid: e43d9f97-7e5c-4b52-879d-c4bf8fda7117
  resourceVersion: "8681"
  uid: e0eb938d-80a2-4025-9b82-44397b0f300c
spec:
  deletionPolicy: Delete
  forProvider:
    azurePolicyEnabled: true
    defaultNodePool:
      name: default
      nodeCount: 1
      vmSize: Standard_B2s
      vnetSubnetId: /subscriptions/038f2b7c-3265-43b8-8624-c9ad5da610a8/resourceGroups/platform-ref-azure-cluster-f7dkm-glbsd/providers/Microsoft.Network/virtualNetworks/platform-ref-azure-cluster-vnet/subnets/platform-ref-azure-cluster-sn
      vnetSubnetIdRef:
        name: platform-ref-azure-cluster-sn
      vnetSubnetIdSelector:
        matchLabels:
          azure.platform.upbound.io/network-id: platform-ref-azure-cluster
    dnsPrefix: platform-ref-azure-cluster
    identity:
      type: SystemAssigned
    imageCleanerIntervalHours: 48
    kubernetesVersion: "1.28"
    location: westus
    networkProfile:
      dnsServiceIp: 10.0.0.10
      ipVersions:
      - IPv4
      loadBalancerProfile:
        managedOutboundIpCount: 1
      loadBalancerSku: standard
      networkPlugin: kubenet
      outboundType: loadBalancer
      podCidr: 10.244.0.0/16
      podCidrs:
      - 10.244.0.0/16
      serviceCidr: 10.0.0.0/16
      serviceCidrs:
      - 10.0.0.0/16
    nodeResourceGroup: MC_platform-ref-azure-cluster-f7dkm-glbsd_platform-ref-azure-cluster-aks_westus
    oidcIssuerEnabled: true
    publicNetworkAccessEnabled: true
    resourceGroupNameSelector:
      matchLabels:
        azure.platform.upbound.io/network-id: platform-ref-azure-cluster
    roleBasedAccessControlEnabled: true
    runCommandEnabled: true
    skuTier: Free
    supportPlan: KubernetesOfficial
    workloadIdentityEnabled: true
  initProvider: {}
  managementPolicies:
  - '*'
  providerConfigRef:
    name: default
  writeConnectionSecretToRef:
    name: e43d9f97-7e5c-4b52-879d-c4bf8fda7117-akscluster
    namespace: upbound-system
status:
  atProvider:
    automaticChannelUpgrade: ""
    azurePolicyEnabled: true
    currentKubernetesVersion: 1.28.14
    defaultNodePool:
      capacityReservationGroupId: ""
      customCaTrustEnabled: false
      enableAutoScaling: false
      enableHostEncryption: false
      enableNodePublicIp: false
      fipsEnabled: false
      gpuInstance: ""
      hostGroupId: ""
      kubeletDiskType: OS
      maxCount: 0
      maxPods: 110
      messageOfTheDay: ""
      minCount: 0
      name: default
      nodeCount: 1
      nodePublicIpPrefixId: ""
      onlyCriticalAddonsEnabled: false
      orchestratorVersion: 1.28.14
      osDiskSizeGb: 128
      osDiskType: Managed
      osSku: Ubuntu
      podSubnetId: ""
      proximityPlacementGroupId: ""
      scaleDownMode: Delete
      snapshotId: ""
      temporaryNameForRotation: ""
      type: VirtualMachineScaleSets
      ultraSsdEnabled: false
      upgradeSettings:
        maxSurge: 10%
      vmSize: Standard_B2s
      vnetSubnetId: /subscriptions/038f2b7c-3265-43b8-8624-c9ad5da610a8/resourceGroups/platform-ref-azure-cluster-f7dkm-glbsd/providers/Microsoft.Network/virtualNetworks/platform-ref-azure-cluster-vnet/subnets/platform-ref-azure-cluster-sn
      workloadRuntime: ""
    diskEncryptionSetId: ""
    dnsPrefix: platform-ref-azure-cluster
    dnsPrefixPrivateCluster: ""
    edgeZone: ""
    enablePodSecurityPolicy: false
    fqdn: platform-ref-azure-cluster-xg4uujij.hcp.westus.azmk8s.io
    httpApplicationRoutingEnabled: false
    httpApplicationRoutingZoneName: ""
    id: /subscriptions/038f2b7c-3265-43b8-8624-c9ad5da610a8/resourceGroups/platform-ref-azure-cluster-f7dkm-glbsd/providers/Microsoft.ContainerService/managedClusters/platform-ref-azure-cluster-aks
    identity:
      principalId: 20a48487-4de0-48f7-92bb-9bf250e2a6d6
      tenantId: b9925bc4-8383-4c37-b9d2-fa456d1bb1c7
      type: SystemAssigned
    imageCleanerEnabled: false
    imageCleanerIntervalHours: 48
    kubeletIdentity:
      clientId: adf88fac-5395-4f19-9b38-0a84293f0e5c
      objectId: 47ce8fa5-75c4-4819-8569-a6658dc511f5
      userAssignedIdentityId: /subscriptions/038f2b7c-3265-43b8-8624-c9ad5da610a8/resourceGroups/MC_platform-ref-azure-cluster-f7dkm-glbsd_platform-ref-azure-cluster-aks_westus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/platform-ref-azure-cluster-aks-agentpool
    kubernetesVersion: "1.28"
    localAccountDisabled: false
    location: westus
    networkProfile:
      dnsServiceIp: 10.0.0.10
      dockerBridgeCidr: ""
      ebpfDataPlane: ""
      ipVersions:
      - IPv4
      loadBalancerProfile:
        effectiveOutboundIps:
        - /subscriptions/038f2b7c-3265-43b8-8624-c9ad5da610a8/resourceGroups/MC_platform-ref-azure-cluster-f7dkm-glbsd_platform-ref-azure-cluster-aks_westus/providers/Microsoft.Network/publicIPAddresses/dcdab027-4e90-48ea-8d3b-c48089b24d1e
        idleTimeoutInMinutes: 30
        managedOutboundIpCount: 1
        managedOutboundIpv6Count: 0
        outboundPortsAllocated: 0
      loadBalancerSku: standard
      networkMode: ""
      networkPlugin: kubenet
      networkPluginMode: ""
      networkPolicy: ""
      outboundType: loadBalancer
      podCidr: 10.244.0.0/16
      podCidrs:
      - 10.244.0.0/16
      serviceCidr: 10.0.0.0/16
      serviceCidrs:
      - 10.0.0.0/16
    nodeResourceGroup: MC_platform-ref-azure-cluster-f7dkm-glbsd_platform-ref-azure-cluster-aks_westus
    nodeResourceGroupId: /subscriptions/038f2b7c-3265-43b8-8624-c9ad5da610a8/resourceGroups/MC_platform-ref-azure-cluster-f7dkm-glbsd_platform-ref-azure-cluster-aks_westus
    oidcIssuerEnabled: true
    oidcIssuerUrl: https://westus.oic.prod-aks.azure.com/b9925bc4-8383-4c37-b9d2-fa456d1bb1c7/6068185d-d46a-4c33-ab4d-2c34d6f40bcb/
    openServiceMeshEnabled: false
    portalFqdn: platform-ref-azure-cluster-xg4uujij.portal.hcp.westus.azmk8s.io
    privateClusterEnabled: false
    privateClusterPublicFqdnEnabled: false
    privateFqdn: ""
    publicNetworkAccessEnabled: true
    resourceGroupName: platform-ref-azure-cluster-f7dkm-glbsd
    roleBasedAccessControlEnabled: true
    runCommandEnabled: true
    skuTier: Free
    supportPlan: KubernetesOfficial
    workloadIdentityEnabled: true
  conditions:
  - lastTransitionTime: "2024-10-29T22:43:10Z"
    message: 'cannot patch the managed resource via server-side apply: KubernetesCluster.containerservice.azure.upbound.io
      "platform-ref-azure-cluster-aks" is invalid: spec: Invalid value: "object":
      spec.forProvider.defaultNodePool is a required parameter'
    reason: ReconcileError
    status: "False"
    type: Synced
  - lastTransitionTime: "2024-10-29T22:23:39Z"
    reason: Available
    status: "True"
    type: Ready
  - lastTransitionTime: "2024-10-29T22:31:24Z"
    reason: Success
    status: "True"
    type: LastAsyncOperation

The problematic state is periodically appearing and going away

crossplane beta trace cluster.azure.platformref.upbound.io/platform-ref-azure-cluster
NAME                                                            SYNCED   READY   STATUS
Cluster/platform-ref-azure-cluster (default)                    True     True    Available
└─ XCluster/platform-ref-azure-cluster-f7dkm                    True     True    Available
   ├─ Usage/platform-ref-azure-cluster-f7dkm-hplr9              -        True    Available
   ├─ Usage/platform-ref-azure-cluster-f7dkm-lmfrz              -        True    Available
   ├─ Usage/platform-ref-azure-cluster-f7dkm-wbmgq              -        True    Available
   ├─ XAKS/platform-ref-azure-cluster-f7dkm-n9ht8               True     True    Available
   │  ├─ KubernetesCluster/platform-ref-azure-cluster-aks       True     True    Available
   │  ├─ ProviderConfig/platform-ref-azure-cluster              -        -
   │  ├─ Object/platform-ref-azure-cluster-f7dkm-f9v4h          True     True    Available
   │  └─ ProviderConfig/platform-ref-azure-cluster              -        -
   ├─ XNetwork/platform-ref-azure-cluster-f7dkm-9t4qm           True     True    Available
   │  ├─ ResourceGroup/platform-ref-azure-cluster-f7dkm-glbsd   True     True    Available
   │  ├─ Subnet/platform-ref-azure-cluster-sn                   True     True    Available
   │  └─ VirtualNetwork/platform-ref-azure-cluster-vnet         True     True    Available
   ├─ XFlux/platform-ref-azure-cluster-f7dkm-f5pkg              True     True    Available
   │  ├─ Release/platform-ref-azure-cluster-f7dkm-296hc         True     True    Available
   │  └─ Release/platform-ref-azure-cluster-f7dkm-cqzz8         True     True    Available
   └─ XOss/platform-ref-azure-cluster-f7dkm-cjq49               True     True    Available
      ├─ Usage/platform-ref-azure-cluster-f7dkm-849qw           -        True    Available
      ├─ Usage/platform-ref-azure-cluster-f7dkm-j62n6           -        True    Available
      ├─ Usage/platform-ref-azure-cluster-f7dkm-k2x2p           -        True    Available
      ├─ Release/platform-ref-azure-cluster                     True     True    Available
      ├─ Object/platform-ref-azure-cluster-f7dkm-42m2l          True     True    Available
      ├─ Object/platform-ref-azure-cluster-f7dkm-g6xvz          True     True    Available
      └─ Object/platform-ref-azure-cluster-f7dkm-g8sgh          True     True    Available

How can we reproduce it?

kubectl apply -f examples/cluster-claim.yaml

and wait for the provisioning

What environment did it happen in?

v0.12.0 release of platform-ref-azure
@ytsarev ytsarev added bug Something isn't working needs-epic-link Needs a link to an epic needs-points-label Needs a story point label needs-project Needs a project labels Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working needs-epic-link Needs a link to an epic needs-points-label Needs a story point label needs-project Needs a project
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ytsarev