Random failure publishing new secrets on changes

[mkherlakian]

We've successfully deployed the Doppler Operator to our environments a little over 2 months ago.

We're noticing however that while everything works as it should post-deployment (we can change a secret in our config, and run kubectl describe secrets --selector=secrets.doppler.com/subtype=dopplerSecret , we will see the changes almost immediately), after a while (days) the operator doesn't seem to pick up the new secrets anymore.

In order to fix this, we have to delete the CRD for that environment and recreate it.

We dug through the logs, but didn't notice anything out of the ordinary.

Happy to provide more info/details/logs that would help in diagnosing this. We are on GKE, k8s 1.23.

[nmanoogian]

Hey @mkherlakian, thanks for reporting this!

When the operator is in the "stuck" state, could you share the output from kubectl describe on the CRD itself? There are attached conditions which are often useful.

Did you check out the operator manager logs with this script? https://github.com/DopplerHQ/kubernetes-operator/blob/main/tools/operator-logs.sh

They're a bit verbose but often tell the whole story.

[nmanoogian]

Hi @mkherlakian,

Are you still running into this issue?

[mkherlakian]

Hi @nmanoogian,

AS it turns out I had to do a deployment in a namespace that we don't use often, and sure enough that one was not picking up the changes. Here's a describe of the CRD.

Name:         dopplersecrets.secrets.doppler.com
Namespace:
Labels:       owner=doppler
Annotations:  controller-gen.kubebuilder.io/version: v0.4.1
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2022-07-29T17:47:45Z
  Generation:          1
  Managed Fields:
    API Version:  apiextensions.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:acceptedNames:
          f:kind:
          f:listKind:
          f:plural:
          f:singular:
        f:conditions:
          k:{"type":"Established"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"NamesAccepted"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
    Manager:      kube-apiserver
    Operation:    Update
    Time:         2022-07-29T17:47:45Z
    API Version:  apiextensions.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:controller-gen.kubebuilder.io/version:
          f:kubectl.kubernetes.io/last-applied-configuration:
        f:labels:
          .:
          f:owner:
      f:spec:
        f:conversion:
          .:
          f:strategy:
        f:group:
        f:names:
          f:kind:
          f:listKind:
          f:plural:
          f:singular:
        f:scope:
        f:versions:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2022-07-29T17:47:45Z
  Resource Version:  349291445
  UID:               8054024a-1c72-410d-b85d-2fc72f8dadca
Spec:
  Conversion:
    Strategy:  None
  Group:       secrets.doppler.com
  Names:
    Kind:       DopplerSecret
    List Kind:  DopplerSecretList
    Plural:     dopplersecrets
    Singular:   dopplersecret
  Scope:        Namespaced
  Versions:
    Name:  v1alpha1
    Schema:
      openAPIV3Schema:
        Description:  DopplerSecret is the Schema for the dopplersecrets API
        Properties:
          API Version:
            Description:  APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
            Type:         string
          Kind:
            Description:  Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
            Type:         string
          Metadata:
            Type:  object
          Spec:
            Description:  DopplerSecretSpec defines the desired state of DopplerSecret
            Properties:
              Host:
                Default:      https://api.doppler.com
                Description:  The Doppler API host
                Type:         string
              Managed Secret:
                Description:  The Kubernetes secret where the operator will store and sync the fetched secrets
                Properties:
                  Name:
                    Description:  The name of the Secret resource
                    Type:         string
                  Namespace:
                    Description:  Namespace of the resource being referred to. Ignored if not cluster scoped
                    Type:         string
                Required:
                  name
                Type:  object
              Processors:
                Additional Properties:
                  Properties:
                    Type:
                      Description:  The type of process to be performed, either "plain" or "base64"
                      Enum:
                        plain
                        base64
                      Type:  string
                  Required:
                    type
                  Type:       object
                Description:  A list of processors to transform the data during ingestion
                Type:         object
              Resync Seconds:
                Default:      60
                Description:  The number of seconds to wait between resyncs
                Format:       int64
                Type:         integer
              Token Secret:
                Description:  The Kubernetes secret containing the Doppler service token
                Properties:
                  Name:
                    Description:  The name of the Secret resource
                    Type:         string
                  Namespace:
                    Description:  Namespace of the resource being referred to. Ignored if not cluster scoped
                    Type:         string
                Required:
                  name
                Type:  object
              Verify TLS:
                Default:      true
                Description:  Whether or not to verify TLS
                Type:         boolean
            Type:             object
          Status:
            Description:  DopplerSecretStatus defines the observed state of DopplerSecret
            Properties:
              Conditions:
                Items:
                  Description:  Condition contains details for one aspect of the current state of this API Resource. --- This struct is intended for direct use as an array at the field path .status.conditions.  For example, type FooStatus struct{     // Represents the observations of a foo's current state.     // Known .status.conditions.type are: "Available", "Progressing", and "Degraded"     // +patchMergeKey=type     // +patchStrategy=merge     // +listType=map     // +listMapKey=type     Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type" protobuf:"bytes,1,rep,name=conditions"`
     // other fields }
                  Properties:
                    Last Transition Time:
                      Description:  lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed.  If that is not known, then using the time when the API field changed is acceptable.
                      Format:       date-time
                      Type:         string
                    Message:
                      Description:  message is a human readable message indicating details about the transition. This may be an empty string.
                      Max Length:   32768
                      Type:         string
                    Observed Generation:
                      Description:  observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.
                      Format:       int64
                      Minimum:      0
                      Type:         integer
                    Reason:
                      Description:  reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.
                      Max Length:   1024
                      Min Length:   1
                      Pattern:      ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
                      Type:         string
                    Status:
                      Description:  status of the condition, one of True, False, Unknown.
                      Enum:
                        True
                        False
                        Unknown
                      Type:  string
                    Type:
                      Description:  type of condition in CamelCase or in foo.example.com/CamelCase. --- Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be useful (see .node.status.conditions), the ability to deconflict is important. The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
                      Max Length:   316
                      Pattern:      ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
                      Type:         string
                  Required:
                    lastTransitionTime
                    message
                    reason
                    status
                    type
                  Type:  object
                Type:    array
            Required:
              conditions
            Type:  object
        Type:      object
    Served:        true
    Storage:       true
    Subresources:
      Status:
Status:
  Accepted Names:
    Kind:       DopplerSecret
    List Kind:  DopplerSecretList
    Plural:     dopplersecrets
    Singular:   dopplersecret
  Conditions:
    Last Transition Time:  2022-07-29T17:47:45Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2022-07-29T17:47:45Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
  Stored Versions:
    v1alpha1

Getting the logs now.

[nmanoogian]

Thanks, @mkherlakian! Any luck on getting those logs? Things are looking normal from the CRD describe.