Permission denied upgrading from resource discovery 0.8.0 to 0.9.0

[deggja]

Hey guys & @tomkerkhove,

I'm trying to upgrade my promitor resource discovery installation using your helm chart. It is currently running 0.8.0 without any issues, but I am having some trouble upgrading:

After fetching the latest chart and running helm upgrade, I'm stuck in a crashloop because of a permission denied error message:

[12:05:27 ERR] BackgroundService failed
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at CronScheduler.Extensions.Internal.SchedulerHostedService.ExecuteAsync(CancellationToken stoppingToken) in C:\projects\cronscheduler-aspnetcore\src\CronScheduler.Extensions\Internal\SchedulerHostedService.cs:line 37
   at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
[12:05:27 FTL] The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at CronScheduler.Extensions.Internal.SchedulerHostedService.ExecuteAsync(CancellationToken stoppingToken) in C:\projects\cronscheduler-aspnetcore\src\CronScheduler.Extensions\Internal\SchedulerHostedService.cs:line 37
   at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
[12:05:27 FTL] Promitor Discovery Agent has encountered an unexpected error. Please open an issue at https://github.com/tomkerkhove/promitor/issues to let us know about it.
System.Net.Sockets.SocketException (13): Permission denied
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass30_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.AnyIPListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(IEnumerable`1 listenOptions, AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Promitor.Agents.ResourceDiscovery.Program.Main(String[] args) in /src/Promitor.Agents.ResourceDiscovery/Program.cs:line 50

I am using the exact same credentials for 0.8.0 and 0.9.0 - so I am a bit confused as to why this should cause any problems. I've checked the release notes and I dont see any mention of any breaking changes in the upgrade. I have rolled back to 0.8.0 for the time being so nothing urgent.

Any advice on how to move forward or how to avoid this issue?

FYI: I upgraded promitor-agent-scraper from 2.8.0 to 2.9.0 the same way without any issues, so whatever the problem is, it seems to be isolated to the resource discovery chart.

Thank you in advance.

[tomkerkhove]

Interesting, it looks like it is not allowed to expose the endpoint. Do you mind sharing how you are deploying it through Helm in terms of configuration?

[deggja]

Thanks for answering Tom.

Sure. I'll append my values below.

# Default values for promitor-agent-resource-discovery.
# This is a YAML-formatted file.

replicaCount: 1

nameOverride: "promitor-agent-resource-discovery-con"
fullnameOverride: ""

image:
  repository: containers.promitor.io/tomkerkhove/promitor-agent-resource-discovery
  pullPolicy: Always
  tag:

azureAuthentication:
  appId: "" # [Deprecated] Prefer identity.id
  appKey: "" # [Deprecated] Prefer identity.key
  mode: "ServicePrincipal"
  identity:
    id: ""
    key: ""
    binding: ""

cache:
  enabled: true
  durationInMinutes: 5

prometheus:
  enabled: true
  enableServiceDiscovery: true
  serviceMonitor:
    enabled: false
    namespace: ""
    labels: {}
    interval: 60s
    timeout: 10s

telemetry:
  defaultLogLevel: information
  applicationInsights:
    enabled: false
    key: ""
    logLevel: ""
  containerLogs:
    enabled: true
    logLevel: ""

## Metric Declaration YAML
azureLandscape:
  cloud: Global
  tenantId: xxxxxxxxx
  subscriptions:
  - xxxxxxxxxxxx
  - xxxxxxxxxxxx
  - xxxxxxxxxxxx
  - xxxxxxxxxxxx
  - xxxxxxxxxxxx

resourceDiscoveryGroups:
- name: sql-databases
  type: SqlDatabase
- name: virtual-machines
  type: VirtualMachine
- name: network-gateway-group
  type: NetworkGateway
- name: storage-accounts
  type: StorageAccount
- name: application-gateways
  type: ApplicationGateway
- name: container-instances
  type: ContainerInstance
- name: redis-cache
  type: RedisCache
- name: app-plans
  type: AppPlan
- name: mysql-database
  type: MySql
  
deployment:
  env:
    extra: []
    # - name: AZURE_STORAGE_QUEUE_SAS_TOKEN
    #   valueFrom:
    #     secretKeyRef:
    #       name: promitor-agent-resource-discovery
    #       key: azure-storage-queue-sas-token

secrets:
  # To use your own secret, set createSecret to false and define the name/keys that your secret uses
  createSecret: true
  secretName: ""
  appKeySecret: azure-app-key
  extra: {}
  # azure-storage-queue-sas-token: <value>

service:
  port: 8889
  targetPort: 88
  loadBalancer:
    enabled: true
    azure:
      dnsPrefix: promitor-scraper-con
      exposeInternally: false

health:
  readiness:
    enabled: true
    verifyDependencies: false
    delay: 5
    interval: 5
    timeoutSeconds: 1
    thresholds:
      failure: 3
      success: 1
  liveness:
    enabled: true
    verifyDependencies: false
    delay: 5
    interval: 30
    timeoutSeconds: 1
    thresholds:
      failure: 3
      success: 1

affinity: {}

podLabels: {}

annotations: {}

priorityClassName: ""

resources: {}
  # limits:
  #  cpu: 100m
  #  memory: 128Mi
  # requests:
  #  cpu: 100m
  #  memory: 128Mi

## Role-based access control
## https://kubernetes.io/docs/reference/access-authn-authz/rbac/
rbac:
  ## If true, create & use RBAC resources
  create: true

  ## If true, create & use Pod Security Policy resources
  ## ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/
  podSecurityPolicyEnabled: false

  ## Service Account for pods
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
  serviceAccount:
    ## Specifies whether a service account should be created
    create: true

    ## The name of the service account to use if create is false
    ## If create is true, a name is generated using the fullname template
    name: promitor-resource-discovery
    annotations: {}

nodeSelector: {}

securityContext:
  enabled: false

tolerations: []

## Pass extra volumeMounts to the promitor container
extraVolumeMounts: []
  # - name: secrets-store-inline
  #   mountPath: "/mnt/promitor-secrets"
  #   readOnly: true

## Pass extra volumes to the promitor deployment
extraVolumes: []
  # - name: secrets-store-inline
  #   csi:
  #       driver: secrets-store.csi.k8s.io
  #       readOnly: true
  #       volumeAttributes:
  #         secretProviderClass: "promitor-secrets"

I am pretty sure its all default config and I've added resourceDiscoveryGroups and azureLandscape. I'm using the following command to upgrade the running install (however I also tried uninstalling and reinstalling the app).

helm upgrade --install promitor-agent-resource-discovery promitor/promitor-agent-resource-discovery \
               --set azureAuthentication.identity.mode='ServicePrincipal' \
               --set azureAuthentication.identity.id='xxxxxxxxxxxx' \
               --set azureAuthentication.identity.key='xxxxxxxxxxxx' \
               --values promitor-resource-discovery.yml

Thanks again!

[deggja]

I tried this again using the new chart version 0.9.1 - but seems like I'm still getting this error message. Any idea @tomkerkhove?

[deggja]

Never mind this - I managed to fix this myself. It was a configuration error all along. I was using my values file from 0.8.0 and there seems to have been something missing from it. I'm not exactly sure what solved it - but when I updated the values to contain all the default values - it works.

[deggja]

It was probably this value:

service:
   port: 8889
-  targetPort: 88
 ## By default this pod is running as a non-root user.
 ## If you choose targetPort <1024 it will fail to start.
 ## targetPort: 5000

Old chart values had port 88 - which I suppose will fail the startup.

[tomkerkhove]

Sorry about that =)