insights-remote checks secrets on deleted environments and hangs

[tobybellwood]

When the API triggers a delete on an environment, the insights-remote seems to see this as an opportunity to check the secret in the namespace. This usually fails, as the namespace is being terminated - and the controller just spits out errors until it no longer exists, then goes back to queue processing.

We should introduce smarts to:
a) stop it processing secrets on a delete
b) fail quickly and neatly on a terminating/missing namespace

Aug 13, 2024 @ 12:35:20.685	api:deleteEnvironment	User deleted environment 'def-123' on project 'abc'
...
2024-08-13T12:35:40+10:00 2024-08-13T02:35:40.877Z	INFO	controller-runtime.manager.controller.namespace	Found secret with name 'insights-token' and namespace 'abc-def-123'	{"reconciler group": "", "reconciler kind": "Namespace", "name": "abc-def-123", "namespace": ""}
2024-08-13T12:35:41+10:00 2024-08-13T02:35:41.457Z	INFO	controller-runtime.manager.controller.namespace	Found secret with name 'insights-token' and namespace 'abc-def-123'	{"reconciler group": "", "reconciler kind": "Namespace", "name": "abc-def-123", "namespace": ""}
2024-08-13T12:35:47+10:00 2024-08-13T02:35:47.225Z	ERROR	controller-runtime.manager.controller.namespace	Reconciler error	{"reconciler group": "", "reconciler kind": "Namespace", "name": "abc-def-123", "namespace": "", "error": "secrets \"insights-token\" is forbidden: unable to create new content in namespace abc-def-123 because it is being terminated"}
2024-08-13T12:35:47+10:00 github.com/go-logr/zapr.(*zapLogger).Error
2024-08-13T12:35:47+10:00 	/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
2024-08-13T12:35:47+10:00 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2024-08-13T12:35:47+10:00 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:302
2024-08-13T12:35:47+10:00 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2024-08-13T12:35:47+10:00 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
2024-08-13T12:35:47+10:00 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
2024-08-13T12:35:47+10:00 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
2024-08-13T12:35:47+10:00 k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
2024-08-13T12:35:47+10:00 	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
2024-08-13T12:35:47+10:00 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
2024-08-13T12:35:47+10:00 	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
2024-08-13T12:35:47+10:00 k8s.io/apimachinery/pkg/util/wait.BackoffUntil
2024-08-13T12:35:47+10:00 	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
2024-08-13T12:35:47+10:00 k8s.io/apimachinery/pkg/util/wait.JitterUntil
2024-08-13T12:35:47+10:00 	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
2024-08-13T12:35:47+10:00 k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
2024-08-13T12:35:47+10:00 	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185
2024-08-13T12:35:47+10:00 k8s.io/apimachinery/pkg/util/wait.UntilWithContext
2024-08-13T12:35:47+10:00 	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99