diff --git a/docs/configure-rbac-permissions.md b/docs/configure-rbac-permissions.md index 0db8b16b..0e64a395 100644 --- a/docs/configure-rbac-permissions.md +++ b/docs/configure-rbac-permissions.md @@ -1,58 +1,64 @@ # Configure RBAC permissions -`Microsoft.ApplicationInsights.Kubernetes` uses the service account to query kubernetes information to enhance telemetries. It is important to have proper permissions configured for kubernetes related information like node, pod and so on to be fetched correctly. +`Microsoft.ApplicationInsights.Kubernetes` uses the service account to query Kubernetes information to enhance telemetries. It is important to have proper permissions configured for Kubernetes-related resources like Node, Pod, and so on to be fetched correctly. -In this post, we will start by describe a method to correctly configure the permissions for an RBAC enabled cluster. And then share a guidance for troubleshooting. +In this post, we will start by describing a method to correctly configure the permissions for an RBAC-enabled cluster. And then share troubleshooting guidance. ## Assumptions In this demo, we will have the following assumptions. Please change the related values accordingly: -* The application will be deployed to namespace of `ai-k8s-demo`. +* The application will be deployed to namespace `ai-k8s-demo`. * The application will leverage the `default` service account. -## Configure ClusterRole and ClusterRoleBinding for the service account - -* Create a yaml file, [sa-role.yaml](./sa-role.yaml) for example. We will deploy it when it is ready. - - * Write spec to define a cluster role, name it `appinsights-k8s-property-reader` for example: - - ```yaml - kind: ClusterRole - apiVersion: rbac.authorization.k8s.io/v1 - metadata: - # "namespace" omitted since ClusterRoles are not namespaced - name: appinsights-k8s-property-reader - rules: - - apiGroups: ["", "apps"] - resources: ["pods", "nodes", "replicasets", "deployments"] - verbs: ["get", "list"] - ``` - That spec defines the name of the role, and what permission does the role has, for example, list pods. - - You don't have to use the exact name, but you will need to making sure the name is referenced correctly in the following steps. - - * Append a Cluster role binding spec: - - ```yaml - --- - # actual binding to the role - kind: ClusterRoleBinding - apiVersion: rbac.authorization.k8s.io/v1 - metadata: - name: appinsights-k8s-property-reader-binding - subjects: - - kind: ServiceAccount - name: default - namespace: ai-k8s-demo - roleRef: - kind: ClusterRole - name: appinsights-k8s-property-reader - apiGroup: rbac.authorization.k8s.io - ``` - - That is to grant the role of `appinsights-k8s-property-reader` to the default service account in namespace of `ai-k8s-demo`. - +## Setup the permissions for the service account + +Depending on various considerations, there could be different strategies to set up the permissions for your service account. Here we list 2 common possibilities, as examples. + +* If you want to get the Node information along with other resource info like Pod, Deployment, and so on, a ClusterRole and a ClusterRoleBinding are required, and here's how to do it: + + * Create a yaml file, [sa-role.yaml](./sa-role.yaml) for example. We will deploy it when it is ready. + + * Write spec to define a cluster role, name it `appinsights-k8s-property-reader` for example: + + ```yaml + kind: ClusterRole + apiVersion: rbac.authorization.k8s.io/v1 + metadata: + # "namespace" omitted since ClusterRoles are not namespaced + name: appinsights-k8s-property-reader + rules: + - apiGroups: ["", "apps"] + resources: ["pods", "nodes", "replicasets", "deployments"] + verbs: ["get", "list"] + ``` + That spec defines the name of the role, and what permission does the role has, for example, list pods. + + You don't have to use the exact name, but you will need to making sure the name is referenced correctly in the following steps. + + * Append a Cluster role binding spec: + + ```yaml + --- + # actual binding to the role + kind: ClusterRoleBinding + apiVersion: rbac.authorization.k8s.io/v1 + metadata: + name: appinsights-k8s-property-reader-binding + subjects: + - kind: ServiceAccount + name: default + namespace: ai-k8s-demo + roleRef: + kind: ClusterRole + name: appinsights-k8s-property-reader + apiGroup: rbac.authorization.k8s.io + ``` + + That is to grant the role of `appinsights-k8s-property-reader` to the default service account in the namespace of `ai-k8s-demo`. + + * If you don't want to create a Cluster Role, it is also possible to use Role and RoleBinding starting with Application Insights for Kubernetes 2.0.6+. Follow the example in [sa-role-none-cluster.yaml](./sa-role-none-cluster.yaml). In that case, you will not have node info on the telemetries. + * Now you can deploy it: ```shell @@ -60,11 +66,11 @@ In this demo, we will have the following assumptions. Please change the related ``` See [sa-role.yaml](sa-role.yaml) for a full example. -> :warning: Check back for various permissions needed. Depends on the implementations, it may change in the over time. +> :warning: Check back for various permissions needed. Depending on the properties we try to fetch, it may change over time. ## Ad-hoc troubleshooting for permission -Kubectl provides an `auth --can-i` sub command for troubleshooting permissions. It supports impersonate the service account. We can leverage it for permission troubleshooting, for example: +Kubectl provides an `auth --can-i` subcommand for troubleshooting permissions. It supports impersonating the service account. We can leverage it for permission troubleshooting, for example: ```shell kubectl auth can-i list pod --namespace ai-k8s-demo --as system:serviceaccount:ai-k8s-demo:default @@ -84,7 +90,7 @@ yes ## Use SubjectAccessReview -Kubernetes also provides `SubjectAccessReview` to check permission for given user on target resource. +Kubernetes also provides `SubjectAccessReview` to check permission for a given user on the target resource. ### Basic usage @@ -147,7 +153,7 @@ Kubernetes also provides `SubjectAccessReview` to check permission for given use * [subject-access-review-key.yaml](./subject-access-review-key.yaml): a subset of permissions for probing the RBAC settings. * [subject-access-review-full.yaml](./subject-access-review-full.yaml): a full list of all permissions needed for RBAC settings in case any specific permission is missing. -Let us know if there's questions, suggestions by filing [issues](https://github.com/microsoft/ApplicationInsights-Kubernetes/issues). +Let us know if there are questions or suggestions by filing [issues](https://github.com/microsoft/ApplicationInsights-Kubernetes/issues). ## References diff --git a/docs/sa-role-none-cluster.yaml b/docs/sa-role-none-cluster.yaml new file mode 100644 index 00000000..9d6bd947 --- /dev/null +++ b/docs/sa-role-none-cluster.yaml @@ -0,0 +1,24 @@ +kind: Role +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + namespace: ai-k8s-demo + name: appinsights-k8s-property-reader-role +rules: +- apiGroups: ["", "apps"] + resources: ["pods", "replicasets", "deployments"] + verbs: ["get", "list"] +--- +# Actual RoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: appinsights-k8s-property-reader-binding + namespace: ai-k8s-demo +subjects: +- kind: ServiceAccount + name: default + namespace: ai-k8s-demo +roleRef: + kind: Role + name: appinsights-k8s-property-reader-role + apiGroup: rbac.authorization.k8s.io diff --git a/src/ApplicationInsights.Kubernetes/IK8sQueryClient.cs b/src/ApplicationInsights.Kubernetes/IK8sQueryClient.cs index 118414f6..94c61cd2 100644 --- a/src/ApplicationInsights.Kubernetes/IK8sQueryClient.cs +++ b/src/ApplicationInsights.Kubernetes/IK8sQueryClient.cs @@ -11,7 +11,7 @@ namespace Microsoft.ApplicationInsights.Kubernetes internal interface IK8sQueryClient : IDisposable { Task> GetDeploymentsAsync(CancellationToken cancellationToken); - Task> GetNodesAsync(CancellationToken cancellationToken); + Task> GetNodesAsync(bool ignoreForbiddenException, CancellationToken cancellationToken); Task> GetPodsAsync(CancellationToken cancellationToken); Task GetPodAsync(string podName, CancellationToken cancellationToken); Task> GetReplicasAsync(CancellationToken cancellationToken); diff --git a/src/ApplicationInsights.Kubernetes/K8sEnvironmentFactory.cs b/src/ApplicationInsights.Kubernetes/K8sEnvironmentFactory.cs index a2a8d272..1574a40e 100644 --- a/src/ApplicationInsights.Kubernetes/K8sEnvironmentFactory.cs +++ b/src/ApplicationInsights.Kubernetes/K8sEnvironmentFactory.cs @@ -106,14 +106,15 @@ public K8sEnvironmentFactory( if (instance.myPod is not null) { - IEnumerable nodeList = await queryClient.GetNodesAsync(cancellationToken).ConfigureAwait(false); + IEnumerable nodeList = await queryClient.GetNodesAsync(ignoreForbiddenException: true, cancellationToken: cancellationToken).ConfigureAwait(false); string nodeName = instance.myPod.Spec.NodeName; - if (!string.IsNullOrEmpty(nodeName)) + if (!string.IsNullOrEmpty(nodeName) && nodeList.Any()) { instance.myNode = nodeList.FirstOrDefault(node => string.Equals(node.Metadata?.Name, nodeName, StringComparison.OrdinalIgnoreCase)); } } } + return instance; } catch (UnauthorizedAccessException ex) diff --git a/src/ApplicationInsights.Kubernetes/K8sQueryClient.cs b/src/ApplicationInsights.Kubernetes/K8sQueryClient.cs index 749c1fc4..c4046790 100644 --- a/src/ApplicationInsights.Kubernetes/K8sQueryClient.cs +++ b/src/ApplicationInsights.Kubernetes/K8sQueryClient.cs @@ -91,12 +91,27 @@ public Task> GetDeploymentsAsync(CancellationToken ca #endregion #region Node - public Task> GetNodesAsync(CancellationToken cancellationToken) + public async Task> GetNodesAsync(bool ignoreForbiddenException, CancellationToken cancellationToken) { EnsureNotDisposed(); - string url = Invariant($"api/v1/nodes"); - return GetAllItemsAsync(url, cancellationToken); + try + { + string url = Invariant($"api/v1/nodes"); + return await GetAllItemsAsync(url, cancellationToken).ConfigureAwait(false); + } + catch (UnauthorizedAccessException ex) + { + // Prefer ignoring the unauthorized access exception + if (ignoreForbiddenException) + { + _logger.LogDebug(ex.Message); + _logger.LogTrace(ex.ToString()); + return Enumerable.Empty(); + } + // else + throw; + } } #endregion diff --git a/src/ApplicationInsights.Kubernetes/TelemetryInitializers/KubernetesTelemetryInitializer.cs b/src/ApplicationInsights.Kubernetes/TelemetryInitializers/KubernetesTelemetryInitializer.cs index 170e790c..ed7b055d 100644 --- a/src/ApplicationInsights.Kubernetes/TelemetryInitializers/KubernetesTelemetryInitializer.cs +++ b/src/ApplicationInsights.Kubernetes/TelemetryInitializers/KubernetesTelemetryInitializer.cs @@ -159,8 +159,8 @@ private void SetCustomDimensions(ISupportProperties telemetry) SetCustomDimension(telemetry, Deployment.Name, this._k8sEnvironment.DeploymentName, isValueOptional: true); // Node - SetCustomDimension(telemetry, Node.ID, this._k8sEnvironment.NodeUid); - SetCustomDimension(telemetry, Node.Name, this._k8sEnvironment.NodeName); + SetCustomDimension(telemetry, Node.ID, this._k8sEnvironment.NodeUid, isValueOptional: true); + SetCustomDimension(telemetry, Node.Name, this._k8sEnvironment.NodeName, isValueOptional: true); } private void SetCustomDimension(ISupportProperties telemetry, string key, string value, bool isValueOptional = false) diff --git a/tests/UnitTests/K8sQueryClientTests.cs b/tests/UnitTests/K8sQueryClientTests.cs index c24bb41b..230f54d3 100644 --- a/tests/UnitTests/K8sQueryClientTests.cs +++ b/tests/UnitTests/K8sQueryClientTests.cs @@ -215,7 +215,7 @@ public async Task GetNodesAsyncShouldHitsTheUri() httpClientMock.Setup(httpClient => httpClient.SendAsync(It.IsAny(), It.IsAny())).Returns(Task.FromResult(response)); using (K8sQueryClient target = new K8sQueryClient(httpClientMock.Object)) { - await target.GetNodesAsync(cancellationToken: default); + await target.GetNodesAsync(ignoreForbiddenException: true, cancellationToken: default); } httpClientMock.Verify(mock => mock.SendAsync(It.Is(m => m.RequestUri.AbsoluteUri.Equals("https://baseaddress/api/v1/nodes")), It.IsAny()), Times.Once); @@ -245,7 +245,7 @@ public async Task GetNodesAsyncShouldReturnsMultipleNodes() using (K8sQueryClient target = new K8sQueryClient(httpClientMock.Object)) { - IEnumerable result = await target.GetNodesAsync(cancellationToken: default); + IEnumerable result = await target.GetNodesAsync(ignoreForbiddenException: true, cancellationToken: default); Assert.NotNull(result); Assert.Equal(2, result.Count()); diff --git a/tests/UnitTests/KubernetesTelemetryInitializerTests.cs b/tests/UnitTests/KubernetesTelemetryInitializerTests.cs index df03b2be..f02a39ae 100644 --- a/tests/UnitTests/KubernetesTelemetryInitializerTests.cs +++ b/tests/UnitTests/KubernetesTelemetryInitializerTests.cs @@ -144,17 +144,18 @@ public void InitializeWithEmptyForRequiredPropertyDoesLogError() var envMock = new Mock(); envMock.Setup(env => env.ContainerName).Returns("Hello RoleName"); + // These 2 properties are required. + envMock.Setup(env => env.PodID).Returns(null); + envMock.Setup(env => env.PodName).Returns(null); + envMock.Setup(env => env.ContainerID).Returns("Cid"); envMock.Setup(env => env.ContainerName).Returns("CName"); - envMock.Setup(env => env.PodID).Returns("Pid"); - envMock.Setup(env => env.PodName).Returns("PName"); envMock.Setup(env => env.PodLabels).Returns("PLabels"); envMock.Setup(env => env.ReplicaSetUid).Returns(null); envMock.Setup(env => env.ReplicaSetName).Returns(null); envMock.Setup(env => env.DeploymentUid).Returns(null); envMock.Setup(env => env.DeploymentName).Returns(null); envMock.Setup(env => env.PodNamespace).Returns(null); - // These 2 properties are required. envMock.Setup(env => env.NodeUid).Returns(null); envMock.Setup(env => env.NodeName).Returns(null);