It is common to work with Kubernetes resources from within internal pods. The platform is obviously a great scheduler and orchestrator, but you may have custom logic that needs to make decisions from, or decisions for, Kubernetes resources.
This post will use a simple example. Say you have the requirement to list out all of the pods, but you need to do this from pods in the cluster (maybe you have to programmatically make decisions based on the state of resources). Here’s a simple Dockerfile for a sample image…
1 2 3 4 5 6 7 8 9 FROM debian:buster RUN apt update && \ apt install -y curl && \ curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl && \ chmod +x ./kubectl && \ mv ./kubectl /usr/local/bin/kubectl CMD kubectl get po
This is a way to create a docker image that includes the
kubectl bin. And then finally any container created from this image will just run
kubectl get po.
Your instinct might be to create a pod with the following config…
1 2 3 4 5 6 7 8 apiVersion: v1 kind: Pod metadata: name: internal-kubectl spec: containers: - name: internal-kubectl image: trstringer/internal-kubectl:latest
Attempting to run this pod (
$ kubectl apply -f pod.yaml), you would see the following error from the logs…
Error from server (Forbidden): pods is forbidden: User “system:serviceaccount:default:default” cannot list resource “pods” in API group “” in the namespace “default”
Unless specified otherwise (like above), pods will run under the
default service account, which is out-of-the-box for each namespace. As you can tell by the error message,
default doesn’t have the right permissions to list the pods.
The underlying force that is preventing us, and that we need to configure correctly, is RBAC. The way to tell Kubernetes that we want this pod to have an identity that can list the pods is through the combination of a few different resources…
1 2 3 4 apiVersion: v1 kind: ServiceAccount metadata: name: internal-kubectl
The identity object that we want to assign to our pod will be a service account. But by itself it has no permissions. That’s where roles come in.
1 2 3 4 5 6 7 8 9 10 11 12 apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: modify-pods rules: - apiGroups: [""] resources: - pods verbs: - get - list - delete
The role above specifies that we want to be able to get, list, and delete pods. But we need a way to correlate our new service account with our new role. Role bindings are the bridges for that…
1 2 3 4 5 6 7 8 9 10 11 apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: modify-pods-to-sa subjects: - kind: ServiceAccount name: internal-kubectl roleRef: kind: Role name: modify-pods apiGroup: rbac.authorization.k8s.io
This role binding connects our service account to the role that has the permissions we need. Now we just have to modify our pod config to include the service account…
1 2 3 4 5 6 7 8 9 apiVersion: v1 kind: Pod metadata: name: internal-kubectl spec: serviceAccountName: internal-kubectl containers: - name: internal-kubectl image: trstringer/internal-kubectl:latest
spec.serviceAccountName this changes us from using the
default service account to our new one that has the correct permissions. Running our new pod we should see the correct output…
1 2 3 $ kubectl logs internal-kubectl NAME READY STATUS RESTARTS AGE internal-kubectl 1/1 Running 1 5s