Posts Create a Basic Kubernetes Mutating Webhook
Post
Cancel

Create a Basic Kubernetes Mutating Webhook

Kubernetes is a very powerful platform, and one of its biggest powers is how extendable it is. One of these features is dynamic admission control. What that means is that we can add custom logic to determine what and if Kubernetes objects are created. This is a small blog series that focuses on this topic:

So how exactly does this work? What happens and when does it happen?

Mutating webhook workflow

When you do an operation on a resource (e.g. pod) the Kubernetes API server will look at the webhook configurations to see if there are any admission controls that it needs to apply. For all matched operations, it will first apply the mutating webhooks and then take that resource output and apply that to the validating webhooks. In this post we will be working with the former.

Note: The complete code can be found on my GitHub repository.

An admission control webhook is not much more than a webserver. Let’s create the initial function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
func runWebhookServer(certFile, keyFile string) {
    cert, err := tls.LoadX509KeyPair(certFile, keyFile)
    if err != nil {
        panic(err)
    }

    fmt.Println("Starting webhook server")
    http.HandleFunc("/mutate", mutatePod)
    server := http.Server{
        Addr: fmt.Sprintf(":%d", port),
        TLSConfig: &tls.Config{
            Certificates: []tls.Certificate{cert},
        },
        ErrorLog: logger,
    }

    if err := server.ListenAndServeTLS("", ""); err != nil {
        panic(err)
    }
}

Here we create an HTTP server (with TLS, more on that in a following blog post) that listens on a single route: /mutate. And the handler for this route is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
func admissionReviewFromRequest(r *http.Request, deserializer runtime.Decoder) (*admissionv1.AdmissionReview, error) {
    // Validate that the incoming content type is correct.
    if r.Header.Get("Content-Type") != "application/json" {
        return nil, fmt.Errorf("expected application/json content-type")
    }

    // Get the body data, which will be the AdmissionReview
    // content for the request.
    var body []byte
    if r.Body != nil {
        requestData, err := ioutil.ReadAll(r.Body)
        if err != nil {
            return nil, err
        }
        body = requestData
    }

    // Decode the request body into
    admissionReviewRequest := &admissionv1.AdmissionReview{}
    if _, _, err := deserializer.Decode(body, nil, admissionReviewRequest); err != nil {
        return nil, err
    }

    return admissionReviewRequest, nil
}

func mutatePod(w http.ResponseWriter, r *http.Request) {
    logger.Printf("received message on mutate")

    deserializer := codecs.UniversalDeserializer()

    // Parse the AdmissionReview from the http request.
    admissionReviewRequest, err := admissionReviewFromRequest(r, deserializer)
    if err != nil {
        msg := fmt.Sprintf("error getting admission review from request: %v", err)
        logger.Printf(msg)
        w.WriteHeader(400)
        w.Write([]byte(msg))
        return
    }

    // Do server-side validation that we are only dealing with a pod resource. This
    // should also be part of the MutatingWebhookConfiguration in the cluster, but
    // we should verify here before continuing.
    podResource := metav1.GroupVersionResource{Group: "", Version: "v1", Resource: "pods"}
    if admissionReviewRequest.Request.Resource != podResource {
        msg := fmt.Sprintf("did not receive pod, got %s", admissionReviewRequest.Request.Resource.Resource)
        logger.Printf(msg)
        w.WriteHeader(400)
        w.Write([]byte(msg))
        return
    }

    // Decode the pod from the AdmissionReview.
    rawRequest := admissionReviewRequest.Request.Object.Raw
    pod := corev1.Pod{}
    if _, _, err := deserializer.Decode(rawRequest, nil, &pod); err != nil {
        msg := fmt.Sprintf("error decoding raw pod: %v", err)
        logger.Printf(msg)
        w.WriteHeader(500)
        w.Write([]byte(msg))
        return
    }

    // Create a response that will add a label to the pod if it does
    // not already have a label with the key of "hello". In this case
    // it does not matter what the value is, as long as the key exists.
    admissionResponse := &admissionv1.AdmissionResponse{}
    var patch string
    patchType := v1.PatchTypeJSONPatch
    if _, ok := pod.Labels["hello"]; !ok {
        patch = `[{"op":"add","path":"/metadata/labels","value":{"hello":"world"}}]`
    }

    admissionResponse.Allowed = true
    if patch != "" {
        admissionResponse.PatchType = &patchType
        admissionResponse.Patch = []byte(patch)
    }

    // Construct the response, which is just another AdmissionReview.
    var admissionReviewResponse admissionv1.AdmissionReview
    admissionReviewResponse.Response = admissionResponse
    admissionReviewResponse.SetGroupVersionKind(admissionReviewRequest.GroupVersionKind())
    admissionReviewResponse.Response.UID = admissionReviewRequest.Request.UID

    resp, err := json.Marshal(admissionReviewResponse)
    if err != nil {
        msg := fmt.Sprintf("error marshalling response json: %v", err)
        logger.Printf(msg)
        w.WriteHeader(500)
        w.Write([]byte(msg))
        return
    }

    w.Header().Set("Content-Type", "application/json")
    w.Write(resp)
}

Hopefully the comments of this main function explain what’s happening, but in short we get the AdmissionReview from the request and then do a test on the labels that the pod has:

1
2
3
    if _, ok := pod.Labels["hello"]; !ok {
        patch = `[{"op":"add","path":"/metadata/labels","value":{"hello":"world"}}]`
    }

If the pod has a label with the key hello then there will be no action. But if the pod does not have this label, then it will create a JSON patch that adds a new label with the key hello and the value world.

Now let’s create this webhook in the Kubernetes cluster. First define the Dockerfile:

Dockerfile

1
2
3
4
FROM ubuntu:focal
WORKDIR /opt
COPY ./bin/mutating-webhook .
CMD ["./mutating-webhook", "--tls-cert", "/etc/opt/tls.crt", "--tls-key", "/etc/opt/tls.key"]

And now create the deployment:

webhook-deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
kind: Deployment
apiVersion: apps/v1
metadata:
  name: mutating-webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mutating-webhook
  template:
    metadata:
      labels:
        app: mutating-webhook
    spec:
      containers:
        - name: mutating-webhook
          image: localhost:5000/mutating-webhook:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 443
          volumeMounts:
            - name: cert
              mountPath: /etc/opt
              readOnly: true
      volumes:
        - name: cert
          secret:
            secretName: server-cert

And now we need the service that points to our webhook:

webhook-service.yaml

1
2
3
4
5
6
7
8
9
10
kind: Service
apiVersion: v1
metadata:
  name: mutating-webhook
spec:
  selector:
    app: mutating-webhook
  ports:
    - port: 443
      targetPort: 443

And finally, we need to create the MutatingWebhookConfiguration which tells the Kubernetes API server when and how to call our mutating webhook:

mutating-webhook-config.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
kind: MutatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
metadata:
  name: pod-label-add
  annotations:
    cert-manager.io/inject-ca-from: default/client
webhooks:
  - name: pod-label-add.trstringer.com
    clientConfig:
      service:
        namespace: default
        name: mutating-webhook
        path: /mutate
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
        operations: ["CREATE"]
        scope: Namespaced
    sideEffects: None
    admissionReviewVersions: ["v1"]

This does a couple of interesting things. It defines the clientConfig, which points to the mutating-webhook Kubernetes service that we created above (which, in turn, points to the deployment’s pods for the webhook). It also defines that the API server should use the /mutate route that we specified in our webhook server. Then we create the rules on which resources to apply this mutating webhook to. In this case, the mutating webhook should apply to all pods.

Let’s see how this works! Let’s create a pod:

1
2
3
4
5
6
7
8
9
10
11
12
$ kubectl apply -f - <<EOF
kind: Pod
apiVersion: v1
metadata:
  name: test-app
spec:
  containers:
    - name: test-app
      image: ubuntu:focal
      command: ["/bin/bash"]
      args: ["-c", "sleep infinity"]
EOF

Notice that we don’t define any labels for this pod. After the pod is created, let’s take a look at it:

1
2
3
4
5
6
7
8
9
10
$ kubectl describe po test-app
Name:         test-app
Namespace:    default
Priority:     0
Node:         kind-control-plane/172.18.0.3
Start Time:   Sun, 14 Nov 2021 14:42:16 -0500
Labels:       hello=world
Annotations:  <none>
Status:       Running
... output removed for brevity

It looks like our mutating webhook worked! You can see that the label hello=world was added to the pod. But if we specify the hello key, let’s verify that it doesn’t mutate the pod:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ kubectl apply -f - <<EOF
kind: Pod
apiVersion: v1
metadata:
  name: test-app
  labels:
    hello: universe
spec:
  containers:
    - name: test-app
      image: ubuntu:focal
      command: ["/bin/bash"]
      args: ["-c", "sleep infinity"]
EOF

And now let’s see the labels:

1
2
3
4
5
6
7
8
9
10
$ kubectl describe po test-app
Name:         test-app
Namespace:    default
Priority:     0
Node:         kind-control-plane/172.18.0.3
Start Time:   Sun, 14 Nov 2021 14:47:44 -0500
Labels:       hello=universe
Annotations:  <none>
Status:       Running
... output removed for brevity

Great! Our hello label was preserved and not overwritten.

Hopefully this blog post has illustrated how to implement a basic mutating webhook for dynamic admission control!

This post is licensed under CC BY 4.0 by the author.