I was recently writing some tutorials on Kubernetes RBAC and needed a little bit of insight into why my authorization wasn't working. But, in order to do that, I needed to turn on the audit logging in Kubernetes. Unforutnately, there isn't a single switch you can flip in Docker Desktop to do that. But, there's a way to manually do it! Here's how!
Enabling Audit Logging
The first thing you need to do is get inside the VM that Docker Desktop is using. Run the following command to do so:
docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh
I actually have this setup as an alias named
We need to create a Audit Policy document with the audit rules we want. In my case, I want to simply log all requests.
cat <<EOF > /etc/kubernetes/audit-policy.yaml apiVersion: audit.k8s.io/v1 # This is required. kind: Policy rules: # A catch-all rule to log all other requests at the Metadata level. - level: Metadata # Long-running requests like watches that fall under this rule will not # generate an audit event in RequestReceived. omitStages: - "RequestReceived" EOF
Now, we are going to navigate to
/etc/kuberentes/manifests. These are manifests that are manually applied by kubelet (learn more about "static pods" here). If we make changes to any of these manifests, the changes will be applied automatically, as kubelet is watching this directory.
First, make a copy of the file so we can easily revert the change later:
cp kube-apiserver.yaml kube-apiserver.yaml.orig
The file just can't have the
.yamlextension because kubelet will try to deploy it!
kube-apiserver.yamlfile, we are going to make the following changes:
spec.containers.command, we want to add the following command arguments:
- --audit-policy-file=/etc/kubernetes/audit-policy.yaml - --audit-log-path=/var/log/kubernetes/audit/audit.log
This will tell the API server that we want to use the audit policy and where to send the log events.
spec.containers.volumeMounts, we want to add the following:
- mountPath: /etc/kubernetes/audit-policy.yaml name: audit readOnly: true - mountPath: /var/log/kubernetes/audit/ name: audit-log readOnly: false
This sets up the mount points, which we'll connect to our host in just a second.
spec.volumes, we want to add the following:
- name: audit hostPath: path: /etc/kubernetes/audit-policy.yaml type: File - name: audit-log hostPath: path: /var/log/kubernetes/audit/ type: DirectoryOrCreate
This will cause the audit logs to drop in at
/var/log/kubernetes/audit/audit.login the Docker Desktop VM.
Save the file.
After a moment, you'll see the pod restart. Don't worry if you messed something up. You can always adjust this file and kubelet will redeploy the update.
With that, you can simply
tail -f /var/log/kubernetes/audit/audit.log and see your log messages. With the policy file I defined earlier, it'll be quite chatty.
Once you're done, it probably wouldn't be a bad idea to revert the changes to quite the audit logs.
mv /etc/kubernetes/manifests/kube-apiserver.yaml.orig /etc/kubernetes/manifests/kube-apiserver.yaml
That's it! If this was useful, let me know!