This article is a collection of techniques that have proven valuable when interacting with a Kubernetes cluster, especially when developing or debugging applications deployed to the cluster.

Entering a Pod/Container to “poke around”

If a container or application doesn’t behave as expected, one thing to do during debugging is to exec-into the misbehaving container to inspect the environment, the file system (mounted or in the image).

The following command launches a shell inside the first container of a Pod

$ kubectl exec -it your-pod-name -- sh

Depending on the image, other shells might be available, like bash or zsh. During development, it is perfectly fine to install any required debugging tools inside the running Pod. You will find me installing vim in various containers during debugging. The key is: Containers are ephemeral. Once the root cause of the issue is found, it’s easy to recreate the Pod, and the debugging mess is gone. That’s even more comfortable than debugging in a local clone of a repository.

Once inside, the following list of actions have proven useful:

  • Check the version of the code (maybe it’s not the correct Docker image)
  • Check the environment variables: env | sort
  • Run part of the application by hand to see directly what it’s doing
  • Check if mounted volumes are available: mount
  • Check file and directory permissions, especially for mounted volumes
  • Inspect running processes:
    • If available ps or top or htop
    • Otherwise, have a look at /proc. There is a lot of information in /proc, for example running commands /proc/1/cmdline, their working directories /proc/1/cwd.
  • Connect to dependent services
    • With curl URL for HTTP connections.
    • With nc -zv IP PORT for plain TCP (and UDP) connections.
    • With any application-specific tool, e.g., psql to debug PostgreSQL-related problems.

View content of a PVC

An easy solution to view the content of a PersistantVolumeClaim (PVC), is to mount the PVC in a dedicated container and exec-into the container. This solution works for virtually every storage provider.

apiVersion: v1
kind: Pod
metadata:
  name: pvc-inspector
spec:
  containers:
  - image: busybox
    name: pvc-inspector
    command: ["tail"]
    args: ["-f", "/dev/null"]
    volumeMounts:
    - mountPath: /pvc
      name: pvc-mount
  volumes:
  - name: pvc-mount
    persistentVolumeClaim:
      claimName: YOUR_CLAIM_NAME_HERE

More info on StackOverflow or using my templating service.

Entering a constantly failing Pod/Container

If a Pod is constantly failing at startup and ends up in the dreaded CrashLoopBackOff state, you cannot use kubectl exec -it to go into the container. Entering the container requires it to be running. If the pod is part of a managed resource, e.g., a Deployment, there is a trick to force the container to start.

Consider the following modification.

 apiVersion: apps/v1
 kind: Deployment
 metadata:
   labels:
     app: test
   name: test
 spec:
   selector:
     matchLabels:
       app: test
   template:
     metadata:
       labels:
         app: test
     spec:
       containers:
       - image: "myapplication:0.1.0"
-        command: ["python", "myapp.py"]
+        command: ["tail", "-f", "/dev/null"]
         name: test

Using tail -f /dev/null is a common trick to block the execution of a container indefinitely. tail -f reads from the given file and waits until content is available. However, the special file /dev/null will always be empty.

With this modification, the Pod will enter the Running state, and entering the container is possible again. Start debugging by launching the original application, here, python myapp.py.

Edit the application before launching it

Let’s assume you need additional debugging output from an application. However, the application reads the logging level upon start, so once you enter the container, it’s already to late too change the log verbosity. The situation is similar when working in an interpreted language. You might want to add additional debug output, but once the application starts, it’s to late too change the script.

To tackle this situation, replace the default Pod command by tail -f /dev/null to prevent it from starting the application.

       containers:
       - image: "myapplication:0.1.0"
-        command: ["python", "myapp.py"]
+        command: ["tail", "-f", "/dev/null"]

If you enter the container now, there is plenty of time to do any modifications to the environment variables, config files, or scripts before launching the application by hand. Install the text editor of your choice.

Probing ports and services

Many applications in Kubernetes use network communication between individual services of the application or external services and clients. During debugging, it is essential to see if a connection is possible. To test external connection, e.g. via port forwarding, or internally after entering a container, use

$ nc -vz IP_ADDRESS_OR_HOSTNAME PORT

to check if the TCP handshake succeeds. The command can be used with IP addresses or hostnames to test DNS resolution at the same time. Test connections to Pod IP addresses and Service IP addresses (ClusterIP or NodePort).

Prepare a configuration file

When using pre-built Docker images and applications, it can be challenging at times to dynamically create a configuration file based on environment variables, config maps, or secrets. Usually, this is achieved with container entry points. However, entry points are usually already in use by pre-build Images, and templating tools to create the config might not be available in these images.

In these cases, I suggest creating an emptyDir PVC, adding an init container to the Pod, mounting the PVC in the init container, and the main container. The init job can be used to create the dynamic config in the PVC. The main container will be able to read the config from the PVC.

Force Deployment rollout if config maps change in Helm

When a config map changes in a Helm-based deployment, Pods consuming the config map are not necessarily restarted automatically. This will likely lead to unexpected behavior. To save you headaches and a debugging session, you can add the config map’s checksum as an annotation. Deployments will automatically restart their Pods if the config map and therefore the annotation changed.


kind: Deployment
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

See the Helm documentation for more details.

Auto-generate secrets in Helm

Almost all Helm-based applications require secrets. For example, to set up internal databases. It is convenient to create a secret upon installation, that is then shared between the database and the application consuming the database. This can be achieved using a switch based on .Release.IsInstall. When the chart is first installed, a random sercret is created. Every subsequent upgrade will read the existing secret from the cluster, yielding an identical manifest.


apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: {{ print .Release.Name | trunc -63 }}
data:
  {{- if .Release.IsInstall }}
  SECRET_KEY: {{ randAlphaNum 20 | b64enc }}
  {{- else }}
  SECRET_KEY: {{ (lookup "v1" "Secret" .Release.Namespace (print .Release.Name | trunc -63)).data.SECRET_KEY }}
  {{- end }}

More?

What’s your favorite technique? Drop me an e-mail to extend the list.