Kubernetes Security 101: Best Practices to Secure your Cluster – SecAdvent Day 17

Container adoption has surged in recent years, with the “2019 cloud native computing foundation survey” reporting 84% of their respondents use some type of containerization in production.

The same survey also found 78% of respondents use Kubernetes in production, making it the market leader container orchestration solution and widely adopted across global tech companies and startups.

With clear benefits and rising adoption, it is critical that the security of Kubernetes is well-understood by any Developer or DevOps person implementing this service in their environment. To help those engineers distill all the information available on this topic, here are the key steps for securing your clusters.

What needs to be secured?

Any part of a cluster could be abused if accessed by an attacker. Let’s look at some specific steps to ensure your deployments are secure, focusing on the control plane and worker nodes along with the sub-parts within each.

The control plane

If you run your clusters using managed services such as AKS, EKS or GKE, the cloud provider handles control plane security. This is based on the Shared Responsibility Model for the Cloud.

The Master Node, aka the Control Plane, serves as the brain that keeps the complex system of nodes running. It serves as the main node(s) of your cluster as it manages the worker nodes.

Figure 1 – A diagram of a Kubernetes Architecture and its main components

There are attackers and bots constantly searching the internet for exposed Kube API Servers. It is critical that the kube-apiserver is not left publicly exposed otherwise it may be an easy target for those bots. Although the default setting on unmanaged clusters is that the API server is not exposed, that’s not the case for managed Kubernetes services such as EKS (Elastic Kubernetes Service). Exposed API servers are still the main entry point for attackers to compromise a K8s cluster. The basic recommendation for securing API servers is to only allow engineers to access the API via the internal network or corporate VPNs and even then, you should limit the access for specific users and machines.

Using RBAC

RBAC (Role Based Access Control) authorization is the next step in creating a more secure cluster now that access to the API server is restricted. RBAC allows you to configure who has access to what in a cluster. It also allows you to restrict users from accessing the kube-system namespace, which houses all the control plane pods (see Figure 1 above).

When using RBAC authorization, there are four kinds of API objects that you can use:

• Role. Contains rules that represent a set of permissions within a namespace.
• RoleBinding. Grants the permissions of a Role to one or more users.
• ClusterRole. Contains rules that represent a set of permissions, but it is not tied to a namespace, and it will be applied on the cluster level.
• ClusterRoleBinding. Grants permissions for a ClusterRole to a set of users.

The permissions for the Role and the ClusterRole are usually formed with a combination of a verb and a noun, which represents an object and a resource. Some examples include:

• Get Services
• List Pods
• Watch Secrets

Figure 2 – How users are related to Roles via the RoleBindings (same thing for ClusterRoles and ClusterRoleBindings)

What is ETCD?

The ETCD is the main data storage location for your cluster, which means all the cluster objects are saved here. Leaving the etcd exposed can potentially leak critical data. Unfortunately, etcd misconfiguration remains rampant as we’ve seen more than 2,300 exposed etcd services on Shodan in December alone.

Figure 3 – Total number of etcd instances currently exposed to the internet. Shodan query used: etcd port:”2379″ (December 1st, 2020)

The same security principles should be applied to etcd as with any data storage system. Encryption should be implemented in transit and at rest. The latest default installation, using kubeadm, sets up the proper keys and certificates with TLS encryption for etcd. However, if an attacker somehow bypasses the API server and can manipulate objects directly into etcd, it would be the same as having full access to the entire cluster as they would have the ability to change all the configurations of the cluster and the controller manager would reflect those changes in the actual cluster itself.

Any part of a cluster could be abused if accessed by an attacker. let’s look at some specific steps to ensure your deployments are secure.

Network policies

Network policies can help address the issue of open pod communication. By default, the cluster network allows all pods on the same cluster to communicate with each other, including pods from different namespaces. A network policy specifies how groups of pods can communicate with each other and with other network endpoints. NetworkPolicy API resources use labels to select pods and define rules that specify what type of traffic is allowed for the selected pods. These policies can help you restrict access between pods or namespaces. All the access can be configured via labels in YAML files, allowing you to block pods from accessing other pods on the kube-system namespace, for example.

The worker nodes

If the control plane is the brain, worker nodes are the muscle of a cluster. They run and control all the pods and containers in your cluster. While worker nodes are not required, it is not recommended to run and control all pods on the same node as the control plane, so you should have at least one.

The main components of a worker node are: the kubelet, the kube-proxy and the container runtime. The kubelet is the agent that runs on each node on your cluster to make sure all containers are running in a pod, it is responsible for managing the container runtime executing containers when necessary and collecting execution information. The kube-proxy manages network communication allowing different containers to communicate with each other in addition to being responsible for external requests. Even the Master Node has a kubelet and a kube-proxy, although they usually don’t show that on the architecture diagrams. The container runtime, is actually the component that creates and executes the containers themselves Also, the default container runtime for a Kubernetes cluster, up until v1.19.5, was Docker, but that changed on December 8th 2020, when v1.20.0 was released, causing major panic to everyone, since it was announced that Dockershim, the CRI (Container Runtime Interface) shim for Docker, was being deprecated. Although it is not the goal of this post, you do not need to panic!

Really, don’t go crazy about it! Not a lot will change with this, but if you want to know more and be prepared, please read this post by the Kubernetes community: Don’t Panic: Kubernetes and Docker. Other CRIs that you could use with your Kubernetes cluster are containerd or CRI-O, which are both projects under the CNCF. I highly recommend that you read the post above (after this one, of course!) if you want to know more details about these changes and what will happen in the future.

There are three main steps to ensure the minimum level of security for the pods themselves.

• Limit resources to ensure all pods can perform as needed. If one pod starts consuming all the computing resources available, it could cause a Denial of Service (DoS) on the node. ResourceQuotas are the solution, allowing you to set hard and soft resource limits in a namespace.
• Create and apply a Security Context to define privilege and access control permissions for a pod or container. A few to always include are:
  • AllowPrivilegeEscalation – controls whether a process can gain more privileges than its parent process. This should be set to false.
  • ReadOnlyRootFileSystem – defines whether the container has a read-only root filesystem or not. The default setting is false, but we recommend that you set it to true.
  • RunAsNonRoot – indicates if the container must run as a non-root user and should be set to true. By doing this, in any event that the container tries to run as a root user UID 0 (user ID), the kubelet will validate it and fail to start the container.
• Use a Linux kernel securityfeature like Seccomp, AppArmor or SELinux. These can be set up via the Security Context as well.

Audit Logs

Audit logs are an important part of a cluster, as they can record all the requests made to the Kube API Server. Audit logs are disabled by default since they increase memory consumption; however, we highly recommend you enable them before putting your cluster in production. Audit logs will help you detect any security issues and will help your developers with debugging and troubleshooting.

Enabling logs and having a proper policy can increase the likelihood of identifying a misconfiguration before a breach occurs IF (yes, big if) a person or system is tasked with analyzing them to look for suspicious activity. There is no point in having logs if no one or no system is looking and analyzing those.

Don’t forget the basics!

All the above configurations and tweaks are important for maintaining a secure cluster. But amidst all this, don’t forget some basic rules for day-to-day work with Kubernetes:

1. Update your environment version as early and as often as possible using this command: kubectl update.   Obviously:  Please apply this in a test environment before applying in production.
2. Don’t use the admin user for your daily work, the admin should only be used by CI/CD tools.
3. If you can, use managed services such as AKS, EKS and GKE. They usually have better defaults for your security posture and the costs for maintaining the control plane can be extremely low. But also, be aware of their security defaults!
4. CIS Kubernetes Benchmark document for more security best practices. They also have specific benchmarks for EKS, GKE and OKE (from Oracle Cloud).

For more information, recommendations, and to learn more about Kubernetes and Kubernetes Security, please visit this Awesome Kubernetes Security list on GitHub that I created, which contains blogs, books, articles, presentations, videos, training, and tools about attacking and defending your clusters.