Kubernetes isn’t even six years old, but it’s already everyone’s favorite container orchestration program. Datadog, a cloud and infrastructure monitoring company, found Kubernetes dominating the container space: “Roughly 45% of Datadog customers running containers use Kubernetes.” Other container orchestration programs, such as Marathon and Docker swarm mode, have dropped away.
Kubernetes is even more popular on the public cloud. For example, by Datadog’s count, “Roughly 80 percent of Datadog customers running containers in Azure are now using Kubernetes”. While on Amazon Web Services (AWS), which was slow to warm off to Kubernetes, Datadog found its popularity had doubled to 45% over the past two years.
With popularity comes danger. As Windows users know to their sorrow, the more popular a program is, the more likely it is to be attacked. The same is true for Kubernetes. If you’re using it, you must secure it.
That’s not easy.
Kubernetes’ parent organization, the Cloud Native Computing Foundation (CNCF), recently ordered a security audit from Trail of Bits and Atredis Partners. Trail of Bits reported that many security problems exist because Kubernetes’ “configuration and deployment of Kubernetes [is] non-trivial, with certain components having confusing default settings, missing operational controls, and implicitly designed security controls.”
Really, really not easy.
Still, we have to try. Here are my top five ways to lock down Kubernetes.
1. Make sure the containers themselves are safe
It doesn’t do any good to secure Kubernetes if the containers are fouled up. In 2019 Snyk, an open-source security company, analyzed the 10 most popular Docker images. They found that all the images included vulnerable system libraries.
As Dirk Hohndel, VMware‘s VP and chief open source officer said at the 2019 Open Source Leadership Summit, “Container packaging format is similar to .exe in Windows and .dmg in macOS, where you basically ship a whole file system with all of your dependencies included. Since you are now including those dependencies [in containers], you have to worry about those binaries—where they come from, how they were produced, and their corresponding sources.”
In short, you must make certain your containers’ contents are trustworthy before deploying them for Kubernetes to manage or anywhere else. Otherwise, you run into one of those oldest of computer problems: Garbage In/Garbage Out (GIGO). Therefore, you have no choice but to check every production container for potential security woes.
Yes, it’s a pain. No one ever said security was easy.
2. Locking down your container’s Linux kernels
It only gets harder from here. Besides making sure your containers are secure, since all your containers run off a single Linux kernel, it makes sense to ensure that that kernel is as secure as possible. And, that means locking it down with AppArmor or SELinux.
These limit user control, programs, file access, and system services. They do this as Linux Security Modules, which impose a mandatory access control (MAC) architecture on Linux. This is a fundamentally different security approach (restrict everything unless explicitly permitted) to Linux’s built-in security model (permit everything unless explicitly forbidden).
Because of this, many–and I mean many–system administrators have trouble setting either up properly. Worse still, if you get it wrong, it’s not just a matter of leaving a security window open, which should have been closed. No, if you get these wrong, your Linux system can be frozen and you’ll be left having to set it up all over again. Or, you may think it’s set up correctly, and your containers may be as wide open to attacks as ever.
Getting it right takes time and expertise. You may also have trouble getting applications to run on the AppArmor or SELinux protected kernels. Far too many programs take security liberties with the underlying operating system. These protected kernels won’t allow this.
If either of these prove to be too much trouble–no one has ever pretended that deploying either correctly is easy–you can get some protection by blocking the kernel from automatically loading kernel modules. For example, since even unprivileged processes can force some network-protocol-related kernel modules to be loaded, just by creating a particular network socket, you can add rules to block questionable modules. You do this with the forbidden module configuration file:
/etc/modprobe.d/kubernetes-blacklist.conf
Within it, you place instructions such as:
# SCTP is not used in most Kubernetes clusters, and has also had
# vulnerabilities in the past.
blacklist sctp
Still, it’s better to bite the bullet and install AppArmor or SELinux.
3. Use Role-Based Access Control (RBAC)
Starting with Kubernetes 1.8, you can use RBAC to control what users can do. That’s important because trying to wrangle users running on dozens of instances and clusters, all woven together to deliver a service is a horror show. With RBAC, a zero-trust security approach, users and components, like apps and nodes, get no more permission than they absolutely need to accomplish any given job.
The RBAC works at the application programming interface (API) level. That way, RBAC rules lock down users even when the authorizer itself isn’t in use. RBAC also stops users from giving themselves more privileges by editing roles or role bindings.
It also blocks outsiders by default. RBAC policies grant scoped permissions to control-plane components, nodes, and controllers. But, and this is important, it grants no permissions to service accounts outside the kube-system namespace.
To make it easier to manage Kubernetes RBAC comes with predefined roles. This way developers, operators, and cluster administrators get just the permissions they require without any they don’t need.
In this system, the cluster-admin role is the equivalent to Unix and Linux’s superuser. Cluster administrators have the ability to create, edit, or delete just about any cluster resource. Needless to say, cluster administrator role access must be carefully protected, which leads me to my next point.
4. The hard work of keeping secrets, secret
Kubernetes Secrets objects contain sensitive elements such as passwords, OAuth tokens, or ssh keys. In short, they’re the keys to the Kubernetes kingdom. You must protect them.
Kubernetes automatically creates secrets for accessing the API and modifies your Pods to use them. Your secrets, though, such as the username and password a Pod needs to access a database, are under your control and your protection.
Simple enough right? Unfortunately, Kubernetes secrets come with numerous built-in security weaknesses. The official list is downright frightening.
- In the API server, secret data is stored in etcd, Kubernetes’ default distributed key-value store. By default, etcd data is not encrypted and, therefore, neither are your secrets. Ow! So, you must enable encryption at rest and restrict access to etcd to admin users.
- Many users keep their secrets in a JavaScript Object Notation (JSON) or YAML There, the secret data is encoded–not encrypted–as base64. That means if you share the file or check it into a source repository, you’re just asking for someone to read all your secrets.
- Once an application has used a secret, the app can’t be allowed to log it or transmit it to an untrusted party.
- A user who can create a Pod that uses a secret can also see the value of that secret. Even if the API server policy does not allow that user to read it, the user could run a Pod, which exposes the secret.
- Currently, anyone with root permission on any node can read any secret from the API server, by impersonating the kubelet. This is actually a feature. The idea is that by only sharing secrets with nodes, which require them you limit the damage from a root exploit to a single node.
It’s an ugly situation. You can mitigate it by encrypting secrets whenever you can. Whenever possible you should also keep the secret separate from an image or pod. One way to do this is to split processes into separate containers. For example, you can divide an application into a front-end container and a backend container. The backend has the private key secret and responds to signing requests from the front-end.
Unless you’re both a security and Kubernetes wizard you must turn to a third-party secret tool to protect your secrets. These include AWS Secrets Manager, Google Cloud Platform KMS, and Azure Key Vault for public clouds. And, programs such as Hashicorp Vault, CyberArk/Conjur, Confidant. and Aqua Security Kubernetes Security for the Enterprise.
5. Keeping your network safe
As you can figure out from above, it’s very dangerous to allow access to etcd. As Andrew Martin, co-founder of ControlPlane, a Kubernetes security business wrote in Kubernetes’ own documentation, “Write access to the API server’s etcd is equivalent to gaining root on the entire cluster, and even read access can be used to escalate privileges fairly easily.”
Therefore, Martin suggests, “etcd should be configured with peer and client TLS certificates, and deployed on dedicated nodes. To mitigate against private keys being stolen and used from worker nodes, the cluster can also be firewalled to the API server.” I’d change “can also” to “must.”
It’s not just etcd that requires network protection. Since Kubernetes is entirely API driven, all internal cluster communications are encrypted by default with TLS.
But, that’s not enough. Kubernetes pods accept traffic from any source by default. Let me repeat that: “Any source.” Pods are wide open for network connections, and that’s not safe. It’s up to you to define network policies, which safely define how pods communicate within a cluster and with external resources.
You do that by adding a networking plugin, which supports a Network Policy controller. Without such a controller, any network policies you set won’t take effect. Unfortunately, Kubernetes doesn’t have a native or even a preferred Network Policy controller. Instead, you must use third-party plugins as Calico, Cilium, Kube-router, Romana, or Weave Net. For what it’s worth, my favorite is Calico.
I recommend you start with a “default-deny-all” network policy. That way only connections you subsequently explicitly permit with other network policy rules will be allowed. Since network policies are set in a namespace, it’s up to you to set up a network policy for each namespace.
Once set up, you can start allowing the appropriate network access rules. For a good starting point look to these Kubernetes Network Policy Recipes.
Securing Kubernetes is a major job.
As I said at the beginning, securing Kubernetes really isn’t easy. By now, I’m sure you’ll agree.
Kubernetes was designed from the get-go to be easy to deploy. Unfortunately, that also means it’s insecure by design. That means it’s up to you–yes lucky you–to add in the security to make your Kubernetes clusters production worthy.
I’ve only touched on the most important issues you’ll need to address with Kubernetes security. There are many others. But, I do hope I’ve given you enough to realize just how big the job is. And, why you must start securing Kubernetes now before a hacker rudely reminds you that you’ve left your company’s crown jewels open to attack.
