K8s Best Security Practices

Here would be a good point to emphasise the "non-human account" aspect of a ServiceAccount and clear up any confusion regarding human users being authenticated/authorised into a cluster. Here is some information regarding ServiceAccounts and Users.  

Essentially, user access is usually handled using some kind of account management solution. Kubernetes does have one built-in, but this is generally integrated with another user management system such as LDAP or AD. Non-human access to the cluster and its resources is handled using Kubernetes ServiceAccounts.

ServiceAccounts are very easy to define using kubectl. You can create a ServiceAccount on your cluster using the following command.
Note: serviceaccount can be abbreviated as sa

If you wanted this ServiceAccount to be associated with a specific pod, you would define that in the pod/deployment configuration YAML like so:

Why RBAC?

Role: What can this role do, and to what resource(s)? Permissions are additive, meaning there are no deny permissions. Think of this as allowlisting for API access. You will provide a verb (what can this role do), e.g. "get" or "delete", and a resource (and to what), e.g. "pods" or "secrets". Note that sub-resources can also be defined here using a / after the resource name, e.g. pods/log. This will be visualised with a configuration example soon. Roles are namespaces, meaning they can allow actions within a given namespace, not the entire cluster.

RoleBinding: The RoleBinding object does what it says on the tin. It binds a Role (defined above) to a ServiceAccount or User. As Roles are namespaced, the role mentioned in the RoleBinding configuration YAML will need to exist in the same namespace as the RoleBinding.

ClusterRole: A ClusterRole is functionally the same as a Role, except roles are granted on a (non-namespaced) cluster level. This would be used if access is needed to cluster-level resources like nodes or resources across multiple namespaces. Again, only if this is NEEDED, with least privilege in mind.

ClusterRoleBinding: A ClusterRoleBinding is functionally the same as a RoleBinding except it binds a ClusterRole to a ServiceAccount or user.  

Now that we've established the different kinds of RBAC objects, let's give some more context as to where these would be used. As mentioned, roles can be bound to either ServiceAccounts or Users. ServiceAccounts will be used if we are granting authorisation to an application, e.g. an application running in a pod within the cluster or a 3rd party external service like Istio. Meanwhile, users will be used if we are granting authorisation to, you guessed it, users! Let's now look at RBAC in the context of both of these.

Let's consider the example mentioned earlier. Let's say we have a pod that is supposed to check the running status of pods in the "example-namespace" namespace. The pod identifies as ExampleServiceAccount. The pod definition YAML may look something like this: 

apiVersion: v1
kind: Pod
metadata:
  name: up-checker
  namespace: example-namespace
spec:
  serviceAccountName: ExampleServiceAccount
  containers:
  - name: kubectl
    image: bitnami/kubectl:latest
    command:
  # commands for pod status check logic would go here

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: up-checker-role
  namespace: example-namespace
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: up-checker-role-binding
  namespace: example-namespace
subjects:
- kind: ServiceAccount
  name: ExampleServiceAccount
  namespace: example-namespace
roleRef:
  kind: Role
  name: up-checker-role
  apiGroup: rbac.authorization.k8s.io

Okay, so if you're going to secure anything and follow best practices, it's fundamental that you understand how exactly it is being accessed. Between this room and the prerequisite Cluster Hardening room, you have now accrued enough knowledge to understand how a Kubernetes cluster is accessed. This task will collect the knowledge covered so far and use it to explain methods of accessing a Kubernetes cluster and the API request stages involved. 

Those are the different methods that can be used to access the Kubernetes API/Cluster. From this point, the conversation turns to how we control access to the cluster. You'll be pleased to know that much of what you've learned thus far has been methods of controlling access to a cluster. Let's contextualise them by looking at Kubernetes API request stages. When you communicate with the Kubernetes API, you will be asking to do something, whether that be getting some logs, creating a pod or deleting a service. When you perform one of these actions, you will be sending a request to the API. This request will go through multiple stages before being actioned. Let's take a look at what those stages are: 

Authenticating an API request can be done in multiple ways. These include methods covered in the Cluster Hardening room (covered in Tasks 4 and 5), such as setting up client certificate authentication or using API bearer tokens. Authentication can also be done using other auth plugins such as Authenticating Proxy or HTTP basic auth. Another authentication method would be ServiceAccount tokens, which are used to authenticate a Kubernetes ServiceAccount. Multiple authentication methods can be used, and it is often the case that at least two would be used (ServiceAccount Tokens and another auth plugin for user authentication). A request passes this stage only when it has the right attributes needed to confirm the requestee identity.  

Once the requester's identity has been authenticated, it needs to be determined if this user/ServiceAccount is allowed to make this request (e.g., access this resource). For example, does user example-user have sufficient permissions to delete pod example-pod. The request needs to be authorised. Kubernetes has four authorisation modes. One commonly used and the most important for our current context (Kubernetes best security practices) is RBAC. There are three other authorisation modes:

Using one of the above methods, a request is checked to see if the authenticated user is authorised to access the resources/perform the action requested. Before the request is accepted, there is one more stage.

After a request has been authenticated and authorised the next API request stage is Admission Controllers. Covered in the Cluster Hardening room (covered in Task 6), Admission Controllers are checks done against an API request. They can be validating (is this action permitted, does it breach a check) or Mutating (e.g. in a request to create a pod, the pod's name might need to be changed to match a custom naming convention). As well as built-in admission controllers, you may remember it is possible to define custom checks. These admission controllers are triggered after the authentication and authorisation stages, and only after all admission controller checks have been passed will this request persist.

We have followed an API Request through all of its stages. When asked how to restrict access to a Kubernetes cluster, you should now be able to elaborate more than just "harden your cluster" and discuss how a cluster is accessed and how access can be restricted at each stage. Great work! Your DevSecOps skills are growing with every task.  

With the knowledge gained so far, your understanding of Kubernetes is growing. A more in-depth look at RBAC and Kubernetes ServiceAccounts has allowed for a deeper understanding of cluster access. This task is going to be dedicated to shouting out some other notable Kubernetes Best Security Practices, a collection of DOs when it comes to K8s security. Without further ado lets get started!

Let's start by taking things back a step. Why do we want Kubernetes in the first place? To deploy our apps, right? Well, those apps are made using images. The app image will contain everything needed for the app to run on the container, but what else does it contain? Image scanning is one way to find out! Images are generally made up of a series of commands (or Layers) that create directories/users , copy/run scripts, and install tools/libraries. All of these have security implications: 

These security implications are not to be taken lightly, as a vulnerability in your application image could be just what an attacker is looking for. An attacker could exploit this vulnerability and use it to break out the container the application is running in and access the host, wreaking havoc on the cluster and gaining access to critical cluster information such as kubelet API authentications certs/tokens. This would be a massive security incident. This is why image scanning should always be introduced into a CI/CD pipeline as a best security practice.  

Image scanning tools (like Snyk) scan the image being used against a database of vulnerabilities and look for things like hard-coded secrets. With an image scanning step in place, you can rest easy knowing that what is being deployed into your cluster is secure. 

Keeping up to date is a general best security practice, not one specific to Kubernetes. Running an old version of Kubernetes comes with all the same security concerns; if an attacker finds a cluster running an older version node, they may know a few vulnerabilities that have since been patched. And, of course, updating a Kubernetes cluster comes with the same Stable vs Latest considerations. Stable releases have been thoroughly and fully tested (and so are generally favoured from a security perspective). However, the latest version (which will have been internally tested) will include patches for issues in the stable version. Ultimately, as long as you are at least using the stable version, you are following the best k8s security practices.

Note: When choosing the version to upgrade to (the latest stable version, for example), you cannot skip minor versions. For example, you cannot upgrade your cluster from version 1.21 -> 1.23 as you would be skipping minor version 1.22. So if the latest stable version were 1.23, you would have to do 1.21 -> 1.22, then 1.22 -> 1.23. 

Upgrading a cluster usually falls to a DevSecOps engineer, or at the very least, a DevSecOps engineer will work alongside a DevOps engineer to ensure this is done. Due to the nature of versioning, there are constantly new versions being released, and keeping the cluster up to date can become quite a task. Because of this, teams sometimes overlook the upgrade process and fall behind on a few versions. Following best k8s security practices, your team will regularly assign you tickets to get the cluster upgraded. This will usually have to be done in the following order: 

Unless there is an operational need for it (which there almost never is), a container should be run as a non-root user. This is because if, like previously covered, your image has vulnerabilities in it and an attacker gains access to the container, you're making their life a lot easier if that container has root privileges.

This can be done when defining the image. However, certain specifications in your pod configuration YAML can override this. As a DevSecOps engineer, you'll likely be reviewing your fair share of resource configurations. If you see in the spec, under “securityContext” values like runAsUser: <UID> (where the UID has root privileges) or allowPrivilegeEscalation: true, your alarm bells should be ringing. You should verify this is not required (which again, it almost never will be) and change the pod spec to look more like this:

As you may remember, etcd is Kubernetes' key/value store where cluster data is kept. You hear "cluster data is kept", an attacker hears "goldmine", and they're right. Changes made in the cluster (such as a pod being deleted) are then reflected in the etcd store. However, it goes both ways; if a change were made in the etcd (such as a new pod entry), that change would be reflected in the cluster. Imagine now an attacker gaining access to the etcd. Suddenly, they're not so concerned with finding ways to gain API access to the cluster, but they've found a way to bypass the API altogether. An attacker gaining full access to the etcd is one of the worst things that could happen in a Kubernetes cluster, as it is essentially unlimited access.