Surviving the NGINX Ingress CVE-2021-25742 with Checkov

CVE-2021-25742 was announced on October 21st, with some of the top experts in Kubernetes security engaging immediately in discussions around its seriousness in tandem with policy-as-code solutions to prevent exploitation. In short, the vulnerability was related to the fact that users with limited access to a Kubernetes cluster but with the ability to create an Ingress object based on the NGINX Ingress Controller had the ability to elevate privilege and access full cluster secrets.

It’s one thing to identify a problem, but it’s another to identify the specifics of the abuse cases succinctly enough to create security policies that simultaneously allow DevOps teams to continue to operate without disruption while also blocking the exploit. So let’s explore how at Bridgecrew, we investigated the exploit and subsequently created the rules for Checkov.

About the CVE

The NGINX Ingress handles incoming requests from outside of the k8s cluster and routes (and load balances) those requests to application services inside the cluster based on specific Ingress rules. Sounds simple enough!

From a Kubernetes perspective, a user with broad cluster admin rights will deploy the Kubernetes Ingress Controller, creating the ingress deployment (pod, role, service account, service, etc.) and establishing an IngressClass type.

From there, applications will have ingress rules created and will create Kubernetes objects of type Ingress, which, as an abstraction, can be viewed as an object based on the previously defined IngressClass with set rules, some of which can override the base IngressClass. The reality is that the Ingress object does not have a unique pod or resource presence. Instead, it is all handled through the IngressClass pod.

Source

Reviewing the RBAC for the default deployment of the NGINX Ingress requires considerable visibility and access to the cluster.

Glancing at the ClusterRole in the deployment (below), we can see that the rules for the service account associated with the pod provides access to list all nodes, namespaces, secrets, pods, etc., to discover and correct route traffic. Thus, gaining access to a service account with these RBAC rules is an attacker’s candy land.

rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
      - namespaces
    verbs:
      - list
      - watch

It is possible within the nginx.conf which powers the NGINX Ingress controller to create stock rules enclosed within server blocks. For example:

server {
		location /healthz {
			return 200;
		}
}

In this instance, the server configuration uses the location directive to create a health check, often leveraged by liveness probes to ensure our NGINX is alive and well (see article on NGINX security).

There is also a scripting language called “Lua” which is used for more complex configuration responses in the nginx.conf. In the same default nginx.conf section designed for health checks you’ll find this simple example:

server {
		location / {
			content_by_lua_block {
				ngx.exit(ngx.HTTP_NOT_FOUND)
			}
		} 
}

The above indicates a default HTTP_NOT_FOUND response for the path location /. The nginx.conf default is complicated and out of scope for this article, but this should set the scene to describe the problem when we introduce snippets.

What is a “snippet”?

Snippets are enabled by the line in the NGINX Ingress deployment ConfigMap.

data:
  allow-snippet-annotations: 'true'

Since snippets are commonly used, this default is warranted.

A snippet has four flavours—auth, configuration, ModSecurity, and server—but we’re going to focus specifically on the last one in the list, server snippets.

Each snippet type can be added as an “annotation” in the metadata section of a Kubernetes ingress object and will be interpreted as a form of override on the default NGINX configuration we looked at earlier—specifically in the case of a server snippet, the server {} block. Given our problem that it’s primarily focusing on a server snippet, this specific annotation can overwrite the server block in the nginx.conf, changing the behaviour of the ingress.

kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: proxy_ssl_server_name on

In this simple case above, overriding the default “off” provides a more secure result. This is a positive case for server snippets. We will see shortly, however, how this capability can be abused. Security rules created to detect misuse need to be crafted to find malicious intent without hindering normal operational behaviour.

The crux of the problem with this CVE is a combination of two things:

  1. The required level of access that the Ingress Controller needs to carry out its core duties.
  2. The Ingress object providing the capability for somebody with reduced cluster access to be able to override the Ingress Controller behaviour, resulting in remote code execution and/or interrogation of the filesystem within its pod.

Remote code execution is powerful, so it doesn’t take much imagination to extend the possibilities of owning both the cluster and the host nodes once this has been achieved!

Like data breach-related opportunities, this is a system-level problem where no single piece of the puzzle is necessarily the root cause.

How do we exploit it?

During the discussion of the CVE on GitHub, the first finger was pointed at the fact that, within a server snippet annotation, one could execute “Lua” script code. Lua does have a format of “exec” called io.popen that can execute incredibly dangerous system-level commands.

If we recall the rather vast capabilities of the service account associated with the pod, we can leverage the fact that this service account will be mounted into the pod at /var/run/secrets/kubernetes.io/serviceaccount.

With Lua code, we can simply run the equivalent of a linux cat on the /var/run/secrets/kubernetes.io/serviceaccount/token and the /var/run/secrets/kubernetes.io/serviceaccount/ca.crt. Presto! We can combine these via a simple curl command to access all of the bounties the cluster has to offer (specifically, the secrets would be handy)! It also doesn’t help that the default NGINX Ingress Controller container already has curl installed on it.

However, there is one more rather subtle and less programmatic attack vector. There is an NGINX server directive called alias that can be added as a perfectly valid snippet.

The directive alias defines a replacement for the specified location.

For example, take the following configuration:

location /hack {

    alias /var/run/secrets/kubernetes.io/serviceaccount/token;

}

On request of “https://mywebsite.com/hack,” the file /var/run/secrets/
kubernetes.io/serviceaccount/token
will be sent. Uh oh! That was much easier than learning Lua, wasn’t it!

To recap:

  1. An NGINX Ingress Controller is deployed into the cluster of IngressClass : nginx by cluster administrators.
    1. This is created in the namespace “ingress-nginx”
    2. The kubeconfig for the admin team is cluster-wide
    3. My application team creates their application, service, and ingress. The Ingress uses class nginx
    4. This is created in the namespace “developer”
    5. The kubeconfig for the application team is locked down only to the developer namespace
  2. The ingress containers’ snippets add routes which can:
    1. Extract the token from the Ingress pod service account (running in a different namespace)
    2. Extract the ca.crt content from the Ingress pod service account (running in a different namespace)
  3. You can then run: curl --header "Authorization: Bearer <TOKEN>" --cacert <CACRT FILE> https:/k8s/api/v1/secrets
    1. NOTE: The k8s API server and port is available within the pod or within the existing kubeconfig

Before coming up with policy-as-code solutions for Checkov, we started looking into how snippets are used in the real world.

Approaching the solution with infrastructure as code (IaC)

Searching GitHub for evidence of server-snippet usage shows relatively limited adoption of this feature to mutate the Ingress behaviour:

For the most part, snippets were not that popular but were popular enough that a solution of blocking them entirely wouldn’t be sufficient. Mainly they were used for minor but essential server configuration tweaks like changing the client_header_buffer_size and for relatively straightforward “location” based operations. Those operations did not include the “alias” directive (which definitely sounds like a 1960s Michael Caine spy film).

Before setting out on our new rules, we checked to see what other solutions were doing. An OPA-based solution quickly offered a regex-based check that looked a bit like this:

 badInjectionPatterns = "\\blua_|_lua\\b|_lua_|\\bkubernetes\\.io\\b"

It looks for the patterns “lua_” or “_lua” or “_lua_” or a reference to “kubernetes.io,” which is an extremely suspicious reference to the name of the directory where service account details are mounted. This was already a great start, so incorporating that into the solution was a definite yes. It does not, however, catch all possible attack routes.

We found that some solutions focused on the NGINX Ingress deployment itself and checked for the default statement allow-snippet-annotations: 'true'.

To us, this definitely seemed like a no-go, as it would trigger on every NGINX Ingress and would be incredibly noisy. The same policy also checked for certain older versions of ingress, which did not allow for snippets to be turned off. No-go again, as that would be both noisy and disruptive.

Our Mitigation Solution

Given our decision that snippets are a necessary evil but that the exploit methods did not represent common usage patterns within snippets, we felt the best course of action was to be opinionated in our solution. 

We implemented three new policies within Checkov that range from making relatively broad generic security strokes to checking for specific capabilities within server snippets that create clear and present danger. Let’s traverse this in order of priority from low to high.

CKV_K8S_153 – Prevent All NGINX Ingress annotation snippets.

This policy checks for the use of snippets and is a broad statement. However, it’s worth noting that this policy can be disabled if snippets are required in your organisation.

CKV_K8S_152 – Prevent NGINX Ingress annotation snippets which contain Lua code execution.

This only checks for snippets that are leveraging Lua script code within them. As stated above, we found precious few examples where this was needed or used in practice, and our opinion is that allowing full Lua scripting capability is straight-up dangerous.

CKV_K8S_154 – Prevent NGINX Ingress annotation snippets which contain alias statements.

The use of alias was the only other exploit path we found no practical use for in a snippet but was highly dangerous to allow. We didn’t find any other policies which looked for this exploit specifically.

Is there a fix?

There is! If you consider a word blocklist a fix.

Partway through writing this blog, NGINX Ingress released version 1.0.5 which added "load_module,lua_package,_by_lua,location,root,proxy_pass,serviceaccount,{,},',\" as defaults to the annotation-value-word-blocklist.

While this does an excellent job of blocking the exploits and much more, it’s the “much more” that may be a problem. An immediate upgrade to 1.0.5 will also break some ingress deployments with reasonable usage of snippets. We can only assume that there may have been more exploits than we found, resulting in a heavy response. The reality is that many users will still be using 1.0.4 or earlier (hence the creation of policies checking for quite old versions), and many keen for a fix will struggle to upgrade.

A final statement on mitigation

The ideal mitigation that doesn’t involve a security automation tool or a blocklist of any kind is to namespace your Ingress Controllers rather than allowing a single Ingress Controller to manage the entire cluster. Creating a form of multi-tenant Ingress with multiple controllers (one per application/namespace) is a far safer option even if these exploits are available and a codified security tool like Checkov isn’t in place to backstop your security efforts.

A final statement on risk

You might look at all of this and think, “But Steve, I’ve already got access to the cluster if I can create Ingress rules,” and you would be correct. This isn’t an opportunity for an initial foothold like the infamous Apache Struts CVE, but a serious opportunity for expansion should even limited access to the cluster be achieved. Gaining access to all cluster secrets can be a game-over scenario for not just that cluster but also potentially for your infrastructure and supply chain in general. Given the nature of some of the supply chain attacks we’ve seen in the past year, eliminating or mitigating all forms of attack and lateral movement is becoming an essential part of modern cloud-native security.