Yo! Long time no see since my last post. I hope you guys are all doing well! Sorry for the silence, I’ve had tons of stuff to handle lately, mostly focused on enhancing the W’xOps IDP, especially after figuring out what needed my top priority: W’xOps Core. Today, while testing, applying, and implementing several AuthN and AuthZ features inside a Kubernetes cluster, I wanted to share a few notes on what you can leverage to secure your Kubernetes cluster from a Platform Engineer or Cluster-Admin perspective. Let’s delve deeper into it!
AuthN, AuthZ and Admission in Kubernetes, Interesting but Insane
The security and system weakness are always the stuff that everyone look forward and interest, espeically Kubernetes. We can’t go far for success without the backbone not good enough, that always correct for any system, and Kubernetes take them as the base for build their system more reliable, and security as possible. From Kubernetes v1.34.x, The version I am working on now, to latest v1.37.x, I think Kubernetes already introduce a lot of concepts to let you access to Kubernetes Cluster as protected, including
Authentication (AuthN)
First of all, we will talk a walkthrough into Authentication (AuthN) in Kubernetes. But before hitting to my perspective, you can visit these links to explore more
As you can see, there are pretty much solutions here, and each of them being introduced to tackle several demands, I don’t want to tell which one is the best, because it depends on your expectation, so let’s explore each of them from simple to complex, and understand why you should choose, or not it for your Kubernetes Cluster.
Bootstrap Token
What is the bootstrap tokens ?
Bootstrap tokens are a simple bearer token that is meant to be used when creating new clusters or joining new nodes to an existing cluster. It was built to support kubeadm, but can be used in other contexts for users that wish to start clusters without kubeadm. It is also built to work, via RBAC policy, with the kubelet TLS Bootstrapping system.
This feature is becoming stable at Kubernetes v1.18, and for my usage and entire time to worked with Kubernetes, I don’t see this feature more, both managed and self-hosted Kubernetes Cluster. Because that one mostly built for kubeadm with flag API server configuration to run this feature
--enable-bootstrap-token-auth
Bootstrap Tokens are defined with a specific type (bootstrap.kubernetes.io/token) of secrets that lives in the kube-system namespace. These Secrets are then read by the Bootstrap Authenticator in the API Server. Itself have format abcdef.0123456789abcdef. More formally, they must match the regular expression [a-z0-9]{6}\.[a-z0-9]{16}.
Kubernetes supports client authentication using X509 Client certificates signed by a trusted cluster Certificate Authority (CA), coupled with authorization mechanisms (such as RBAC) to determine whether a user is permitted to perform specific operations on a resource.
This method is widely adopted and used by many Kubernetes distributions (e.g., RKE2, K3s) for cluster-admin configuration during cluster bootstrapping. X.509 Client Certificates provide a highly secure method for accessing the Kubernetes API through mutual TLS (mTLS) validation, enforcing continuous authentication until a certificate expires and is rejected by the cluster CA.
Because managing certificate expiration schedules is challenging and revoking issued client certificates is complex, a compromised certificate poses a significant security risk to your cluster. Consequently, certificate management requires extreme care: when generating or updating the cluster CA, you must properly handle CA rotation and certificate re-issuance across all existing control plane and cluster components.
Explore more method to requested signed certificate with CertificateSigningRequest in Kubernetes at
e.g: You can generate certificate for specfic user in RKE2 by these step below with openssl
Generate 2048-bit RSA private key
openssl genrsa -out developer.key 2048
Create Certificate Signing Request
Info
It is important to set the CN and O attributes of the CSR. CN is the name of the user, and O is the group that this user will belong to. You can refer to RBAC for standard groups.
cat <<EOF | kubectl apply -f -apiVersion: certificates.k8s.io/v1kind: CertificateSigningRequestmetadata: name: developerspec: # This is an encoded CSR. Change this to the base64-encoded contents of myuser.csr request: $(cat developer.csr | base64 | tr -d "\n") signerName: kubernetes.io/kube-apiserver-client expirationSeconds: 86400 # one day usages: - client authEOF
Approve the certificate request and retrieve the certificate
Generate the credential with certificate and key to ~/.kube/config file
Info
if you want to set the built config to another file, you can use --kubeconfig=new-path to build it, or that will merge into default ~/.kube/config
# Add the user identity into kubeconfig# if not `--embed-certs` that with absolute pathkubectl config set-credentials rke2-developer --client-key developer.key --client-certificate developer.crt --embed-certs=true# set cluster with server-cert when you init the cluster# rke2: /var/lib/rancher/rke2/server/tls/server-ca.crtkubectl config set-cluster rke2-cluster \ --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt \ --server=https://<RKE2_SERVER_IP>:6443 \ --embed-certs=true# set-context into kubeconfigkubectl config set-context rke2-developer --cluster=rke2-cluster --user=rke2-developer# use the context with kubeconfigkubectl config use-context rke2-developer# auth whoami to see who you arekubectl auth whoami
Service Account Token
What is Service Account Token ?
A service account is an automatically enabled authenticator that uses signed bearer tokens to verify requests.
ServiceAccounts are usually created automatically by the API server and are associated with pods running in the cluster through the ServiceAccountAdmission Controller. Bearer tokens are mounted into pods at well-known locations, and allow in-cluster clients to authenticate to the API server. ServiceAccounts may be explicitly associated with Pods using the serviceAccountName field within the spec of a Pod.
This authentication method is used lots in Kubernetes, especially gain permission for pod to workaround with another resources in Kubernetes. This JWT token was signed by Kubernetes Certificate Authority and contains information about the service account such as Kubernetes Namespace, Service Account Name, and Service Account ID. BTW, you can use it outside the cluster to get the authentication with Kubernetes API, with command kubectl create token, explore more at Kubernetes - Bound service account tokens
Set the context and use it like what we did before
kubectl config use-context token-cred-auth
Authenticating Proxy
These three way above is the standard way of Kubernetes, but instead, Kubernetes support various external integration, and Authentication Proxy is one of them
What is the Authentication Proxy ?
The API server can be configured to identify users from request header values, such as X-Remote-User. It is designed for use in combination with an authenticating proxy that sets these headers.
In this mode, the API receives requests from an intermediate proxy that manages authentication and sends the details associated with the user — including the username or RBAC group — via HTTP headers.
Risky
If you have a certificate authority (CA) that is also used in a different context, do not trust that certificate authority to identify authenticating proxy clients, unless you understand the risks and the mechanisms to protect that CA’s usage.
Since the Kubernetes API server has no way to validate the content of the headers, the only way to secure the platform against header spoofing attacks is to ensure their provenance. To prevent header spoofing attacks, the authenticating proxy must present a valid X.509 client certificate to the API server for validation against the configured CA.
With this method, I got the experience when I use Rancher as the Proxy to control and generate the authentication for Kubernetes API instead the directly one, explore more if you want to find the solution
Kubernetes webhook authentication is a mechanism to make an HTTP callout for verifying bearer tokens.
This Webhook Token is confuring via KubeAPI server option, including
--authentication-token-webhook-config-file a configuration file describing how to access the remote webhook service.
--authentication-token-webhook-cache-ttl how long to cache authentication decisions. Defaults to two minutes.
--authentication-token-webhook-version determines whether to use authentication.k8s.io/v1beta1 or authentication.k8s.io/v1TokenReview objects to send/receive information from the webhook. Defaults to v1beta1.
When a client attempts to authenticate with the API server using a bearer token, the authentication webhook POSTs a JSON-serialized TokenReview object containing the token to the remote service.
{ "apiVersion": "authentication.k8s.io/v1", "kind": "TokenReview", "spec": { # Opaque bearer token sent to the API server "token": "014fbff9a07c...", # Optional list of the audience identifiers for the server the token was presented to. # Audience-aware token authenticators (for example, OIDC token authenticators) # should verify the token was intended for at least one of the audiences in this list, # and return the intersection of this list and the valid audiences for the token in the response status. # This ensures the token is valid to authenticate to the server it was presented to. # If no audiences are provided, the token should be validated to authenticate to the Kubernetes API server. "audiences": ["https://myserver.example.com", "https://myserver.internal.example.com"] }
The remote service is expected to fill the status field of the request to indicate the success of the login. The response body’s spec field is ignored and may be omitted. The remote service must return a response using the same TokenReview API version that it received. A successful validation of the bearer token would return:
{ "apiVersion": "authentication.k8s.io/v1", "kind": "TokenReview", "status": { "authenticated": true, "user": { # Required "username": "janedoe@example.com", # Optional "uid": "42", # Optional group memberships "groups": ["developers", "qa"], # Optional additional information provided by the authenticator. # This should not contain confidential data, as it can be recorded in logs # or API objects, and is made available to admission webhooks. "extra": { "extrafield1": [ "extravalue1", "extravalue2" ] } }, # Optional list audience-aware token authenticators can return, # containing the audiences from the `spec.audiences` list for which the provided token was valid. # If this is omitted, the token is considered to be valid to authenticate to the Kubernetes API server. "audiences": ["https://myserver.example.com"] }}
Or unsuccessful
{ "apiVersion": "authentication.k8s.io/v1", "kind": "TokenReview", "status": { "authenticated": false, # Optionally include details about why authentication failed. # If no error is provided, the API will return a generic Unauthorized message. # The error field is ignored when authenticated=true. "error": "Credentials are expired" }}
Kubernetes has native support for JWT and for OpenID Connect (OIDC). The authenticator attempts to parse a raw ID token, verify it’s been signed by the configured issuer. For externally issued tokens, the public key to verify the signature is discovered from the issuer’s public endpoint using OIDC discovery.
What is OIDC (OpenID Connect) ?
OpenID Connect (OIDC) is an extension of the OAuth 2.0 authentication standard. The main change in the protocol compared to OAuth2 is an additional field returned with the access token called the ID Token. This token contains various information about the user; namely the email, user ID, groups, etc. The ID Token is encoded as a JSON Web Token (JWT) and signed by a third party with their private key. While the private key is private, the public key is available at a well-known URI — /.well-known/openid-configuration — and is used to validate the token.
To identify the user, the authenticator use id_token from the OAuth2 token response as a bearer token with this workflow, that same as the illustrator
With this model, you usually face off with couple of challenges and annoys, e.g
There is no browser to trigger the authentication process, that why you need the integration help you the hand, and authenticate to identity provider first to get response token back
The id_token can’t be revoked, so that will be short-lived, so it can make you look annoy when login again when token become expired (Careful configuration)
To authenticate to the Kubernetes dashboard, you must use the kubectl proxy command or a reverse proxy that injects the id_token.
Following this concept, you can discover lots of opensource, and cloud native support this mechanism, e.g:
The last one in my knowledge, and I think that truly useful compare with standard OIDC solution for let you interact with Kubernetes API Sever without touching into their default configuration. In couple of large clusters or enterprise situations, Impersonate Proxy is quite optimization and highly recommended solution for getting involved. With Impersonate mechanism supported by Kubernetes let us switch to specific user and get the Authorization to get connect to API Server.
What is User Impersonate ?
User impersonation is a method of allowing authenticated users to act as another user, group, or service account through HTTP headers. A user can act as another user through impersonation headers. These let requests manually override the user info a request authenticates as.
Impersonation requests first authenticate as the requesting user, then switch to the impersonated user info.
A user makes an API call with their credentials and impersonation headers.
API server authenticates the user.
API server ensures the authenticated users have impersonation privileges.
Request user info is replaced with impersonation values.
Request is evaluated, authorization acts on impersonated user info.
The following HTTP headers can be used to performing an impersonation request:
Impersonate-User: The username to act as.
Impersonate-Uid: A unique identifier that represents the user being impersonated. Optional. Requires “Impersonate-User”. Kubernetes does not impose any format requirements on this string.
Impersonate-Group: A group name to act as. Can be provided multiple times to set multiple groups. Optional. Requires “Impersonate-User”.
Impersonate-Extra-( extra name ): A dynamic header used to associate extra fields with the user. Optional. Requires “Impersonate-User”. In order to be preserved consistently, ( extra name ) must be lower-case, and any characters which aren’t legal in HTTP header labels MUST be utf8 and percent-encoded.
Impersonate-User: jane.doe@example.comImpersonate-Uid: 06f6ce97-e2c5-4ab8-7ba5-7654dd08d52bImpersonate-Extra-dn: cn=jane,ou=engineers,dc=example,dc=comImpersonate-Extra-acme.com%2Fproject: some-projectImpersonate-Extra-scopes: viewImpersonate-Extra-scopes: development
For my perspective, and experience, I think Impersonate Proxy is one of enterprise solution, that shoule be widely adopt to leverage their concept to flexible in authentication, except the boundaries by Managed Cluster with their own IAM Cloud model, if you want to keep same concept to login to various clusters in your system. You can find to several solution below
Pinniped: Batteries-included for Kubernetes authentication with architecture Supervisor (Hub) and **Concierge (Spoke)
OpenUnison: OpenUnison provides SSO and authentication for your Kubernetes clusters, no matter where they run or how your users need to authenticate.
teleport: The easiest, and most secure way to access and protect all of your infrastructure.
kube-oidc-proxy: Reverse proxy to authenticate to managed Kubernetes API servers via OIDC.
With W’xOps IDP, I’m using this mechanism for the core authentication into multi-cluster model with Identity providers DexiDP with several upstream, e.g: Gitea, Github, …
Authorization (AuthZ)
Once authentication is established, determining granular permissions inside Kubernetes presents the next critical architectural challenge. Several access control paradigms exist to fit distinct operational requirements.
Before diving into details, take a moment to review these foundational authorization and policy resources:
I can’t cover whole of concept Authorization (AuthZ) of Kubernetes, and I just focus on to the RBAC and highlighted usage, e.g: Node, Webhook, and ABAC. But you can use one or enable serveral authorization mode in Kubernetes with --authorization-mode=...,Node,ABAC,
Authorization Mode
--authorization-mode Flag
Mechanism / How It Works
Primary Use Case
Pros & Cons
RBAC (Role-Based Access Control)
RBAC
Matches incoming users/groups against dynamic Role or ClusterRole API objects mapped via RoleBinding or ClusterRoleBinding.
Standard user & service account permission management in production clusters.
Pros: Dynamic, manageable via kubectl, fine-grained control.
Cons: Role sprawl if not properly managed.
Node
Node
Authorizes API requests specifically originating from kubelet agents based on the resources assigned to their specific node.
Securing kubelet-to-APIServer communication.
Pros: Automated isolation between nodes.
Cons: Only applicable to kubelet traffic; usually paired with NodeRestriction admission plugin.
ABAC (Attribute-Based Access Control)
ABAC
Evaluates requests against static JSON policy files stored locally on the API server disk based on user, verb, resource, and namespace.
Legacy static permission definitions.
Pros: Simple declarative file format.
Cons: Requires API server restarts to update policies; hard to audit/scale.
Webhook
Webhook
Serializes request attributes into a SubjectAccessReview object and POSTs it to an external HTTP REST endpoint for an authorization decision.
Integration with external IAM services (e.g., AWS IAM, HashiCorp Vault).
Cons: Adds network latency and introduces external dependency risk.
AlwaysAllow
AlwaysAllow
Approves all incoming API requests without checking permissions.
Local single-node development (e.g., Minikube testing).
Pros: Zero configuration required.
Cons:Insecure; bypasses all cluster security.
AlwaysDeny
AlwaysDeny
Rejects all incoming API requests regardless of identity or permissions.
Security testing and debugging authorization fallback behavior.
Pros: Useful for verification tests.
Cons: Completely locks down the cluster.
Recommend For Production
In almost situation, I think use should consider use only RBAC and Node in Kubernetes for easier scale, and ensure reliable for Kubernetes Authozation model
RBAC
If you work with Kubernetes, RBAC is one of standard AuthZ introduced and use widely in this environment.
RBAC Authorization
Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization.
RBAC authorization uses the rbac.authorization.k8s.ioAPI group to drive authorization decisions, allowing you to dynamically configure policies through the Kubernetes API.
As usual, you will see these 4 common resources to define RBAC, including Role, ClusterRole, RoleBinding and ClusterRoleBinding. These 4 show the relationship between each others, and you entirely assigned this AuthZ for your users, groups, or service account to make you can interact with Kubernetes workloads
When you bootstrap the cluster, your certificate is define you authentication into system:masters group with cluster-admin ClusterRole who have highest priviledge inside Kubernetes Cluster, for range use control with ClusterRole or Role scope. Out of them, Kubernetes introduce a lots of default these RBAC to help you control Kubernetes easier, but in sometime that overflow what you actual hit to
API Discovery Roles: To read API information that is deemed safe to be publicly accessible
User-facing Roles: These are intended to be user-facing roles. They include super-user roles (cluster-admin), roles intended to be granted cluster-wide using ClusterRoleBindings, and roles intended to be granted within particular namespaces using RoleBindings (admin, edit, view).
Core Components Roles: To define roles for specific core components in Kubernetes, e.g: kube-scheduler, kube-controller-manager, …
Other Components Roles: To define roles for utilites components in Kubernetes, e.g: kube-dns, monitoring, kubelet-api-admin, …
Roles for built-in controllers: Corresponding roles exist for each built-in controller, prefixed with system:controller:
When I worked with RBAC, you have couple of features, that offer to help you manage these binding more easier with
Aggregated ClusterRoles: You can aggregate several ClusterRoles into one combined ClusterRole. A controller, running as part of the cluster control plane, watches for ClusterRole objects with an aggregationRule set. The aggregationRule defines a label selector that the controller uses to match other ClusterRole objects that should be combined into the rules field of this one.
apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: name: monitoringaggregationRule: clusterRoleSelectors: - matchLabels: rbac.example.com/aggregate-to-monitoring: "true"# The control plane automatically fills in the rules
With W’xOps IDP, I try to leverage these concept to ensure the matrix permission can setup, and add enought platform api of W’xOps Core Controller to let each package will come with right permission when the new API Introduced
Other feautes, Referring to special subjects, as the usual, you can setup subject for each role with User, Group, and ServiceAccount. But Authenticator modules provide group information. Groups, like users, are represented as strings, and that string has no format requirements, other than that the prefix system: is reserved.
Warning
The prefix system: is reserved for Kubernetes system use, so you should ensure that you don’t have users or groups with names that start with system: by accident. Other than this special prefix, the RBAC authorization system does not require any format for usernames.
Especially prefix work with ServiceAccount like system:serviceaccount and system:serviceaccounts
# For all service accounts in the <namespace> namespace:subjects:- kind: Group name: system:serviceaccounts:<namespace> apiGroup: rbac.authorization.k8s.io# For all service accounts in any namespace:subjects:- kind: Group name: system:serviceaccounts apiGroup: rbac.authorization.k8s.io
Others, you can work with authenticated and unauthenticated group user
# authenticatedsubjects:- kind: Group name: system:authenticated apiGroup: rbac.authorization.k8s.io# unauthenticatedsubjects:- kind: Group name: system:unauthenticated apiGroup: rbac.authorization.k8s.io# for allsubjects:- kind: Group name: system:authenticated apiGroup: rbac.authorization.k8s.io- kind: Group name: system:unauthenticated apiGroup: rbac.authorization.k8s.io
Info
To build the system:unauthenticated, as usually, you can skip define user part or not provide token/cert in your kubeconfig. Explore at Kubelet authentication/authorization
Node authorization is a special-purpose authorization mode that specifically authorizes API requests made by kubelets.
Let’s took it back to Node permission, from v1.8, system:node role is introduced, this role will grant permission for kubelet (node) in group system:nodes, including read access to all secrets, and write access to all pod status objects.
Recommendation from Kubernetes
User should use the Node authorizer and NodeRestriction admission plugin instead of the system:node role, and allow granting API access to kubelets based on the Pods scheduled to run on them.
Feature / Attribute
Before Node Authorization (v1.8)
With Node Authorization & NodeRestriction (Modern K8s)
Authorization Mode
Broad RBAC (ClusterRoleBinding to system:nodes)
Specialized Node authorization engine
Secret/ConfigMap Scope
Cluster-wide. A node could read any secret anywhere.
Graph-isolated. A node can only read secrets mounted by pods explicitly scheduled to it.
Object Modification
Could modify, delete, or fake status updates for any pod.
Bound strictly to pods where .spec.nodeName matches that specific node.
Identity Enforcement
Verified the credential belonged to a node.
Enforces that the username matches exactly: system:node:<nodeName>.
In order to be authorized by the Node authorizer, kubelets must use a credential that identifies them as being in the system:nodes group, with a username of system:node:<nodeName>. This group and user name format match the identity created for each kubelet as part of kubelet TLS bootstrapping.
Explore more advantage and playgroup with Node Authorization
ABAC (Attribute-based access control ) in Kubernetes
Attribute-based access control (ABAC) defines an access control paradigm whereby access rights are granted to users through the use of policies which combine attributes together.
Because ABAC use static file --authorization-policy-file=SOME_FILENAME to define ABAC so it means that hard to let you can scale in big-cluster RBAC and Node can, but ABAC took couple advantage when RBAC can solved, e.g
User attributes (department, clearance level, location)
When specified, mode Webhook causes Kubernetes to query an outside REST service when determining user privileges.
When faced with an authorization decision, the API Server POSTs a JSON- serialized authorization.k8s.io/v1beta1SubjectAccessReview object describing the action. This object contains fields describing the user attempting to make the request, and either details about the resource being accessed or requests attributes.
In the CloudNative, I think you consider to use Kyeverno as the opensource supported check this request for SubjectAccessReview, but instead you can implement them by yourself with support of SDK from Golang or Rust. Explore more below
GitHub - authorino: K8s-native AuthN/AuthZ service to protect your APIs.
Admission
Beside standard Role-Based Access Control (RBAC) often lacks the expressiveness needed for highly dynamic or complex policies, alternative models, such as Admission, or Policy-as-Code (PaC) frameworks, should be considered to build a strictly governed Kuberntes Security model.
Before deep diving into Admission, you should read these articles for more information
After AuthN and AuthZ in Kubernetes, you have one more layer (phase) to protect the request to Kubernetes API, before this event write down to etcd, and operate your action request. That’s called Admission Control
Admission Control
An admission controller is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the resource, but after the request is authenticated and authorized.
Admission controllers apply to requests that create, delete, or modify objects. Admission controllers can also block custom verbs, such as a request to connect to a pod via an API server proxy.
Admission control mechanisms may be validating, mutating, or both. Mutating controllers may modify the data for the resource being modified; validating controllers may not.
Attention
Admission controllers do not (and cannot) block requests to read (get, watch or list) objects, because reads bypass the admission control layer.
Info
By default, admission feature will be setup default by almost Kubernetes Cluster Distribution, e.g: RKE2, K3s, EKS, AKS, … You can double-check with command
These controllers divide your KubeAPI request to two part with two webhook controllers execute the mutating and validating (respectively) admission control webhooks which are configured in the API. MutatingAdmissionPolicy and ValidatingAdmissionPolicy provides a way to embed declarative mutation/validation code within the API, without relying on any external HTTP callouts.
In Addmission Phases, Mutating will execute first and after it will execute Validation. If any of the controllers in either phase reject the request, the entire request is rejected immediately and an error is returned to the end-user.
To enable and disable, you can set the option flag with Kubernetes API Server
To run admission in Kubernetes, it usually run as webhooks configured at Runtime.
Admission WebHooks are HTTP callbacks that receive admission requests and do something with them. There are two type webhook to define, validating admission webhook and mutating admission webhook. Following the Kubernetes accessible workflow, Mutation admission webhook will invoke first, and modify objects sent to Kubernetes Server enforce custom default, after that will continue to invoke by validation admission webhook, that can reject request to enforce custom policies.
admissionReviewVersions is required when creating ValidationWebhookConfiguration, and at least one version unserstood by Kubernetes API Server. If none in the list supported, your configuration will not be allowed to be created.
If an API server encounters a webhook configuration that was previously created and does not support any of the AdmissionReview versions the API server knows how to send, attempts to call to the webhook will fail and be subject to the failure policy.
To Response, Webhook will response with 200 HTTP status code, with Content-type: json and AdmissionReview API objects, but add the response with at least respond.uid and reponse.allowed (true or false). When got rejected, that will response customize HTTP status code (e.g: 403) with one more field status, include status.code and status.message.
Warning
Admission webhooks can optionally return warning messages that are returned to the requesting client in HTTP Warning headers with a warning code of 299. Warnings can be sent with allowed or rejected admission responses. Use warnings message to reponse, instead of message.
For confguration, you must define them as valid DNS subdomain name. You can define one or more webhook with each configuration, but should be define unique name for each one to get easier read audit logs and metrics for active configuration.
Matching requests: rules: An incoming request matches one of the specified operations, groups, versions, resources, and scope for any of a webhook’s rules, the request is sent to the webhook.
Matching requests: objectSelector: Webhooks may optionally limit which requests are intercepted based on the labels of the objects they would be sent, by specifying an objectSelector
Matching requests: namespaceSelector: Webhooks may optionally limit which requests for namespaced resources are intercepted, based on the labels of the containing namespace, by specifying a namespaceSelector.
Matching requests: matchPolicy: The matchPolicy lets a webhook define how its rules are used to match incoming requests. Allowed values are Exact or Equivalent.
Exact means a request should be intercepted only if it exactly matches a specified rule.
Equivalent means a request should be intercepted if it modifies a resource listed in rules, even via another API group or version.
The matchPolicy for an admission webhooks defaults to Equivalent.
Matching requests: matchConditions: You can define match conditions for webhooks if you need fine-grained request filtering. These conditions are useful if you find that match rules, objectSelectors and namespaceSelectors still doesn’t provide the filtering you want over when to call out over HTTP. Match conditions are CEL expressions. All match conditions must evaluate to true for the webhook to be called.
apiVersion: admissionregistration.k8s.io/v1kind: ValidatingWebhookConfigurationwebhooks: - name: my-webhook.example.com matchPolicy: Equivalent rules: - operations: ['CREATE','UPDATE'] apiGroups: ['*'] apiVersions: ['*'] resources: ['*'] # You can have up to 64 matchConditions per webhook matchConditions: - name: 'exclude-leases' # Each match condition must have a unique name expression: '!(request.resource.group == "coordination.k8s.io" && request.resource.resource == "leases")' # Match non-lease resources. - name: 'exclude-kubelet-requests' expression: '!("system:nodes" in request.userInfo.groups)' # Match requests made by non-node users. - name: 'rbac' # Skip RBAC requests, which are handled by the second webhook. expression: 'request.resource.group != "rbac.authorization.k8s.io"'
Info
There are a maximum of 64 match conditions allowed.
The exact matching logic is (in order):
If ANY matchCondition evaluates to FALSE, the webhook is skipped.
If ALL matchConditions evaluate to TRUE, the webhook is called.
If any matchCondition evaluates to an error (but none are FALSE):
If failurePolicy=Fail, reject the request
If failurePolicy=Ignore, the error is ignored and the webhook is skipped
Match conditions have access to the following CEL variables
object - The object from the incoming request. The value is null for DELETE requests. The object version may be converted based on the matchPolicy.
oldObject - The existing object. The value is null for CREATE requests.
request - The request portion of the AdmissionReview, excluding object and oldObject.
authorizer - A CEL Authorizer. May be used to perform authorization checks for the principal (authenticated user) of the request. See Authz in the Kubernetes CEL library documentation for more details.
authorizer.requestResource - A shortcut for an authorization check configured with the request resource (group, resource, (subresource), namespace, name).
To contact the webhook, you can define the contact information inside clientConfig. Webhooks can either be called via a URL or a service reference, and can optionally include a custom CA bundle to use to verify the TLS connection.
URL: Gives the location of the webhook, in standard URL form (scheme://host:port/path). The scheme must be “https”; the URL must begin with “https://“.
Service reference: The service stanza inside clientConfig is a reference to the service for this webhook. If the webhook is running within the cluster, then you should use service instead of url. The service namespace and name are required. The port is optional and defaults to 443. The path is optional and defaults to ”/“.
Side effect: Webhooks that make out-of-band changes (“side effects”) must also have a reconciliation mechanism (like a controller) that periodically determines the actual state of the world, and adjusts the out-of-band data modified by the admission webhook to reflect reality. Webhooks indicate whether they have side effects using the sideEffects field in the webhook configuration: None or NoneOnDryRun
Timeouts: timeoutSeconds allows configuring how long the API server should wait for a webhook to respond before treating the call as a failure. The timeout value must be between 1 and 30 seconds.
Reinvocation policy: built-in mutating admission plugins are re-run if a mutating webhook modifies an object, and mutating webhooks can specify a reinvocationPolicy to control whether they are reinvoked as well. reinvocationPolicy may be set to Never or IfNeeded. It defaults to Never.
Failure policy: defines how errors encountered while calling the admission webhook are handled. Allowed values are Ignore or Fail. The default failurePolicy for an admission webhooks is Fail.
Ignore means that an error calling the webhook is ignored and the API request is allowed to continue.
Fail means that an error calling the webhook causes the admission to fail and the API request to be rejected.
The failure policy applies to the following types of errors:
Network errors, timeouts, or connection failures when contacting the webhook.
The webhook returns a non-2xx HTTP response or a malformed response.
The API server fails to serialize the admission request or create an internal HTTP client for the webhook.
(Only for mutating webhooks) the response contains an undecodable or unsupported patch type.
Admission Webhook provide you two ways to observe the action activitity
Mutating webhook auditing annotations: The Kubernetes API server performs auditing on each mutating webhook invocation. Each invocation generates an auditing annotation capturing if a request object is mutated by the invocation, and optionally generates an annotation capturing the applied patch from the webhook admission response. The annotations are set in the audit event for given request on given stage of its execution, which is then pre-processed according to a certain policy and written to a backend.
Admission webhook metrics: The API server exposes Prometheus metrics from the /metrics endpoint, which can be used for monitoring and diagnosing API server status. The following metrics record status related to admission webhooks.
MutatingAdmissionPolicies allow you to change what happens when someone writes a change to the Kubernetes API. This is the native way and implemented by Kubernetes, and worked as the solution alternative for Mutating Policies Webhook without webhook latency.
ValidatingAdmissionPolicies allow you validate the interact with Kubernetes API. This one is ultra-fast, and zero-latency to made it as the alternative of Validation Admission Webhooks. Same as MAP, VAP use the Common Expression Language (CEL) to declare validation rules of a Policy.
It also combine by three resources to build policy, same as MAP
ValidatingAdmissionPolicy: Describes the abstract logic of a policy.
Parameter Resource: Provides information to a ValidatingAdmissionPolicy to make it a concrete statement.
A ValidatingAdmissionPolicyBinding: Links the above resources together and provides scoping.
From my perspective, I’m not actual notice MAP and VAP of Kubernetes first, but actual Policies as Code Engine already start with me from day 1 to execute Policy Mangement, and Kyverno is the first Policy Engine, that I working with from Lab, and It’s integrate into W’xOps IDP as part of BACK Framework (K = Kyverno - Governance & Policies in Platform).
Policy Management
Base on CNCF - Kubernetes Policy Managment WhitePaper V1, A policy is a set of rules adopted by an organization to help achieve its desired outcomes. Policies can help guide behaviors across wide ranging concerns like costs, security, and productivity.
In Kubernetes, a policy serves as a digital contract between application owners, cluster administrators, and security stakeholders. Kubernetes administrators deploy policies that describe what configurations and behaviors are allowed based on the organization’s security and compliance requirements. The policies are configured to allow, deny, or audit the configurations and behaviours of interest.
By IBM, Policy as code is a method of expressing rules and constraints—including security, compliance and organizational policies—in machine-readable code that is automatically enforced across IT architectures and DevOps pipelines.
By Hashicorp, Policy as code gives you an automated way to check, in seconds, if your IT and business stakeholders’ requirements are being followed — in every deployment.
By CNCF, The main idea around PaC is to handle configuration and standardization in your environments in an easier and more governed way.
The mechanism typically associated with the term X as Code signifies the transition from a manual process to an automated one.
Things like security and environment manageability become continuously improving while adopting software engineering best practices like source code versioning and rollback capabilities. This will allow you and your company to evaluate every change that has been made and the impacts it has shown.
That reason why there are many tools, and concepts introduced before we got MAP and VAP appear in Kubernetes. By the reconcile loop mechanism with various resource definiton, it enable the capabilities to protect this Kubernetes resource via Webhook.
I don’t actual tell you should choose between PAC Engine (Kyverno, OPA Gatekeeper, …) and VAP, both of them are the good options to work with at Admission Phases, and here reason you can choose one of them
Use Native VAP / MAP if: You only need validating or mutating rules for Kubernetes objects (e.g., enforcing label constraints, setting default security contexts, blocking root users) and you want maximum performance, zero webhook latency, and no extra pods to manage.
Use Kyverno if: You need end-to-end governance—including auto-generating resources, verifying image supply chains, scheduling automated cleanups, generating audit reports for existing resources, or running policies in local CI/CD testing pipelines
You can explore more with Admission Policies solutions below, and several articles to deep diving.
When exploring policy paradigms for authorization, you will encounter a model that isn’t typically highlighted in standard Kubernetes documentation. The final member of the access control trio (alongside RBAC and ABAC) is Relationship-Based Access Control (ReBAC). You might wonder whether ReBAC is applicable to Kubernetes, if anyone has implemented it, or if it is suitable for production environments.
The answer is yes, ReBAC solutions for Kubernetes do exist experiement but rare, and the answer is No for who use this technique for production. While ReBAC is primarily used at the application layer using graph-based relationship mechanisms (such as Google Zanzibar or OpenFGA), several experimental initiatives are pioneering how to integrate ReBAC directly into Kubernetes authorization webhooks.
Before diving into the implementation details, explore the resources below to learn more about ReBAC and its application to Kubernetes authorization models:
ReBAC is an authorization model where access is defined by your relationship to the resource, not just your assigned role. A ReBAC system uses subjects, objects, and relationships, where permissions are derived from stored relationship facts checked against a rulebook.
Exploring the kube-rebac-authorizer repository by Lucas Käldström (CNCF Ambassador and contributor to Cedar Policy Language & kubeadm) reveals an intriguing approach: unifying standard Kubernetes RBAC and Node Authorizers into a single, cohesive engine based on Google’s Zanzibar architecture.
To implement Relationship-Based Access Control (ReBAC), the project translates standard Kubernetes RBAC definitions and Node API requests into relation tuples. These tuples construct a directed graph defining relationships between subjects (users, groups, service accounts, nodes) and Kubernetes resources:
Tuple Mapping: Maps structures such as clusterrole:<name>#assignee to resource:<apiGroup>/<resource> through specific verb relations (e.g., get, list, watch, update, delete).
Engine Evaluation: A controller processes Kubernetes API objects, derives the desired relationship graph tuples, and delegates authorization decisions to ReBAC engines like OpenFGA, Authzed SpiceDB, or Ory Keto.
Declarative CRDs: Instead of writing imperative authorization webhooks in code, operators can define custom graph-based authorization rules using declarative Kubernetes Custom Resource Definitions (CRDs).
More than Auth, Kubernetes do more things for secure and reliable
Not only Auth modelization with 3 layers, Kubernetes also provide a lot of mechanism to protect your deployment, especially Resources, Security Context for Pod & Container, and Networking. Before talk a bit about these topics, you can double-check via these links below
Limit Range: To ensure that Pods specify their resource requirements.
Dynamic Resource Allocation (DRA): a Kubernetes feature that lets you request and share resources among Pods. These resources are often attached devices like hardware accelerators. This feature is alternative for Device Plugin mechanism, define device classes that are available to claim in workloads. Kubernetes allocates matching devices to specific claims and places the corresponding Pods on nodes that can access the allocated devices.
Quality of Service Classes (QoS): Kubernetes classifies the Pods that you run and allocates each Pod into a specific quality of service (QoS) class. Kubernetes uses that classification to influence how different pods are handled. Kubernetes does this classification based on the resource requests of the Containers in that Pod, along with how those requests relate to resource limits.
TL;DR
Why I conclude DRA in to Resource Manager, because I think DRA introduced the mechanism to protect expensive resource mounted into pods via authorized by DeviceClass
Though neither is a standalone security tool, DRA significantly improves security posture over legacy Device Plugins:
Legacy Device Plugins (Integer-Based): Device Plugins could only count units (nvidia.com/gpu: 1). Because they couldn’t inspect or request fine-grained hardware properties, cluster operators often had to run privileged container hooks or allow dangerous node-level environment variables (like ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED) to route workloads.
DRA (Structured Parameters & CDI): DRA uses structured device attributes, CEL expression filtering, and Container Device Interface (CDI) specs. It provides:
Least-Privilege Mounting: Injects only the specific hardware slices or virtual function endpoints authorized by the DeviceClass.
Multi-Tenant Sharing Controls: Allows admins to define explicit sharing policies (e.g., strict hardware MIG slice vs. shared device claims) via declarative K8s APIs rather than ad-hoc runtime scripts.
Fine-Grained RBAC: Access to compute accelerators can be governed using standard Kubernetes RBAC on ResourceClaim and DeviceClass objects.
In Kubernetes, There are many several network security measures introduced, such as
NetworkPolicy: An application-centric construct which allow you to specify how a pod is allowed to communicate with various network “entities” (we use the word “entity” here to avoid overloading the more common terms such as “endpoints” and “services”, which have specific Kubernetes connotations) over the network. To use network policies, you must be using a networking solution which supports NetworkPolicy.
ServiceMesh: Address this new set of challenges by managing traffic (i.e., communication) between services and adding reliability, observability, and security features uniformly across all services.
eBPF: a technology that allows small, sandboxed programs or scripts to run in the kernel space of a Linux system without having to change the kernel’s source code or load Linux kernel modules. Explore detail at eBPF Documentation
Beside Resource and Network, I think Kubernetes provide you a lots of concept, and the things which you usually work with Pod & Container have own the way to protect with various mechanism, including
To delve deeper into Kubernetes security enhancement with admission control, Kyverno is an excellent choice for a hands-on playground and a valuable solution to explore. For Kubewekend, I don’t want to be overly constrained by specific policies or authentication concepts, the final setup is up to you. However, if you’re asking for my preference on where to start, Kyverno is a solid option.
You can easily install Kyverno policies into your cluster using the following helm commands:
This comamnd will install key components of Kyverno, including
CRDs
Admission controller
Reports controller
Cleanup controller
Background controller
Combining with my perspective, I recommend you to find and explore Kyverno and experience from reference above and couple link below to understand Kyverno as Policy Manamament in Kubernetes
Kyverno is a Cloud Native Policy Engine. It orignally built for Kubernetes and now can available to use out of Kubernetes cluster as Unified Policy Language.
Kyverno allows platform engineers to automate security, compliance, and best practices validation and deliver secure self-service to application teams.
Providing CRDs to easier defining Policies in Kubernetes
Supporting various Policy Types for help you define for any Kubernetes Resources, e.g: Mutating, Validating, Generating, Deleting, …
Supporting Policy Report in Kubernetes using the OpenReport format.
Support Policy Exception
Work as Policies as Code
Others, …
The architecture of Kyverno concisely includes,
Kyverno Admission Controller run as a dynamic admission controller, that will receive admission webhook HTTP callback from Kubenetes API Server and applies matching policies to return results that enforce admission policies or reject requests.
These Policies will dynamically configured by WebHook Controller which watches the installed policies and modifies the webhooks to request only the resources matched by those policies.
The Cert Renewer is responsible for watching and renewing the certificates, stored as Kubernetes Secrets, needed by the webhook.
The Background Controller handles all generate and mutate-existing policies by reconciling UpdateRequests, an intermediary resource
Policy enforcement is captured using Kubernetes events. You can also use Policy Reports in the Cluster for detail reporting by Kyverno Report Controller
Beside the core, Kyverno also support serval features, and sub-project, including
Kyverno Metrics is the metrics route exposed by admission controller to show information about Kyverno Policies
Kyverno CLI to pefrom apply Kyverno Policies to clusters or resource file, with purpose validate and test policy behavior, and also it can used in CI/CD Pipelines as validation assistant and available use binary or integrate into kubectl.
Kyverno Authz Server as Application Authorization with Kyverno Policies. It suppports Envoy and HTTP payloads, and can be run as a central service or as a sidecar container.
For the day old. Kyverno use YAML to define Kubernetes CRDs and these policies types, including ClusterPolicy, Policy, and CleanupPolicy. It defined depend on JMESPath as a fast and effecient JSON processing language.
With ClusterPolicy, you can define the rule by declare a match declaration, an optional exclude declaration, and one of a validate, mutate, generate, or verifyImages declaration. Each rule can contain only a single validate, mutate, generate, or verifyImages child declaration.
With Policy, this is actual ClusterPolicy but instead cluster-scoped, it defines for namespace-scoped.
As my perspective, I apply a lot of concept in early version W’xOps IDP, I try to integrate Kyverno to handle couple of rules for
Generate: This option I use most for defining networkpolicies, resourcesquotas, and secret-sync. With secret-sync, I think that pretty cool work to make me can easier replicate secrets for another namespace, e.g: Registry credentials, and it’s helping me reducing the complex when I can remove kubernetes-reflector out of my scope.
Validating: This option is handling a lot of validation also, I got several them for security to ensure priviledge in namespace, label for workload in any namespace, …
Let’s try to implementation one of them, but I don’t recommend you to following this syntax, you should migrate them into CEL, for long-term usage and stable. Explore more at Migrating to CEL Policies
But in v1.19.X, you can still try this concept before it’s hit to v1.20.X
So let’s try with one example by trying Generate resource across namespace, this is highly value function, and help you ensure your secrets only come from single of trust. In my situation, Secret is Private Registry from Vault to Kubernetes via ESO (External Secret Operator).
Kyverno called this mechanism, CloneSource, the generate policy will clone from the resource name and location defined in the rule to create the new resource. Use of the clone object implies no modification during the path from source to destination and Kyverno is not able to modify its contents (aside from metadata used for processing and tracking).
By default, Kyverno don’t have permission to touch into Kubernetes resources, e.g: Secrets, NetworkPolicies, or ResourceQuotas. To let Kyverno can worked with these resources, you should grant permission for it via aggregation with ClusterRole of Kyverno
# RBAC aggregation ClusterRoles for all Kyverno GeneratingPolicies.# Each resource type managed by a GeneratingPolicy requires two ClusterRoles:# :view — aggregated to background-controller, reports-controller, admission-controller# so Kyverno can read existing resources during evaluation and reporting.# :manage — aggregated to background-controller only# so Kyverno can create, update, and delete the generated resources.## Kyverno picks these up automatically via label aggregation — no ClusterRoleBinding needed.# =============================================================================# Secrets (sync-secrets.yaml)# =============================================================================apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: name: kyverno:secrets:view labels: rbac.kyverno.io/aggregate-to-background-controller: "true" rbac.kyverno.io/aggregate-to-reports-controller: "true" rbac.kyverno.io/aggregate-to-admission-controller: "true"rules: - apiGroups: - "" resources: - secrets verbs: - get - list - watch---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: name: kyverno:secrets:manage labels: rbac.kyverno.io/aggregate-to-background-controller: "true"rules: - apiGroups: - "" resources: - secrets verbs: - create - update - delete - patch# =============================================================================# NetworkPolicy (network-policies.yaml)# =============================================================================---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: name: kyverno:networkpolicies:view labels: rbac.kyverno.io/aggregate-to-background-controller: "true" rbac.kyverno.io/aggregate-to-reports-controller: "true" rbac.kyverno.io/aggregate-to-admission-controller: "true"rules: - apiGroups: - networking.k8s.io resources: - networkpolicies verbs: - get - list - watch---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: name: kyverno:networkpolicies:manage labels: rbac.kyverno.io/aggregate-to-background-controller: "true"rules: - apiGroups: - networking.k8s.io resources: - networkpolicies verbs: - create - update - delete# =============================================================================# ResourceQuota + LimitRange (resource-quota.yaml)# =============================================================================---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: name: kyverno:resourcequotas:view labels: rbac.kyverno.io/aggregate-to-background-controller: "true" rbac.kyverno.io/aggregate-to-reports-controller: "true" rbac.kyverno.io/aggregate-to-admission-controller: "true"rules: - apiGroups: - "" resources: - resourcequotas - limitranges verbs: - get - list - watch---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: name: kyverno:resourcequotas:manage labels: rbac.kyverno.io/aggregate-to-background-controller: "true"rules: - apiGroups: - "" resources: - resourcequotas - limitranges verbs: - create - update - delete
After this worked, you can try to create Generating Policies to sync secrets between available namespace to another namespace. Explore more at Kyverno Sample - Sync Secrets
apiVersion: kyverno.io/v1kind: ClusterPolicymetadata: name: sync-secrets annotations: policies.kyverno.io/title: Sync Secrets policies.kyverno.io/category: Sample policies.kyverno.io/subject: Secret policies.kyverno.io/minversion: 1.6.0spec: rules: - name: sync-image-pull-secret # ref: https://kyverno.io/docs/policy-types/cluster-policy/match-exclude/#match-statements match: any: - resources: kinds: - Namespace names: - "tenant-*" generate: apiVersion: v1 kind: Secret name: regcred namespace: "{{request.object.metadata.name}}" synchronize: true # ref: https://kyverno.io/docs/policy-types/cluster-policy/generate/#clone-source # warning: for the clone source, kyverno will assign labels to sources for tracking, so make sure it should not remove by any controller or operator # that manages secrets in the source Namespace. Otherwise, the policy will not work as expected. clone: namespace: external-secrets name: regcred # generateExisting is required to ensure that the policy will update existing Secrets if the source Secret changes. # If this field is not set, the policy will only create new Secrets for new Namespaces, but will not update existing Secrets in already created Namespaces. generateExisting: true
YAML (CEL)
Now from Kyverno v1.19.X, you can use YAML (CEL) instead YAML (JMESPath) as stable version, and It supports entirely the old version features provided.
Common Expression Language (CEL)
The Common Expression Language (CEL) is used in the Kubernetes API to declare validation rules, policy rules, and other constraints or conditions.
CEL expressions are evaluated directly in the API server, making CEL a convenient alternative to out-of-process mechanisms, such as webhooks, for many extensibility use cases
Recently, CEL already integrate and support widely in the core of Kubernetes from v1.29+. Beside, Kyverno catchup and evolve with same mechanism of Kubernetes, it also create the method to let Kyverno generate Kubernetes Native by spec.auto feature of Validation Admission Policy and Mutating Admission Policy.
Now trying to implement the GeneratingPolicy with same idea above about regcred secrets generated by external-secrets
Attention
Remembering applying the RBAC aggregation for Kyverno controller to be able to create the Kubernetes resources.
That’s all for today. I think this covers plenty of material for you to discover regarding security, as well as where to focus and get involved in your Kubernetes ecosystem. Each phase in the Kubernetes authorization model looks interesting and continues to evolve every day, so I look forward to seeing what comes next.
Quote
After taking time to sit back and review everything I’ve worked on, investigated, and explored deeply within Kubernetes and security, I’ve gained numerous ideas, concepts, and updates on the latest Kubernetes versions for managing the entire system. I hope you had a great time researching and learning alongside me. Stay healthy, keep up the momentum, and I will see you in the next post. Bye and good luck!!