The Kubernetes validating admission policy provides a mechanism to allow users to add custom validation policies into the Kubernetes admission control flow. These policies are defined as Kubernetes admission resources, containing validation rules expressed by Common Expression Language (CEL).
By using validating admission policy, users no longer need to hard-code validation policies in validating admission webhook.
This write-up goes over the validating admission policy APIs, code path, metrics and examples on how to enforce immutable and unique resource properties.
The validating admission policy has reached general availability (stable) since Kubernetes 1.30.
What It Is
The admission control flow is made up of a number of admission controllers, which are gatekeeping plugins within the Kubernetes API Server that run to intercept and check authenticated API requests sent to interact with Kubernetes resources. These controllers can mutate and validate Kubernetes resources, and allow and deny these API requests.
Prior to validating admission policy, the way to introduce custom validation policies into the admission control flow is to embed them in validating admission webhooks, which run during the validating phase of the admission control flow.
📝 An example of validation policy may be one that includes rules that prohibit all admitted pods from exposing port 80 and binding host path volumes.
These webhooks are usually implemented as 3rd party components that the Kubernetes API server communicates with. They normally come with infrastructure, software and security maintenance overheads.
As an alternative, the in-tree validation admission policy mechanism can be used to safely satisfy many admission validation scenarios.
📝 Admission webhooks are still suitable for use cases that, for example, involve interaction with 3rd party services.
At the heart of this built-in policy framework is the ability to express validation rules in CEL. The CEL is sufficiently lightweight and safe to be run directly in the Kubernetes API Server. It has a straight-forward syntax and grammar, which comes with built-in pre-parsing and type-checking mechanism.
🛠️ The CEL playground is a great tool to help write and verify CEL expressions.
Let’s go over the APIs and resources of the validating admission policy.
API Definition
The validating admission policy is composed of 3 core APIs.
The ValidatingAdmissionPolicy resource defines a collection of CEL-based validation rules. This resource also defines:
- The kind of resources the policy applies to
- The kind of resources that are used to parameterize the policy
- How violation of the policy is reported
To enforce the policy, a corresponding ValidatingAdmissionPolicyBinding resource must be defined. Without it, the policy will have no effects on the rest of the cluster. A single policy can be reused by multiple bindings. Each binding can reference its own set of parameters.
Both the ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding are mandatory cluster-scoped resources. The optional parameter resources can be represented by native types such as ConfigMap or custom CRD types. The parameter resources can be scoped to either the cluster level or namespace level. This means parameters can be applied to either the entire cluster or the different namespaces.
Let’s dive into the Kubernetes codes to see how this works 🤓.
How It Works
This code walkthrough aims at providing a sense of how the different Kubernetes components work and where the relevant code are.
Controller Reconciliation
Within the kube-controller-managercode, a controller named validatingadmissionpolicy-status-controller is registered to manage the ValidatingAdmissionPolicy kind:
This controller is started along with other Kubernetes controllers when the kube-controller-manager runs:
The initFunc routine of the relevant controllerDescriptor initializes the underlying validatingadmissionpolicystatus controller defined in the k8s.io/kubernetes/pkg/controller/validatingadmissionpolicystatuspackage. As its name implied, this controller reconciles the status of the ValidatingAdmissionPolicy kind:
The controller’s reconciliation loop invokes the type checker to validate the rules and message expressions defined with the policy’s spec.validations list:
This reconciliation which involves the expression checks are repeated every time the ValidatingAdmissionPolicy resources are changed.
Resource Admission
During the admission control setup, a validating admission plugin named ValidatingAdmissionPolicy was added to the control flow via the admission.Plugins registration mechanism:
This plugin implements the admission.ValidationInterface interface and delegates the actual rules validation to the PolicyHook dispatcher:
The PolicyHook type has an Evaluator that knows how to validate the ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding resources:
The Evaluator provides an implementation of the validating.Validator interface. When the dispatcher received an admission request, it calls the Evaluator to convert the CEL expressions into policy decisions:
The policy decisions contain actions to be taken following the CEL evaluation, accounting for the policy’s failurePolicy setting.
Expression Cost Analysis
To ensure that the Kubernetes API Server doesn’t get overwhelmed by the CEL expressions evaluation, Kubernetes uses the cel-go cost subsystem to estimate and track the runtime cost of evaluating CEL expressions.
The cost of processing a ValidatingAdmissionPolicy resource is dictated by the complexity of the CEL expressions used to represent the validation rules, compounded by the number of associated ValidatingAdmissionPolicyBinding resources.
Within the Kubernetes code, constant cost budgets, call limit, check frequency and request size are defined to constrain the runtime analysis and evaluation.
If the cost unit of the rules within a ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding pair exceeds the permitted runtime cost budget, the execution of the expressions will be halted with an error returned.
Following the Dispatcher code above, the validator.Validate() method calculates the cost of evaluating CEL expressions found in the spec.validations[*].expression property, and returns the remaining budget:
Within the same method, there is also a messageFilter used to evaluate the cost of the CEL expressions found in the spec.validations[*].messageExpression property, and an auditAnnotationFilter which determines the cost of the CEL expressions found in the list of spec.auditAnnotations property.
These filters implement the cel.ConditionEvaluator interface. Within the implementation of the ForInputs() method, the activation.Evaluate() method is invoked to use the cel.Program cost subsystem to calculate the CEL expressions’ actual cost and the policy’s remaining budget:
If the cost budget is exhausted, the validation failed due to running out of cost budget, no further validation rules will be run error will be returned:
Hopefully, the code is not too difficult to follow.
The next section explores two examples on how to use the validating admission policy to enforce immutable and unique resource specification.
Examples
Example 1 — Immutable Resource Specification
Imagine a CRD named vnodes.virt.dev which is used to represent virtual nodes running in Kubernetes.
An example VNode instance looks like this:
apiVersion: virt.dev/v1alpha1
kind: VNode
metadata:
name: vnode
labels:
virt.dev/immutability: enforced
spec:
cpu:
cores: 1
sockets: 1
threads: 1
memory:
guest: 3996Mi
devices:
disks:
- disk:
bus: virtio
name: disk-0
interfaces:
- macAddress: da:5a:20:f5:e4:ce
masquerade: {}
model: virtio
name: defaultIt is possible to define a simple validating admission policy with rules to ensure that once a VNode resource is created, its specification remains immutable.
💡 Using the validating admission policy like this is useful when it isn’t easy to directly modify the CRD schema with transition rules.
This is what the ValidatingAdmissionPolicy looks like:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: vnodes-immutability
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["virt.dev"]
apiVersions: ["v1alpha1"]
operations: ["UPDATE"]
resources: ["vnodes"]
validations:
- expression: "oldObject.spec == object.spec"
message: "VNode spec is immutable"
reason: InvalidHere’s quick explanation on some of the properties:
metadata.name— This is the name of the policy. All associatedValidatingAdmissionPolicyBindingresources must reference this name.spec.failurePolicy— This value defines the handling of policy failures which include cases like invalid CEL expressions, misconfigured bindings, non-existent parameter kinds etc. However, it doesn’t specify what to do with validations that are evaluated tofalse. That is done by theValidatingAdmissionPolicyBindingresource’sspec.validationActionsproperty.spec.matchConstraints— This property defines the kinds of resources that the policy validates.spec.validations— This is the list of validation rules and their associated messages, represented by CEL expressions.
❗The policy is not enforced until an associated
ValidatingAdmissionPolicyBindingresource is deployed.
The following ValidatingAdmissionPolicyBinding resource references the vnodes-immutability policy. It refines the the resource matching criteria to just VNodes resources with the virt.dev/immutability: enforced label:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: vnodes-immutability
spec:
policyName: vnodes-immutability
validationActions: [Deny]
matchResources:
objectSelector:
matchLabels:
virt.dev/immutability: enforcedWith this policy binding deployed, any attempt to update the previously created VNode resource specification will fail.
For example, attempting to use kubectl edit vnode vnode to change the value of the spec.cpu.cores property of the previous VNode resource will result in the following error message:
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
# vnodes.virt.dev "vnode" was not valid:
# * : ValidatingAdmissionPolicy 'vnodes-immutability' with binding 'vnodes-immutability' denied request: VNode spec is immutable
#👀 The
VNode spec is immutableerror message comes from theValidatingAdmissionPolicyresource.
Example 2 — Unique Specification Property
Let’s extend the VNode specification with an optional .spec.staticIP:
apiVersion: virt.dev/v1alpha1
kind: VNode
metadata:
name: vnode2
labels:
virt.dev/immutability: enforced
spec:
staticIP: 192.168.255.100 # new property
cpu:
cores: 1
sockets: 1
threads: 1
memory:
guest: 3996Mi
devices:
disks:
- disk:
bus: virtio
name: disk-0
interfaces:
- macAddress: da:5a:20:f5:e4:ce
masquerade: {}
model: virtio
name: defaultWe can use the following ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding resources to ensure every VNode resource gets an unique static IP:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: vnodes-unique-static-ip
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["virt.dev"]
apiVersions: ["v1alpha1"]
operations: ["CREATE","UPDATE"]
resources: ["vnodes"]
validations:
- expression: "params.allocations.all(e, e.ipAddress != object.spec.staticIP)"
messageExpression: "'invalid static IP. conflicting allocation: '
+ params.allocations.filter(e, e.ipAddress == object.spec.staticIP)[0].hwAddress
+ '/'
+ params.allocations.filter(e, e.ipAddress == object.spec.staticIP)[0].ipAddress"
reason: Invalid
paramKind:
apiVersion: virt.dev/v1alpha1
kind: StaticIPAllocation
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: vnodes-unique-static-ip
spec:
policyName: vnodes-unique-static-ip
validationActions: [Deny]
paramRef:
name: allocated
namespace: default
parameterNotFoundAction: DenyThe policy references the StaticIPAllocation kind as its parameter resource. This CRD is used to track the list of allocated (hence, unavailable) static IPs.
The custom controller that manages
StaticIPAllocationresources is not included in this post.
The policy binding definition narrows down the reference relationship to a specific instance of the StaticIPAllocation kind named allocated in the default namespace.
An example StaticIPAllocation resource looks like:
apiVersion: virt.dev/v1alpha1
kind: StaticIPAllocation
metadata:
name: allocated
allocations:
- ipAddress: 192.168.255.100
hwAddress: da:5a:20:f5:e4:ce
- ipAddress: 192.168.255.108
hwAddress: 00:1A:2B:3C:4D:5E
- ipAddress: 192.168.255.167
hwAddress: 2C:54:91:88:C9:E3Attempting to deploy the vnode2 virtual node which requests for the static IP 192.168.255.100 will fail with the following error:
The vnodes "vnode2" is invalid: : ValidatingAdmissionPolicy 'vnodes-unique-static-ip' with binding 'vnodes-unique-static-ip' denied request:
conflicting static IP allocation exists: da:5a:20:f5:e4:ce/192.168.255.100💡 Since the
StaticIPAllocationkind is namespace-scoped, multiple parameter sets can be defined to enforce different constrains in different namespaces.
Prometheus Metrics
With Prometheus installed, the apiserver_validating_admission_policy_check_duration_seconds can be used to determine the policy violation count and latency:
The Prometheus histogram_quantile function can be used to calculate the φ-quantile of check duration for specific policy binding pairs, over a certain time interval.
The following sample graph shows the 99-percentile check duration (in seconds) of the vnodes-immutability policy:
The next graph shows the 99-percentile check duration (in seconds) of the vnodes-unique-static-ip policy:
These metrics can be used to configure alerts when policy failures or evaluation latency exceed certain SLO thresholds.
Conclusion
The Kubernetes validating admission policy allows users to add validation policy to the admission control workflow. The validation rules, expressed as CEL expressions, are checked, evaluated and run by the Kubernetes API Server. This provides an alternative to implementing validating admission webhooks with hard-coded validation rules.
The code walkthrough in this post shows how the validating admission policy feature is implemented as a Kubernetes controller and an admission plugin.
Two examples are included to demonstrate how to utilize CEL expressions within the ValidatingAdmissionPolicy resource to enforce immutable and unique resource properties.
Finally, Kubernetes exposes key metrics to calculate failed policy counts and policy check duration latency.
