This is the full developer documentation for Pepr # Pepr > Pepr Documentation - latest # Pepr [Section titled “Pepr”](#pepr) | | | | | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | [](https://docs.pepr.dev) | [](https://npmjs.com/package/pepr) | [](https://npmjs.com/package/pepr) | [](https://npmjs.com/package/pepr) | | [](https://securityscorecards.dev/viewer/?uri=github.com/defenseunicorns/pepr) | [](https://codecov.io/github/defenseunicorns/pepr) | [](./contribute/code-of-conduct) | | ## ***Type safe Kubernetes middleware for humans*** [Section titled “Type safe Kubernetes middleware for humans”](#type-safe-kubernetes-middleware-for-humans) | | | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------: | | Pepr simplifies Kubernetes management by providing an alternative to complex YAML configurations, custom scripts, and ad-hoc solutions. As a Kubernetes controller, Pepr enables you to define Kubernetes transformations using TypeScript, accessible through straightforward configurations even without extensive development expertise. Pepr transforms disparate implementation approaches into a cohesive, well-structured, and maintainable system. With Pepr, you can efficiently convert organizational knowledge into code, improving documentation, testing, validation, and change management for more predictable outcomes. |  | ## Features [Section titled “Features”](#features) * Zero-config K8s Mutating and Validating Webhooks plus Controller generation * Automatic leader-elected K8s resource watching * Lightweight async key-value store backed by K8s for stateful operations with the [Pepr Store](user-guide/store) * Human-readable fluent API for generating [Pepr Capabilities](#capability) * A fluent API for creating/modifying/watching and server-side applying K8s resources via [Kubernetes Fluent Client](https://github.com/defenseunicorns/kubernetes-fluent-client) * Generate new K8s resources based off of cluster resource changes * Perform other exec/API calls based off of cluster resources changes or any other arbitrary schedule * Out of the box airgap support with [Zarf](https://zarf.dev) * Entire NPM ecosystem available for advanced operations * Realtime K8s debugging system for testing/reacting to cluster changes * Controller network isolation and tamper-resistant module execution * Least-privilege [RBAC](user-guide/rbac) generation * AMD64 and ARM64 support ## Example Pepr Action [Section titled “Example Pepr Action”](#example-pepr-action) This quick sample shows how to react to a ConfigMap being created or updated in the cluster. It adds a label and annotation to the ConfigMap and adds some data to the ConfigMap. It also creates a Validating Webhook to make sure the “pepr” label still exists. Finally, after the ConfigMap is created, it logs a message to the Pepr controller and creates or updates a separate ConfigMap with the [kubernetes-fluent-client](https://github.com/defenseunicorns/kubernetes-fluent-client) using server-side apply. For more details see [actions](actions) section. ```ts When(a.ConfigMap) .IsCreatedOrUpdated() .InNamespace("pepr-demo") .WithLabel("example", "value") // Create a Mutate Action for the ConfigMap .Mutate(request => { // Add a label and annotation to the ConfigMap request.SetLabel("pepr", "was-here").SetAnnotation("pepr.dev", "annotations-work-too"); // Add some data to the ConfigMap request.Raw.data["doug-says"] = "Pepr is awesome!"; // Log a message to the Pepr controller logs Log.info("A ConfigMap was created or updated:"); }) // Create a Validate Action for the ConfigMap .Validate(request => { // Validate the ConfigMap has a specific label if (request.HasLabel("pepr")) { return request.Approve(); } // Reject the ConfigMap if it doesn't have the label return request.Deny("ConfigMap must have the required pepr label"); }) // Watch behaves like controller-runtime's Manager.Watch() .Watch(async (cm, phase) => { Log.info(cm, `ConfigMap was ${phase}.`); // Apply a ConfigMap using K8s server-side apply (will create or update) await K8s(kind.ConfigMap).Apply({ metadata: { name: "pepr-ssa-demo", namespace: "pepr-demo-2", }, data: { uid: cm.metadata.uid, }, }); }); ``` ## Prerequisites [Section titled “Prerequisites”](#prerequisites) * [Node.js](https://nodejs.org/en/) v20.0.0+ (even-numbered releases only) * To ensure compatibility and optimal performance, it is recommended to use even-numbered releases of Node.js as they are stable releases and receive long-term support for three years. Odd-numbered releases are experimental and may not be supported by certain libraries utilized in Pepr. * [npm](https://www.npmjs.com/) v10.1.0+ * Recommended (optional) tools: * [Visual Studio Code](https://code.visualstudio.com/) for inline debugging and [Pepr Capabilities](#capability) creation. * A Kubernetes cluster for `npx pepr dev`. Pepr modules include `npm run k3d-setup` if you want to test locally with [K3d](https://k3d.io/) and [Docker](https://www.docker.com/). ## Quick Start Guide [Section titled “Quick Start Guide”](#quick-start-guide) ```bash # Create a new Pepr Module npx pepr init # If you already have a K3d cluster you want to use, skip this step npm run k3d-setup # Start playing with Pepr now! # If using Kind, or another local k8s distro instead, # run `npx pepr dev --host ` npx pepr dev kubectl apply -f capabilities/hello-pepr.samples.yaml ``` > \[!TIP] Don’t use IP as your `--host`, it’s not supported. Make sure to check your local k8s distro documentation how to reach your localhost, which is where `pepr dev` is serving the code from. Your browser doesn't support HTML video. [Download the video](https://user-images.githubusercontent.com/882485/230895880-c5623077-f811-4870-bb9f-9bb8e5edc118.mp4) instead. ## Concepts [Section titled “Concepts”](#concepts) ### Module [Section titled “Module”](#module) A module is the top-level collection of capabilities. It is a single, complete TypeScript project that includes an entry point to load all the configuration and capabilities, along with their actions. During the Pepr build process, each module produces a secret containing the transpiled and compressed TypeScript code. Modules with `Mutate()` or `Validate()` actions also produce the corresponding Kubernetes MutatingWebhookConfiguration or ValidatingWebhookConfiguration resources. Modules without admission, watch, queue, finalize, or schedule actions still deploy their module code with an isolated admission controller, but do not produce webhook configuration resources unless admission actions are defined. See [Module](user-guide/pepr-modules) for more details. ### Capability [Section titled “Capability”](#capability) A capability is set of related actions that work together to achieve a specific transformation or operation on Kubernetes resources. Capabilities are user-defined and can include one or more actions. They are defined within a Pepr module and can use MutatingWebhookConfigurations, ValidatingWebhookConfigurations, watchers, queues, or schedules depending on the actions they contain. A Capability can have a specific scope, such as mutating or validating, and can be reused in multiple Pepr modules. See [Capabilities](user-guide/capabilities) for more details. ### Action [Section titled “Action”](#action) Action is a discrete set of behaviors defined in a single function that acts on a given Kubernetes GroupVersionKind (GVK) passed in from Kubernetes. Actions are the atomic operations that are performed on Kubernetes resources by Pepr. For example, an action could be responsible for adding a specific label to a Kubernetes resource, or for modifying a specific field in a resource’s metadata. Actions can be grouped together within a Capability to provide a more comprehensive set of operations that can be performed on Kubernetes resources. There are both `Mutate()` and `Validate()` Actions that can be used to modify or validate Kubernetes resources within the admission controller lifecycle. There are also `Watch()` and `Reconcile()` actions that can be used to watch for changes to Kubernetes resources that already exist. Finally, the `Finalize()` can be used after `Watch()` or `Reconcile()` to perform cleanup operations when the resource is deleted. See [actions](actions) for more details. ## Logical Pepr Flow [Section titled “Logical Pepr Flow”](#logical-pepr-flow)  ## TypeScript [Section titled “TypeScript”](#typescript) [TypeScript](https://www.typescriptlang.org/) is a strongly typed, object-oriented programming language built on top of JavaScript. It provides optional static typing and a rich type system, allowing developers to write more robust code. TypeScript is transpiled to JavaScript, enabling it to run in any environment that supports JavaScript. Pepr allows you to use JavaScript or TypeScript to write capabilities, but TypeScript is recommended for its type safety and rich type system. See the [TypeScript docs](https://www.typescriptlang.org/docs/handbook/typescript-from-scratch.html) to learn more. ## Community [Section titled “Community”](#community) To join our channel go to [Kubernetes Slack](https://inviter.co/kubernetes) and join the `#pepr` channel. [](https://github.com/defenseunicorns/pepr/graphs/contributors) Made with [contrib.rocks](https://contrib.rocks). # Overview > Overview # Actions [Section titled “Actions”](#actions) ## Overview [Section titled “Overview”](#overview) An action is a discrete set of behaviors defined in a single function that acts on a given Kubernetes GroupVersionKind (GVK) passed in during the admission controller lifecycle. Actions are the atomic operations that are performed on Kubernetes resources by Pepr. For example, an action could be responsible for adding a specific label to a Kubernetes resource, or for modifying a specific field in a resource’s metadata. Actions can be grouped together within a Capability to provide a more comprehensive set of operations that can be performed on Kubernetes resources. Actions are `Mutate()`, `Validate()`, `Watch()`, `Reconcile()`, and `Finalize()`. Both Mutate and Validate actions run during the admission controller lifecycle, while Watch and Reconcile actions run in a separate controller that tracks changes to resources, including existing resource; the Finalize action spans both the admission & afterward. Let’s look at some example actions that are included in the `HelloPepr` capability that is created for you when you [`npx pepr init`](../user-guide/pepr-cli#pepr-init): *** In this first example, Pepr is adding a label and annotation to a ConfigMap with the name `example-1` when it is created. Comments are added to each line to explain in more detail what is happening. ```ts // When(a.) filters which GroupVersionKind (GVK) this action should act on. When(a.ConfigMap) // This limits the action to only act on new resources. .IsCreated() // This limits the action to only act on resources with the name "example-1". .WithName("example-1") // Mutate() is where we define the actual behavior of this action. .Mutate(request => { // The request object is a wrapper around the K8s resource that Pepr is acting on. request // Here we are adding a label to the ConfigMap. .SetLabel("pepr", "was-here") // And here we are adding an annotation. .SetAnnotation("pepr.dev", "annotations-work-too"); // Note that we are not returning anything here. This is because Pepr is tracking the changes in each action automatically. }); ``` *** In this example, a Validate action rejects any ConfigMap in the `pepr-demo` namespace that has no data. ```ts When(a.ConfigMap) .IsCreated() .InNamespace("pepr-demo") // Validate() is where we define the actual behavior of this action. .Validate(request => { // If data exists, approve the request. if (request.Raw.data) { return request.Approve(); } // Otherwise, reject the request with a message and optional code. return request.Deny("ConfigMap must have data"); }); ``` *** In this example, a Watch action on the name and phase of any ConfigMap.Watch actions run in a separate controller that tracks changes to resources, including existing resources so that you can react to changes in real-time. It is important to note that Watch actions are not run during the admission controller lifecycle, so they cannot be used to modify or validate resources. They also may run multiple times for the same resource, so it is important to make sure that your Watch actions are idempotent. In a future release, Pepr will provide a better way to control when a Watch action is run to avoid this issue. ```ts When(a.ConfigMap) // Watch() is where we define the actual behavior of this action. .Watch((cm, phase) => { Log.info({cm}, `ConfigMap ${cm.metadata.name} was ${phase}`); }); ``` There are many more examples in the `HelloPepr` capability that you can use as a reference when creating your own actions. Note that each time you run [`npx pepr update`](../user-guide/pepr-cli#pepr-update), Pepr will automatically update the `HelloPepr` capability with the latest examples and best practices for you to reference and test directly in your Pepr Module. *** In some scenarios involving Kubernetes Resource Controllers or Operator patterns, opting for a Reconcile action could be more fitting. Comparable to the Watch functionality, Reconcile is responsible for monitoring the name and phase of any Kubernetes Object. It operates within the Watch controller dedicated to observing modifications to resources, including those already existing, enabling responses to alterations as they occur. Unlike Watch, however, Reconcile employs a Queue to sequentially handle events once they are returned by the Kubernetes API. This allows the operator to handle bursts of events without overwhelming the system or the Kubernetes API. It provides a mechanism to back off when the system is under heavy load, enhancing overall stability and maintaining the state consistency of Kubernetes resources, as the order of operations can impact the final state of a resource. ```ts When(WebApp) .IsCreatedOrUpdated() .Validate(validator) .Reconcile(async instance => { const { namespace, name, generation } = instance.metadata; if (!instance.metadata?.namespace) { Log.error({instance}, `Invalid WebApp definition`); return; } const isPending = instance.status?.phase === Phase.Pending; const isCurrentGeneration = generation === instance.status?.observedGeneration; if (isPending || isCurrentGeneration) { Log.debug({instance}, `Skipping pending or completed instance`); return; } Log.debug({instance}, `Processing instance ${namespace}/${name}`); try { // Set Status to pending await updateStatus(instance, { phase: Phase.Pending }); // Deploy Deployment, ConfigMap, Service, ServiceAccount, and RBAC based on instance await Deploy(instance); // Set Status to ready await updateStatus(instance, { phase: Phase.Ready, observedGeneration: instance.metadata.generation, }); } catch (e) { Log.error(e, `Error configuring for ${namespace}/${name}`); // Set Status to failed void updateStatus(instance, { phase: Phase.Failed, observedGeneration: instance.metadata.generation, }); } }); ``` # Finalize > Finalize # Finalize [Section titled “Finalize”](#finalize) A specialized combination of Pepr’s [Mutate](../mutate) & [Watch](../watch) functionalities that allow a module author to run logic while Kubernetes is [Finalizing](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/) a resource (i.e. cleaning up related resources *after* a deletion request has been accepted). `Finalize()` can only be accessed after a `Watch()` or `Reconcile()`. This method will: 1. Inject a finalizer into the `metadata.finalizers` field of the requested resource during the mutation phase of the admission. 2. Watch appropriate resource lifecycle events & invoke the given callback. 3. Remove the injected finalizer from the `metadata.finalizers` field of the requested resource. ```ts When(a.ConfigMap) .IsCreated() .InNamespace("hello-pepr-finalize-create") .WithName("cm-reconcile-create") .Reconcile(function reconcileCreate(cm) { Log.info({cm}, "external api call (create): reconcile/callback") }) .Finalize(function finalizeCreate(cm) { Log.info({cm}, "external api call (create): reconcile/finalize") }); ``` # Mutate > Mutate # Mutate [Section titled “Mutate”](#mutate) Mutating admission webhooks are invoked first and can modify objects sent to the API server to enforce custom defaults. After an object is sent to Pepr’s Mutating Admission Webhook, Pepr will [annotate the object](https://github.com/defenseunicorns/pepr/blob/f01f5eeda16c13ecd0d51b26b8a16ed7e4c1b080/src/lib/mutate-processor.ts#L64) to indicate the status. After a successful mutation of an object in a module with UUID static-test, and capability name hello-pepr, expect to see this annotation: `static-test.pepr.dev/hello-pepr: succeeded`. ## Mutate Helpers [Section titled “Mutate Helpers”](#mutate-helpers) ### `SetLabel` [Section titled “SetLabel”](#setlabel) `SetLabel` is used to set a label on a Kubernetes object as part of a Pepr Mutate action. For example, to add a label when a ConfigMap is created: ```typescript When(a.ConfigMap) .IsCreated() .Mutate(request => { request // Here we are adding a label to the ConfigMap. .SetLabel("pepr", "was-here") // Note that we are not returning anything here. This is because Pepr is tracking the changes in each action automatically. }); ``` ### `RemoveLabel` [Section titled “RemoveLabel”](#removelabel) `RemoveLabel` is used to remove a label on a Kubernetes object as part of a Pepr Mutate action. For example, to remove a label when a ConfigMap is updated: ```typescript When(a.ConfigMap) .IsCreated() .Mutate(request => { request // Here we are removing a label from the ConfigMap. .RemoveLabel("remove-me") // Note that we are not returning anything here. This is because Pepr is tracking the changes in each action automatically. }); ``` ### `SetAnnotation` [Section titled “SetAnnotation”](#setannotation) `SetAnnotation` is used to set an annotation on a Kubernetes object as part of a Pepr Mutate action. For example, to add an annotation when a ConfigMap is created: ```typescript When(a.ConfigMap) .IsCreated() .Mutate(request => { request // Here we are adding an annotation to the ConfigMap. .SetAnnotation("pepr.dev", "annotations-work-too"); // Note that we are not returning anything here. This is because Pepr is tracking the changes in each action automatically. }); ``` ### `RemoveAnnotation` [Section titled “RemoveAnnotation”](#removeannotation) `RemoveAnnotation` is used to remove an annotation on a Kubernetes object as part of a Pepr Mutate action. For example, to remove an annotation when a ConfigMap is updated: ```typescript When(a.ConfigMap) .IsUpdated() .Mutate(request => { request // Here we are removing an annotation from the ConfigMap. .RemoveAnnotation("remove-me"); // Note that we are not returning anything here. This is because Pepr is tracking the changes in each action automatically. }); ``` ## See Also [Section titled “See Also”](#see-also) Looking for some more generic helpers? Check out the [Module Author SDK](../../user-guide/sdk) for information on other things that Pepr can help with. # Reconcile > Reconcile # Reconcile [Section titled “Reconcile”](#reconcile) Reconcile functions the same as Watch but is tailored for building Kubernetes Controllers and Operators because it processes callback operations in a [Queue](https://github.com/defenseunicorns/pepr/blob/f01f5eeda16c13ecd0d51b26b8a16ed7e4c1b080/src/lib/watch-processor.ts#L86), guaranteeing ordered and synchronous processing of events, even when the system may be under heavy load. Ordering can be configured to operate in one of two ways: as a single queue that maintains ordering of operations across all resources of a kind (default) or with separate processing queues per resource instance. See [Configuring Reconcile](../../user-guide/customization#configuring-reconcile) for more on configuring how Reconcile behaves. ```ts When(WebApp) .IsCreatedOrUpdated() .Validate(validator) // Validate the shape of the resource .Reconcile(async instance => { try { Store.setItem(instance.metadata.name, JSON.stringify(instance)); await reconciler(instance); // Reconcile the resource - Deploy Kubernetes manifests, etc. } catch (error) { Log.info(`Error reconciling instance of WebApp`); } }); export async function validator(req: PeprValidateRequest) { const ns = req.Raw.metadata?.namespace ?? "_unknown_"; if (req.Raw.spec.replicas > 7) { return req.Deny("max replicas is 7 to prevent noisy neighbors"); } if (invalidNamespaces.includes(ns)) { if (req.Raw.metadata?.namespace === "default") { return req.Deny("default namespace is not allowed"); } return req.Deny("invalid namespace"); } return req.Approve(); } ``` # Using Alias Child Logger in Actions > Using Alias Child Logger in Actions # Using Alias Child Logger in Actions [Section titled “Using Alias Child Logger in Actions”](#using-alias-child-logger-in-actions) You can use the Alias function to include a user-defined alias in the logs for Mutate, Validate, and Watch actions. This can make for easier debugging since your user-defined alias will be included in the action’s logs. This is especially useful when you have multiple actions of the same type in a single module. The below example uses Mutate, Validate, and Watch actions with the Alias function: ```ts When(a.Pod) .IsCreatedOrUpdated() .Alias("mutate") .Mutate((po, logger) => { logger.info(`alias: mutate ${po.Raw.metadata.name}`); }); When(a.Pod) .IsCreatedOrUpdated() .Alias("validate") .Validate((po, logger) => { logger.info(`alias: validate ${po.Raw.metadata.name}`); return po.Approve(); }); When(a.Pod) .IsCreatedOrUpdated() .Alias("watch") .Watch((po, _, logger) => { logger.info(`alias: watch ${po.metadata.name}`); }); When(a.Pod) .IsCreatedOrUpdated() .Alias("reconcile") .Reconcile((po, _, logger) => { logger.info(`alias: reconcile ${po.metadata.name}`); }); ``` This will result in log entries when creating a Pod with the that include the alias: **Logs for Mutate When Pod `red` is Created:** ```bash {"level":30,"time":1726632368808,"pid":16,"hostname":"pepr-static-test-6786948977-6hbnt","uid":"b2221631-e87c-41a2-94c8-cdaef15e7b5f","namespace":"pepr-demo","name":"/red","gvk":{"group":"","version":"v1","kind":"Pod"},"operation":"CREATE","admissionKind":"Mutate","msg":"Incoming request"} {"level":30,"time":1726632368808,"pid":16,"hostname":"pepr-static-test-6786948977-6hbnt","uid":"b2221631-e87c-41a2-94c8-cdaef15e7b5f","namespace":"pepr-demo","name":"/red","msg":"Processing request"} {"level":30,"time":1726632368808,"pid":16,"hostname":"pepr-static-test-6786948977-6hbnt","msg":"Executing mutation action with alias: mutate"} {"level":30,"time":1726632368808,"pid":16,"hostname":"pepr-static-test-6786948977-6hbnt","alias":"mutate","msg":"alias: mutate red"} {"level":30,"time":1726632368808,"pid":16,"hostname":"pepr-static-test-6786948977-6hbnt","uid":"b2221631-e87c-41a2-94c8-cdaef15e7b5f","namespace":"pepr-demo","name":"hello-pepr","msg":"Mutation action succeeded (mutateCallback)"} {"level":30,"time":1726632368808,"pid":16,"hostname":"pepr-static-test-6786948977-6hbnt","uid":"b2221631-e87c-41a2-94c8-cdaef15e7b5f","namespace":"pepr-demo","name":"/red","res":{"uid":"b2221631-e87c-41a2-94c8-cdaef15e7b5f","allowed":true,"patchType":"JSONPatch","patch":"W3sib3AiOiJhZGQiLCJwYXRoIjoiL21ldGFkYXRhL2Fubm90YXRpb25zL3N0YXRpYy10ZXN0LnBlcHIuZGV2fjFoZWxsby1wZXByIiwidmFsdWUiOiJzdWNjZWVkZWQifV0="},"msg":"Check response"} {"level":30,"time":1726632368809,"pid":16,"hostname":"pepr-static-test-6786948977-6hbnt","uid":"b2221631-e87c-41a2-94c8-cdaef15e7b5f","method":"POST","url":"/mutate/c1a7fb6e3f2ab9dc08909d2de4166987520f317d53b759ab882dfd0b1c198479?timeout=10s","status":200,"duration":"1 ms"} ``` **Logs for Validate When Pod `red` is Created:** ```bash {"level":30,"time":1726631437605,"pid":16,"hostname":"pepr-static-test-6786948977-j7f9h","uid":"731eff93-d457-4ffc-a98c-0bcbe4c1727a","namespace":"pepr-demo","name":"/red","gvk":{"group":"","version":"v1","kind":"Pod"},"operation":"CREATE","admissionKind":"Validate","msg":"Incoming request"} {"level":30,"time":1726631437606,"pid":16,"hostname":"pepr-static-test-6786948977-j7f9h","uid":"731eff93-d457-4ffc-a98c-0bcbe4c1727a","namespace":"pepr-demo","name":"/red","msg":"Processing validation request"} {"level":30,"time":1726631437606,"pid":16,"hostname":"pepr-static-test-6786948977-j7f9h","uid":"731eff93-d457-4ffc-a98c-0bcbe4c1727a","namespace":"pepr-demo","name":"hello-pepr","msg":"Processing validation action (validateCallback)"} {"level":30,"time":1726631437606,"pid":16,"hostname":"pepr-static-test-6786948977-j7f9h","msg":"Executing validate action with alias: validate"} {"level":30,"time":1726631437606,"pid":16,"hostname":"pepr-static-test-6786948977-j7f9h","alias":"validate","msg":"alias: validate red"} {"level":30,"time":1726631437606,"pid":16,"hostname":"pepr-static-test-6786948977-j7f9h","uid":"731eff93-d457-4ffc-a98c-0bcbe4c1727a","namespace":"pepr-demo","name":"hello-pepr","msg":"Validation action complete (validateCallback): allowed"} {"level":30,"time":1726631437606,"pid":16,"hostname":"pepr-static-test-6786948977-j7f9h","uid":"731eff93-d457-4ffc-a98c-0bcbe4c1727a","namespace":"pepr-demo","name":"/red","res":{"uid":"731eff93-d457-4ffc-a98c-0bcbe4c1727a","allowed":true},"msg":"Check response"} {"level":30,"time":1726631437606,"pid":16,"hostname":"pepr-static-test-6786948977-j7f9h","uid":"731eff93-d457-4ffc-a98c-0bcbe4c1727a","method":"POST","url":"/validate/c1a7fb6e3f2ab9dc08909d2de4166987520f317d53b759ab882dfd0b1c198479?timeout=10s","status":200,"duration":"5 ms"} ``` **Logs for Watch and Reconcile When Pod `red` is Created:** ```bash {"level":30,"time":1726798504518,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","msg":"Executing reconcile action with alias: reconcile"} {"level":30,"time":1726798504518,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","alias":"reconcile","msg":"alias: reconcile red"} {"level":30,"time":1726798504518,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","msg":"Executing watch action with alias: watch"} {"level":30,"time":1726798504518,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","alias":"watch","msg":"alias: watch red"} {"level":30,"time":1726798504521,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","msg":"Executing reconcile action with alias: reconcile"} {"level":30,"time":1726798504521,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","alias":"reconcile","msg":"alias: reconcile red"} {"level":30,"time":1726798504521,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","msg":"Executing watch action with alias: watch"} {"level":30,"time":1726798504521,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","alias":"watch","msg":"alias: watch red"} {"level":30,"time":1726798504528,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","msg":"Executing reconcile action with alias: reconcile"} {"level":30,"time":1726798504528,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","alias":"reconcile","msg":"alias: reconcile red"} {"level":30,"time":1726798504528,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","msg":"Executing watch action with alias: watch"} {"level":30,"time":1726798504528,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","alias":"watch","msg":"alias: watch red"} {"level":30,"time":1726798510464,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","msg":"Executing watch action with alias: watch"} {"level":30,"time":1726798510464,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","alias":"watch","msg":"alias: watch red"} {"level":30,"time":1726798510466,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","msg":"Executing reconcile action with alias: reconcile"} {"level":30,"time":1726798510466,"pid":16,"hostname":"pepr-static-test-watcher-6dc69654c9-5ql6b","alias":"reconcile","msg":"alias: reconcile red"} ``` > \[!NOTE] The Alias function is optional and can be used to provide additional context in the logs. You must pass the logger object as shown above to the action to use the Alias function. ## See Also [Section titled “See Also”](#see-also) Looking for some more generic helpers? Check out the [Module Author SDK](../../user-guide/sdk) for information on other things that Pepr can help with. # Validate > Validate # Validate [Section titled “Validate”](#validate) After the Mutation phase comes the Validation phase where the validating admission webhooks are invoked and can reject requests to enforce custom policies. Validate does not annotate the objects that are allowed into the cluster, but the validation webhook can be audited with `npx pepr monitor`. Read the [monitoring docs](../../reference/best-practices#monitoring) for more information. ## Basic Validation [Section titled “Basic Validation”](#basic-validation) Validation actions can either approve or deny requests: ```ts When(a.ConfigMap) .IsCreated() .Validate(request => { if (request.HasAnnotation("evil")) { return request.Deny("No evil CM annotations allowed.", 400); } return request.Approve(); }); ``` ## Validation with Warnings [Section titled “Validation with Warnings”](#validation-with-warnings) Pepr supports including warning messages in both approval and denial responses. Warnings provide a way to communicate important information to users without necessarily blocking their requests. ### Approving with Warnings [Section titled “Approving with Warnings”](#approving-with-warnings) ```ts When(a.ConfigMap) .IsCreatedOrUpdated() .InNamespace("pepr-demo") .Validate(request => { const warnings = []; // Check for deprecated fields if (request.Raw.data && request.Raw.data["deprecated-field"]) { warnings.push("Warning: The 'deprecated-field' is being used and will be removed in future versions"); } // Check for missing app label if (!request.HasLabel("app")) { warnings.push("Warning: Best practice is to include an 'app' label for resource identification"); } // Return approval with warnings if any were generated return request.Approve(warnings.length > 0 ? warnings : undefined); }); ``` ### Denying with Warnings [Section titled “Denying with Warnings”](#denying-with-warnings) ```ts When(a.ConfigMap) .IsCreatedOrUpdated() .InNamespace("pepr-demo") .Validate(request => { // Check for dangerous settings if (request.Raw.data && request.Raw.data["dangerous-setting"] === "true") { const warnings = [ "Warning: The 'dangerous-setting' field is set to 'true'", "Consider using a safer configuration option" ]; return request.Deny( "ConfigMap contains dangerous settings that are not allowed", 422, warnings ); } return request.Approve(); }); ``` Warnings will be included in the Kubernetes API response and can be displayed to users by `kubectl` and other Kubernetes clients, providing helpful feedback while still enforcing policies. # Watch > Watch # Watch [Section titled “Watch”](#watch) [Kubernetes](https://kubernetes.io/docs/reference/using-api/api-concepts) supports efficient change notifications on resources via watches. Pepr uses the Watch action for monitoring resources that previously existed in the cluster and for performing long-running asynchronous events upon receiving change notifications on resources, as watches are not limited by [timeouts](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#timeouts). ```ts When(a.Namespace) .IsCreated() .WithName("pepr-demo-2") .Watch(async ns => { Log.info("Namespace pepr-demo-2 was created."); try { // Apply the ConfigMap using K8s server-side apply await K8s(kind.ConfigMap).Apply({ metadata: { name: "pepr-ssa-demo", namespace: "pepr-demo-2", }, data: { "ns-uid": ns.metadata.uid, }, }); } catch (error) { // You can use the Log object to log messages to the Pepr controller pod Log.error({error}, "Failed to apply ConfigMap using server-side apply."); } // You can share data between actions using the Store, including between different types of actions Store.setItem("watch-data", "This data was stored by a Watch Action."); }); ``` # Community and Support > Community and Support # Community and Support [Section titled “Community and Support”](#community-and-support) ## Introduction [Section titled “Introduction”](#introduction) Pepr is a community-driven project. We welcome contributions of all kinds, from bug reports to feature requests to code changes. We also welcome contributions of documentation, tutorials, and examples. ## Contributing [Section titled “Contributing”](#contributing) You can find all the details on contributing to Pepr at: * [Contributing to Pepr](../../contribute/contributor-guide) ### Reporting Bugs [Section titled “Reporting Bugs”](#reporting-bugs) Information on reporting bugs can be found at: * [Reporting Bugs](../support) ### Reporting Security Issues [Section titled “Reporting Security Issues”](#reporting-security-issues) Information on reporting security issues can be found at: * [Reporting Security Issues](../security) # Pepr Media > Pepr Media # Pepr Media [Section titled “Pepr Media”](#pepr-media) ## 2025 [Section titled “2025”](#2025) ### Presentations [Section titled “Presentations”](#presentations) * Building Smarter Kubernetes Workflows: Pepr for the modern SRE * [Shipping Software faster and Safer with Pepr](https://open.spotify.com/episode/3ZtXKLCsMN0hFHXCinCZVT?si=c824f1ace8ca4960) ## 2024 [Section titled “2024”](#2024) ### Session Recordings [Section titled “Session Recordings”](#session-recordings) * [Leveraging Fluent Ecosystems: Leveraging Fluent, Flexible APIs for Admission Control Strategies](https://www.youtube.com/watch?v=32xhSJPMtBU\&t=25s) * [Optimizing Kubernetes Operators and Admission Control - Open Source Summit NA 2024](https://www.youtube.com/watch?v=SqJfvQ1frr4) * [Pepr Analysis and Demo - YouTube](https://www.youtube.com/watch?v=kgSfS68hzbY) ### Workshops [Section titled “Workshops”](#workshops) * [Pepr Scenario Killer Koda](https://killercoda.com/decoder/scenario/pepr) * [DevopsDaysRaleigh Pepr Workshop - Building an enterprise admission controller](https://github.com/cmwylie19/enterprise-admission-controller) ### Blogs [Section titled “Blogs”](#blogs) * [From YAML Chaos to Kubernetes Zen](https://itnext.io/from-yaml-chaos-to-kubernetes-zen-5b9e8242a5cb) ## 2023 [Section titled “2023”](#2023) ### Session Recordings [Section titled “Session Recordings”](#session-recordings-1) * [Optimizing Kubernetes Ecosystems: Leveraging Fluent, Flexible APIs for Admission Control Strategies](https://www.youtube.com/watch?v=32xhSJPMtBU\&t=12s) * [Getting Started with Defense Unicorns Integrating Capabilities with Pepr](https://www.youtube.com/watch?v=2kHIMo0ITo4) # Security Policy > Security Policy # Security Policy [Section titled “Security Policy”](#security-policy) ## Security Report [Section titled “Security Report”](#security-report) [](https://securityscorecards.dev/viewer/?uri=github.com/defenseunicorns/pepr) Pepr strives to achieve and maintain top scores on the [OSSF Scorecard](https://securityscorecards.dev/viewer/?uri=github.com/defenseunicorns/pepr). Rest assured, Pepr is secure; however, it is possible for users to introduce their own vulnerabilities. ## User Responsibilities [Section titled “User Responsibilities”](#user-responsibilities) While Pepr itself is designed with security in mind, users can introduce vulnerabilities through insecure dependencies, hardcoded secrets, unscanned container images, excessive permissions, and inadequate testing. Always audit npm packages, use proper secret management, scan controller images regularly, follow least privilege principles for RBAC, and implement code reviews before deploying to production. ## Reporting a Vulnerability [Section titled “Reporting a Vulnerability”](#reporting-a-vulnerability) If you discover a security vulnerability in Pepr, please report it to us by sending an email to [security@defenseunicorns.com](mailto:security@defenseunicorns.com?subject=vulnerability) or directly through the [GitHub UI](https://github.com/defenseunicorns/pepr/security/advisories/new). Please include the following details in your report: * A clear description of the vulnerability * Steps to reproduce the vulnerability * Any additional information that may be helpful in understanding and fixing the issue We appreciate your help in making Pepr more secure and will acknowledge your contribution in the remediation PR. ## Contact [Section titled “Contact”](#contact) If you have any questions or concerns regarding the security of Pepr, please contact us at . # Support > Support # Support [Section titled “Support”](#support) ## Reporting Bugs [Section titled “Reporting Bugs”](#reporting-bugs) If you find a bug in Pepr, please report it by opening an issue in the [Pepr GitHub repository](https://github.com/defenseunicorns/pepr/issues). Please include as much information as possible in your bug report, including: * The version of Pepr you are using * The version of Kubernetes you are using ## Contact [Section titled “Contact”](#contact) You can contact the Pepr team in the following ways: * [Slack](https://kubernetes.slack.com/archives/C06DGH40UCB) * [GitHub](https://github.com/defenseunicorns/pepr) * [Email](mailto:security@defenseunicorns.com) # Contributor Covenant Code of Conduct > Contributor Covenant Code of Conduct # Contributor Covenant Code of Conduct [Section titled “Contributor Covenant Code of Conduct”](#contributor-covenant-code-of-conduct) ## Our Pledge [Section titled “Our Pledge”](#our-pledge) We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. ## Our Standards [Section titled “Our Standards”](#our-standards) Examples of behavior that contributes to a positive environment for our community include: * Demonstrating empathy and kindness toward other people * Being respectful of differing opinions, viewpoints, and experiences * Giving and gracefully accepting constructive feedback * Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience * Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: * The use of sexualized language or imagery, and sexual attention or advances of any kind * Trolling, insulting or derogatory comments, and personal or political attacks * Public or private harassment * Publishing others’ private information, such as a physical or email address, without their explicit permission * Other conduct which could reasonably be considered inappropriate in a professional setting ## Enforcement Responsibilities [Section titled “Enforcement Responsibilities”](#enforcement-responsibilities) Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate. ## Scope [Section titled “Scope”](#scope) This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline event. ## Enforcement [Section titled “Enforcement”](#enforcement) Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at . All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident. ## Enforcement Guidelines [Section titled “Enforcement Guidelines”](#enforcement-guidelines) Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct: ### 1. Correction [Section titled “1. Correction”](#1-correction) **Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. **Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested. ### 2. Warning [Section titled “2. Warning”](#2-warning) **Community Impact**: A violation through a single incident or series of actions. **Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. ### 3. Temporary Ban [Section titled “3. Temporary Ban”](#3-temporary-ban) **Community Impact**: A serious violation of community standards, including sustained inappropriate behavior. **Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. ### 4. Permanent Ban [Section titled “4. Permanent Ban”](#4-permanent-ban) **Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. **Consequence**: A permanent ban from any sort of public interaction within the community. ## Attribution [Section titled “Attribution”](#attribution) This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org), version 2.1, available at . Community Impact Guidelines were inspired by [Mozilla’s code of conduct enforcement ladder](https://github.com/mozilla/diversity). For answers to common questions about this code of conduct, see the FAQ at . Translations are available at . # Contributor Guide > Contributor Guide # Contributor Guide [Section titled “Contributor Guide”](#contributor-guide) Thank you for your interest in contributing to Pepr! We welcome all contributions and are grateful for your help. This guide outlines how to get started with contributing to this project. ## Table of Contents [Section titled “Table of Contents”](#table-of-contents) * [Contributor Guide](#contributor-guide) * [Table of Contents](#table-of-contents) * [Code of Conduct](#code-of-conduct) * [Getting Started](#getting-started) * [Setup](#setup) * [Kubernetes Fluent Client Contributions](#kubernetes-fluent-client-contributions) * [Submitting a Pull Request](#submitting-a-pull-request) * [PR Requirements](#pr-requirements) * [Coding Guidelines](#coding-guidelines) * [Git Hooks](#git-hooks) * [Running Tests](#running-tests) * [Run Tests Locally](#run-tests-locally) * [Test a Local Development Version](#test-a-local-development-version) * [Contact](#contact) ## Code of Conduct [Section titled “Code of Conduct”](#code-of-conduct) Please follow our [Code of Conduct](../code-of-conduct) to maintain a respectful and collaborative environment. ## Getting Started [Section titled “Getting Started”](#getting-started) * **Repository**: * **npm package**: * **Required Node version**: `>=22.19.0` ### Setup [Section titled “Setup”](#setup) 1. Fork the repository. 2. Clone your fork locally: `git clone https://github.com/your-username/pepr.git`. 3. Install dependencies: `npm ci`. 4. Create a new branch for your feature or fix: `git checkout -b my-feature-branch`. ### Kubernetes Fluent Client Contributions [Section titled “Kubernetes Fluent Client Contributions”](#kubernetes-fluent-client-contributions) [Kubernetes Fluent Client](https://github.com/defenseunicorns/kubernetes-fluent-client) is a library used by Pepr that provides a fluent interface for Kubernetes API clients. In Pepr, we use the `kubernetes-fluent-client` package to interact with Kubernetes resources. Due to the nature of this library, it is important to ensure that any changes made to the Kubernetes Fluent Client are thoroughly tested and validated in Pepr before being merged into the codebase. In particular, we need to ensure that the changes do not break any existing functionality or introduce new bugs, especially in the context of the Pepr Watcher. If you are making changes to the Kubernetes Fluent Client, please ensure that you run the Soak Test in GitHub Actions to validate your changes. To run the Soak Test, you can do the following: * Go to the [GitHub Actions tab](https://github.com/defenseunicorns/pepr/actions) * Select the workflow named “Soak Test” * Click on the “Run workflow” button to get the options to run the workflow * Select the Kubernetes Fluent Client branch you want to test (`KFC dev branch`) and click “Run workflow” ## Submitting a Pull Request [Section titled “Submitting a Pull Request”](#submitting-a-pull-request) 1. **Create an Issue**: For significant changes, please create an issue first, describing the problem or feature proposal. Trivial fixes do not require an issue. 2. **Commit Your Changes**: Make your changes and commit them. All commits must be signed. 3. **Run Tests**: Ensure that your changes pass all tests by running `npm test`. 4. **Push Your Branch**: Push your branch to your fork on GitHub. 5. **Create a Pull Request**: Open a pull request against the `main` branch of the Pepr repository. Please make sure that your PR passes all CI checks. ### PR Requirements [Section titled “PR Requirements”](#pr-requirements) * PRs must be against the `main` branch. * PRs must pass CI checks. * All commits must be signed. * PRs should have a related issue, except for trivial fixes. We take PR reviews seriously and strive to provide a great contributor experience with timely feedback. To help maintain this, we ask external contributors to limit themselves to no more than two open PRs at a time. Having too many open PRs can slow down the review process and impact the quality of feedback ## Coding Guidelines [Section titled “Coding Guidelines”](#coding-guidelines) Please follow the coding conventions and style used in the project. Use ESLint and Prettier for linting and formatting: * Check formatting: `npm run format:check` * Fix formatting: `npm run format:fix` * If regex is used, provide a link to regex101.com with an explanation of the regex pattern. * Do not use emoji in logs or comments, as it can be distracting and is not consistent with the project’s style. ### Git Hooks [Section titled “Git Hooks”](#git-hooks) * This project uses [husky](https://typicode.github.io/husky/) to manage git hooks for pre-commit and pre-push actions. * pre-commit will automatically run linters so that you don’t need to remember to run `npm run format:*` commands * pre-push will warn you if you’ve changed lots of lines on a branch and encourage you to optionally present the changes as several smaller PRs to facilitate easier PR reviews. * The pre-push hook is an opinionated way of working, and is therefore optional. * You can opt-in to using the pre-push hook by setting `PEPR_HOOK_OPT_IN=1` as an environment variable. ## Automated Workflows [Section titled “Automated Workflows”](#automated-workflows) ### Grype Suppression Audit [Section titled “Grype Suppression Audit”](#grype-suppression-audit) A weekly GitHub Actions workflow ([`grype-suppression-audit.yaml`](https://github.com/defenseunicorns/pepr/blob/main/.github/workflows/grype-suppression-audit.yaml)) scans the Pepr controller image for stale CVE suppressions in `.grype.yaml`. When stale entries are found, the workflow removes them and opens (or updates) a PR on the bot-owned branch `grype/suppression-audit`. * **Do not push manual commits to `grype/suppression-audit`** — the workflow force-pushes to this branch weekly and any manual changes will be overwritten. * To add context to a suppression entry (e.g. `reason:`), add it directly to `.grype.yaml` on `main`. The audit script skips multi-key entries and will not remove them automatically. ### Peer-Dependency Updates [Section titled “Peer-Dependency Updates”](#peer-dependency-updates) The `peerDependencies` block in `package.json` is maintained by the [`peer-deps-update`](https://github.com/defenseunicorns/pepr/blob/main/.github/workflows/peer-deps-update.yml) GitHub Actions workflow. The workflow runs every Monday and opens PRs grouped by SemVer risk: a single PR for all minor/patch bumps and a separate PR per major-version bump (so that majors get isolated review and revert paths). * To trigger an off-cycle run, dispatch the workflow manually from the Actions tab. * To inspect the latest peer-dep candidates locally without opening a PR, run `node scripts/update-peer-deps.mjs report`. * To apply bumps locally, run `node scripts/update-peer-deps.mjs apply --kind minor` for the minor/patch group or `node scripts/update-peer-deps.mjs apply --kind major --pkg ` for a single major bump. Dependabot continues to handle the `dependencies` and `devDependencies` blocks; the two automations never touch the same region of `package.json`. ## Running Tests [Section titled “Running Tests”](#running-tests) ### Run Tests Locally [Section titled “Run Tests Locally”](#run-tests-locally) * Run all tests: `npm test` ### Test a Local Development Version [Section titled “Test a Local Development Version”](#test-a-local-development-version) 1. Run `npm test` and wait for completion. 2. Change to the test module directory: `cd pepr-test-module`. 3. You can now run any of the `npx pepr` commands. ### Running Development Version Locally [Section titled “Running Development Version Locally”](#running-development-version-locally) 1. Run `npm run build` to build the package. 2. For running modified pepr, you have two options: 1. Using `npx ts-node ./src/cli.ts init` to run the modified code directly, without installing it locally. You’ll need to also run `npx link ` inside your pepr module, to link to the development version of pepr. 2. Install the pre-build package with `npm install pepr-0.0.0-development.tgz`. You’ll need to re-run the installation after every build, though. 3. Run `npx pepr dev` inside your module’s directory to run the modified version of pepr. > \[!TIP] Make sure to re-run `npm run build` after you modify any of the pepr source files. ## Contact [Section titled “Contact”](#contact) For any questions or concerns, please open an issue on GitHub or contact the maintainers. # Finalize > Finalize # Action: Finalize [Section titled “Action: Finalize”](#action-finalize) Pepr `Finalize()` action runs after a Kubernetes resource is deleted but before it is fully removed. Use it to perform any required pre-deletion logic—cleanup external resources, update systems, or gracefully teardown dependent processes. Finalizers run *after* a delete request but before the object is removed. ## When should you use Finalize? [Section titled “When should you use Finalize?”](#when-should-you-use-finalize) * Clean up external resources tied to a Kubernetes object (cloud assets, external systems, files, credentials, etc.). * Perform teardown steps before the object is permanently removed. * Ensure that dependent systems are updated or notified upon deletion. * Delay final resource removal until cleanup is safely completed. * Implement lightweight cleanup behaviors without writing a full Kubernetes operator. > \[!TIP] `Finalize()` can only be accessed after a `Watch()` or `Reconcile()` ## Code Snippet Examples [Section titled “Code Snippet Examples”](#code-snippet-examples) This example registers a finalizer on a Deployment and performs cleanup when the resource is deleted. > **View full example on [GitHub](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-finalize/capabilities/finalize.ts)** ### Create Event [Section titled “Create Event”](#create-event) ```typescript When(a.ConfigMap) .IsCreated() .InNamespace("hello-pepr-finalize-create") .WithName("cm-reconcile-create") .Reconcile(function reconcileCreate(cm) { Log.info(cm, "external api call (create): reconcile/callback"); }) .Finalize(function finalizeCreate(cm) { Log.info(cm, "external api call (create): reconcile/finalize"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output) ```json {"level":30,"time":,"pid":,"hostname":"pepr--watcher-","metadata":,"msg":"external api call (create): reconcile/callback"} {"level":30,"time":,"pid":,"hostname":"pepr--watcher-","metadata":,"msg":"external api call (create): reconcile/finalize"} ``` **Note:** The log includes the full ConfigMap object that triggered the action, making it useful for debugging which specific resource was reconciled or finalized. ### Create or Update Event [Section titled “Create or Update Event”](#create-or-update-event) ```typescript When(a.ConfigMap) .IsCreatedOrUpdated() .InNamespace("hello-pepr-finalize-createorupdate") .WithName("cm-watch-createorupdate") .Watch(function watchCreateOrUpdate(cm) { if (cm.metadata?.deletionTimestamp) { return; } Log.info(cm, "external api call (createorupdate): watch/callback"); }) .Finalize(function finalizeCreateOrUpdate(cm) { Log.info(cm, "external api call (createorupdate): watch/finalize"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-1) ```json {"level":30,"time":,"pid":,"hostname":"pepr--watcher-","kind":"ConfigMap","apiVersion":"v1","metadata":,"msg":"external api call (createorupdate): watch/callback"} {"level":30,"time":,"pid":,"hostname":"pepr--watcher-","kind":"ConfigMap","apiVersion":"v1","metadata":,"msg":"external api call (createorupdate): watch/finalize"} ``` **Note:** The log includes the full ConfigMap object that triggered the action, making it useful for debugging which specific resource was reconciled or finalized. ### Update Event [Section titled “Update Event”](#update-event) ```typescript When(a.ConfigMap) .IsUpdated() .InNamespace("hello-pepr-finalize-update") .WithName("cm-watch-update") .Watch(function watchUpdate(cm) { // delete with finalizer triggers an UPDATE to add deletionTimestamp; ignore it if (cm.metadata?.deletionTimestamp) { return; } Log.info(cm, "external api call (update): watch/callback"); }) .Finalize(function finalizeUpdate(cm) { Log.info(cm, "external api call (update): watch/finalize"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-2) ```json {"level":30,"time":,"pid":,"hostname":"pepr--watcher-","kind":"ConfigMap","apiVersion":"v1","metadata":,"msg":"external api call (update): watch/callback"} {"level":30,"time":,"pid":,"hostname":"pepr--watcher-","kind":"ConfigMap","apiVersion":"v1","metadata":,"msg":"external api call (update): watch/finalize"} ``` ### Update Opt Out Event [Section titled “Update Opt Out Event”](#update-opt-out-event) ```typescript When(a.ConfigMap) .IsUpdated() .InNamespace("hello-pepr-finalize-update-opt-out") .WithName("cm-watch-update-opt-out") .Watch(function watchUpdateOptOut(cm) { // delete with finalizer triggers an UPDATE to add deletionTimestamp; ignore it if (cm.metadata?.deletionTimestamp) { return; } Log.info(cm, "external api call (update-opt-out): watch/callback"); }) .Finalize(function finalizeUpdateOptOut(cm) { Log.info(cm, "external api call (update-opt-out): watch/pre-finalize"); return false; }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-3) ```json {"level":30,"time":,"pid":,"hostname":"pepr--watcher-","kind":"ConfigMap","apiVersion":"v1","metadata":,"msg":"external api call (update-opt-out): watch/callback"} {"level":30,"time":,"pid":,"hostname":"pepr--watcher-","kind":"ConfigMap","apiVersion":"v1","metadata":,"msg":"external api call (update-opt-out): watch/pre-finalize"} ``` ### Delete Event [Section titled “Delete Event”](#delete-event) ```typescript When(a.ConfigMap) .IsDeleted() .InNamespace("hello-pepr-finalize-delete") .WithName("cm-watch-delete") .Watch(function watchDelete(cm) { // Here be dragons! // // Due to the way kubernetes finalizers work, this deletion callback will fire // AFTER the finalizer callback fires -- be sure to expect & code for this! // (see: https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/#how-finalizers-work ) Log.info(cm, "external api call (delete): watch/callback"); }) .Finalize(function finalizeDelete(cm) { Log.info(cm, "external api call (delete): watch/finalize"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-4) ```json {"level":30,"time":,"pid":,"hostname":"pepr--watcher-","kind":"ConfigMap","apiVersion":"v1","metadata":,"msg":"external api call (delete): watch/finalize"} {"level":30,"time"::,"pid":,"hostname":"pepr--watcher-","kind":"ConfigMap","apiVersion":"v1","metadata":,"msg":"external api call (delete): watch/callback"} ``` > \[!TIP] Learn more about the [Finalize action](https://docs.pepr.dev/actions/finalize/) > **Source:** [hello-pepr-finalize](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-finalize) # Mutate > Mutate # Action: Mutate [Section titled “Action: Mutate”](#action-mutate) Pepr `Mutate()` action is used to modify Kubernetes objects during admission. Mutations allow you to enforce defaults, apply standard labels/annotations, or clean up unwanted metadata before the resource is persisted. ## When should you use Mutate? [Section titled “When should you use Mutate?”](#when-should-you-use-mutate) Use `Mutate()` when you need to: * Apply default labels or annotations to resources as they are created. * Remove unwanted metadata automatically (labels, annotations, fields). * Normalize user input so objects follow your standards. * Enforce consistent cluster hygiene by modifying objects before they are stored. * Transform objects while still allowing them through. > \[!TIP] Consider adding a matching `Validate()` to detect invalid >post-mutation states. ## Code Snippet Examples [Section titled “Code Snippet Examples”](#code-snippet-examples) This example mutates Secret resources and logs different lifecycle events: create, update, delete, and create-or-update. > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-mutate/capabilities/mutate.ts)** ### Create Event [Section titled “Create Event”](#create-event) ```typescript When(a.Secret) .IsCreated() .InNamespace(name) .WithName("create-yay") .Mutate(function createYay() { Log.info("Mutate: create-yay"); }); ``` ##### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"Mutate: create-yay"} ``` ### Create or Update Event [Section titled “Create or Update Event”](#create-or-update-event) ```typescript When(a.Secret) .IsCreatedOrUpdated() .InNamespace(name) .WithName("cou-create-yay") .Mutate(function couCreateYay() { Log.info("Mutate: cou-create-yay"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-1) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"Mutate: cou-create-yay"} ``` ### Update Event [Section titled “Update Event”](#update-event) ```typescript When(a.Secret) .IsUpdated() .InNamespace(name) .WithName("update-yay") .Mutate(function updateYay() { Log.info("Mutate: update-yay"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-2) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"Mutate: update-yay"} ``` ### Delete Event [Section titled “Delete Event”](#delete-event) ```typescript When(a.Secret) .IsDeleted() .InNamespace(name) .WithName("delete-yay") .Mutate(function deleteYay() { Log.info("Mutate: delete-yay"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-3) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"Mutate: delete-yay"} ``` > \[!TIP] Learn more about the [Mutate action](http://localhost:4321/actions/mutate/) > **Source:** [hello-pepr-mutate](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-mutate) # Reconcile - Global Strategy > Reconcile - Global Strategy # Action: Reconcile - Global Strategy [Section titled “Action: Reconcile - Global Strategy”](#action-reconcile---global-strategy) The Reconcile action provides ordered, asynchronous processing of Kubernetes resource changes. The `global` strategy consolidates all events into one queue, offering maximum ordering guarantees at the cost of throughput. ## When to Use [Section titled “When to Use”](#when-to-use) Use the `global` strategy when you need: * Strict ordering guarantees across all resources and kinds * Sequential processing to prevent any race conditions * Simple coordination across different resource types * Strong consistency over performance **Trade-off**: Lower throughput since all events are processed sequentially in a single queue, regardless of resource kind or namespace. ## Code Examples [Section titled “Code Examples”](#code-examples) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-reconcile-global/capabilities/reconcile.ts)** **Module Configuration in package.json:** ```json { "pepr": { "env": { "PEPR_RECONCILE_STRATEGY": "global" } } } ``` **Reconcile Actions:** ```typescript When(a.ConfigMap) .IsCreatedOrUpdated() .InNamespace("hello-pepr-reconcile-one") .Reconcile(function keepsOrder(cm) { const { name } = cm.metadata; if (name === "kube-root-ca.crt") { return; } return name === "cm-slow" ? slow(cm) : name === "cm-fast" ? fast(cm) : oops(cm); }); When(a.Secret) .IsCreatedOrUpdated() .InNamespace("hello-pepr-reconcile-one", "hello-pepr-reconcile-two") .Reconcile(function keepsOrder(se) { const { name } = se.metadata; return name === "se-slow" ? slow(se) : name === "se-fast" ? fast(se) : oops(se); }); ``` ## Behavior [Section titled “Behavior”](#behavior) **With `global` strategy:** * All ConfigMaps and Secrets share one global queue * Everything is processed strictly in order * No parallel processing across any resource types **Example scenario:** 1. `cm-slow` (ConfigMap) is created → starts processing (500ms) 2. `se-fast` (Secret) is created → waits in global queue 3. `cm-fast` (ConfigMap) is created → waits in global queue 4. `cm-slow` completes → `se-fast` starts processing (300ms) 5. `se-fast` completes → `cm-fast` starts processing (300ms) **Processing order guarantee:** * All resources (ConfigMaps, Secrets, everything) are processed in strict FIFO order * No parallel processing - only one resource is reconciled at a time * Complete ordering across all resource kinds and namespaces > \[!TIP] Learn more about [Reconcile Queue Configurations](https://docs.pepr.dev/reference/faq/#reconcile-queue-configurations) > **Source:** [hello-pepr-reconcile-global](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-reconcile-global) # Reconcile - Kind Strategy > Reconcile - Kind Strategy # Action: Reconcile - Kind Strategy [Section titled “Action: Reconcile - Kind Strategy”](#action-reconcile---kind-strategy) The Reconcile action provides ordered, asynchronous processing of Kubernetes resource changes. The `kind` strategy creates one queue per resource kind (e.g., all ConfigMaps share one queue, all Secrets share another), allowing different kinds to be processed in parallel while maintaining order within each kind. ## When to Use [Section titled “When to Use”](#when-to-use) Use the `kind` strategy when you need: * Ordering guarantees within each resource type * Parallel processing across different resource kinds * Balance between throughput and ordering * Simpler queue management (default behavior) **Trade-off**: All instances of the same kind share a queue, so high activity on one resource can delay processing of other resources of the same kind. ## Code Examples [Section titled “Code Examples”](#code-examples) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-reconcile-kind/capabilities/reconcile.ts)** **Module Configuration in package.json:** ```json { "pepr": { "env": { "PEPR_RECONCILE_STRATEGY": "kind" } } } ``` **Reconcile Actions:** ```typescript When(a.ConfigMap) .IsCreatedOrUpdated() .InNamespace("hello-pepr-reconcile") .Reconcile(function keepsOrder(cm) { const { name } = cm.metadata; if (name === "kube-root-ca.crt") { return; } return name === "cm-slow" ? slow(cm) : name === "cm-fast" ? fast(cm) : oops(cm); }); When(a.Secret) .IsCreatedOrUpdated() .InNamespace("hello-pepr-reconcile") .Reconcile(function keepsOrder(se) { const { name } = se.metadata; return name === "se-slow" ? slow(se) : name === "se-fast" ? fast(se) : oops(se); }); ``` ## Behavior [Section titled “Behavior”](#behavior) **With `kind` strategy:** * All ConfigMaps are processed in one queue (in order) * All Secrets are processed in a separate queue (in order) * ConfigMaps and Secrets can be processed in parallel **Example scenario:** 1. `cm-slow` is created → starts processing (500ms) 2. `cm-fast` is created → waits in ConfigMap queue 3. `se-slow` is created → starts processing in parallel (500ms, separate Secret queue) 4. `cm-slow` completes → `cm-fast` starts processing **Processing order guarantee:** * ConfigMaps are always processed in the order they were received * Secrets are always processed in the order they were received * But ConfigMaps and Secrets process independently > \[!TIP] Learn more about [Reconcile Queue Configurations](https://docs.pepr.dev/reference/faq/#reconcile-queue-configurations) > **Source:** [hello-pepr-reconcile-kind](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-reconcile-kind) # Reconcile- kindNs Strategy > Reconcile- kindNs Strategy # Action: Reconcile- kindNs Strategy [Section titled “Action: Reconcile- kindNs Strategy”](#action-reconcile--kindns-strategy) The Reconcile action provides ordered, asynchronous processing of Kubernetes resource changes. The `kindNs` strategy creates one queue per kind within each namespace, allowing resources in different namespaces to be processed in parallel while maintaining order within each kind-namespace combination. ## When to Use [Section titled “When to Use”](#when-to-use) Use the `kindNs` strategy when you need: * Ordering guarantees within each kind per namespace * Parallel processing across namespaces * Isolation between namespace workloads * Multi-tenant environments where namespace isolation is important **Trade-off**: Resources of the same kind in the same namespace share a queue, but different namespaces can process independently in parallel. ## Code Examples [Section titled “Code Examples”](#code-examples) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-reconcile-kindns/capabilities/reconcile.ts)** **Module Configuration in package.json:** ```json { "pepr": { "env": { "PEPR_RECONCILE_STRATEGY": "kindNs" } } } ``` **Reconcile Actions:** ```typescript When(a.ConfigMap) .IsCreatedOrUpdated() .InNamespace("hello-pepr-reconcile-one") .Reconcile(function keepsOrder(cm) { const { name } = cm.metadata; if (name === "kube-root-ca.crt") { return; } return name === "cm-slow" ? slow(cm) : name === "cm-fast" ? fast(cm) : oops(cm); }); When(a.Secret) .IsCreatedOrUpdated() .InNamespace("hello-pepr-reconcile-one", "hello-pepr-reconcile-two") .Reconcile(function keepsOrder(se) { const { name } = se.metadata; return name === "se-slow" ? slow(se) : name === "se-fast" ? fast(se) : oops(se); }); ``` ## Behavior [Section titled “Behavior”](#behavior) **With `kindNs` strategy:** * ConfigMaps in `hello-pepr-reconcile-one` have their own queue * Secrets in `hello-pepr-reconcile-one` have their own queue * Secrets in `hello-pepr-reconcile-two` have their own queue * All three queues can process in parallel **Example scenario:** 1. `cm-slow` (ConfigMap in ns-one) is created → starts processing (500ms) 2. `se-slow` (Secret in ns-one) is created → starts processing in parallel (500ms, different queue) 3. `se-fast` (Secret in ns-two) is created → starts processing in parallel (300ms, different namespace queue) 4. All three process simultaneously **Processing order guarantee:** * ConfigMaps in `hello-pepr-reconcile-one` are processed in order * Secrets in `hello-pepr-reconcile-one` are processed in order (independently from ConfigMaps) * Secrets in `hello-pepr-reconcile-two` are processed in order (independently from ns-one) * Namespace boundaries provide natural parallelization > \[!TIP] Learn more about [Reconcile Queue Configurations](https://docs.pepr.dev/reference/faq/#reconcile-queue-configurations) > **Source:** [hello-pepr-reconcile-kindns](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-reconcile-kindns) # Reconcile- kindNsName Strategy > Reconcile- kindNsName Strategy # Action: Reconcile- kindNsName Strategy [Section titled “Action: Reconcile- kindNsName Strategy”](#action-reconcile--kindnsname-strategy) The Reconcile action provides ordered, asynchronous processing of Kubernetes resource changes. The `kindNsName` strategy provides the finest granularity—each individual resource gets its own dedicated processing queue, keyed by kind + namespace + name. This is the **recommended strategy** for most real-world scenarios. ## When to Use [Section titled “When to Use”](#when-to-use) Use the `kindNsName` strategy when you need: * Maximum throughput with parallel processing * Ordering guarantees for each individual resource * Independent processing of different resources * Optimal performance for production workloads * Prevention of one resource blocking another **Trade-off**: Multiple events for the same resource are still processed in order, but different resources can process completely independently for maximum parallelism. ## Code Examples [Section titled “Code Examples”](#code-examples) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-reconcile-kindnsname/capabilities/reconcile.ts)** **Module Configuration in package.json:** ```json { "pepr": { "env": { "PEPR_RECONCILE_STRATEGY": "kindNsName" } } } ``` **Reconcile Actions:** ```typescript When(a.ConfigMap) .IsCreatedOrUpdated() .InNamespace("hello-pepr-reconcile-one") .Reconcile(function keepsOrder(cm) { const { name } = cm.metadata; if (name === "kube-root-ca.crt") { return; } return name === "cm-slow" ? slow(cm) : name === "cm-fast" ? fast(cm) : oops(cm); }); When(a.Secret) .IsCreatedOrUpdated() .InNamespace("hello-pepr-reconcile-one", "hello-pepr-reconcile-two") .Reconcile(function keepsOrder(se) { const { name } = se.metadata; return name === "se-slow" ? slow(se) : name === "se-fast" ? fast(se) : oops(se); }); ``` ## Behavior [Section titled “Behavior”](#behavior) **With `kindNsName` strategy (recommended):** * Each unique resource has its own queue * `cm-slow` (ConfigMap/ns-one/cm-slow) has its own queue * `cm-fast` (ConfigMap/ns-one/cm-fast) has its own queue * `se-slow` (Secret/ns-one/se-slow) has its own queue * `se-fast` (Secret/ns-two/se-fast) has its own queue * All four can process in parallel **Example scenario:** 1. `cm-slow` is created → starts processing immediately (500ms) 2. `cm-fast` is created → starts processing immediately in parallel (300ms) 3. `se-slow` is created → starts processing immediately in parallel (500ms) 4. `se-fast` is created → starts processing immediately in parallel (300ms) 5. All four resources reconcile simultaneously **Processing order guarantee:** * Multiple updates to `cm-slow` are processed in order * Multiple updates to `cm-fast` are processed in order (but independently from `cm-slow`) * Each resource’s events are ordered, but different resources never block each other * Maximum parallelism with per-resource consistency > \[!TIP] Learn more about [Reconcile Queue Configurations](https://docs.pepr.dev/reference/faq/#reconcile-queue-configurations) > **Source:** [hello-pepr-reconcile-kindnsname](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-reconcile-kindnsname) # Validate > Validate # Action: Validate [Section titled “Action: Validate”](#action-validate) Pepr `Validate()` action runs **during admission phase** to allow or deny Kubernetes objects before they are persisted. Use it to enforce rules, block invalid configurations, and ensure resources meet required standards. ## When should you use Validate? [Section titled “When should you use Validate?”](#when-should-you-use-validate) * Enforce strict correctness rules on Kubernetes resources. * Block invalid or dangerous configurations. * Ensure required labels, annotations, or field conventions are followed. * Reject requests that do not meet security, compliance, or operational standards. * Verify that mutated objects remain valid after transformations. > \[!TIP] Use `Validate()` after `Mutate()` if you want to ensure your mutations produce a valid final object. ## Code Snippet Examples [Section titled “Code Snippet Examples”](#code-snippet-examples) This example validates that Pods include required labels and rejects ones that do not comply. > **View additional examples on [GitHub](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-validate/capabilities/validate.ts)** ### Create Event [Section titled “Create Event”](#create-event) ```typescript When(a.Secret) .IsCreated() .InNamespace(name) .WithName("create-yay") .Validate(function createYay(request) { Log.info("Valid: create-yay"); return request.Approve(); }); ``` ##### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"Valid: create-yay"} ``` ### Create or Update Event [Section titled “Create or Update Event”](#create-or-update-event) ```typescript When(a.Secret) .IsCreatedOrUpdated() .InNamespace(name) .WithName("cou-create-oof") .Validate(function couCreateOof(request) { const failWhen = request.Raw.data.failWhen; const operation = request.Request.operation; return failWhen === operation ? request.Deny("cou-create-oof") : request.Approve(); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-1) ```json Error from server: error when creating "capabilities/scenario.create-or-update.yaml": admission webhook "pepr-.pepr.dev" denied the request: cou-create-oof ``` ### Update Event [Section titled “Update Event”](#update-event) ```typescript When(a.Secret) .IsUpdated() .InNamespace(name) .WithName("update-yay") .Validate(function updateYay(request) { Log.info("Valid: update-yay"); return request.Approve(); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-2) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"Valid: update-yay"} ``` ### Delete Event [Section titled “Delete Event”](#delete-event) ```typescript When(a.Secret) .IsDeleted() .InNamespace(name) .WithName("delete-yay") .Validate(function deleteYay(request) { Log.info("Valid: delete-yay"); return request.Approve(); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-3) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"Valid: delete-yay"} ``` > \[!TIP] Learn more about the [Validate action](https://docs.pepr.dev/actions/validate/) > **Source:** [hello-pepr-validate](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-validate) # Validation Warnings > Validation Warnings # Action: Validation Warnings [Section titled “Action: Validation Warnings”](#action-validation-warnings) This module demonstrates how to use the warnings feature in Pepr validation responses. ## Overview [Section titled “Overview”](#overview) The warnings feature allows Pepr modules to include warning messages in both approval and denial responses during validation. This provides a way to communicate important information to users without necessarily blocking their requests. ## Scenarios [Section titled “Scenarios”](#scenarios) This module includes three key scenarios that demonstrate different ways to use warnings: 1. **Approval with Warnings** (`warnings-approve`) * Approves a ConfigMap while including warnings about: * Use of deprecated fields * Missing recommended labels * Too many configuration items in a single ConfigMap 2. **Denial with Warnings** (`warnings-deny`) * Denies a ConfigMap that contains dangerous settings * Includes warnings that explain why the settings are dangerous * Uses a custom status code (422) for the denial 3. **Multiple Warnings** (`warnings-multiple`) * Demonstrates how to aggregate multiple warnings in a single approval * Shows warnings for deprecated and insecure settings * Shows warnings for missing recommended labels ## Implementation Details [Section titled “Implementation Details”](#implementation-details) The module demonstrates three key validation patterns: 1. Using `request.Approve(warnings)` to approve a request with warnings 2. Using `request.Deny(message, statusCode, warnings)` to deny a request with warnings 3. Collecting and aggregating multiple warnings in a single response These patterns can be used to provide helpful feedback to users while still enforcing policies. The warnings appear in both the API response and in the Pepr logs. ## Testing [Section titled “Testing”](#testing) The module includes end-to-end tests that verify: * Resources with warnings are properly approved when appropriate * Resources with dangerous settings are denied with appropriate warnings * Multiple warnings are correctly aggregated and returned in responses > \[!TIP] Learn more about [Validation Warnings](https://docs.pepr.dev/actions/validate/#validation-with-warnings) > **Source:** [hello-pepr-warnings](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-warnings) # Watch > Watch # Action: Watch [Section titled “Action: Watch”](#action-watch) Pepr `Watch()` actions are ideal for asynchronous operations that need to respond to resource changes without timeout constraints. ## Use Watch when you need to: [Section titled “Use Watch when you need to:”](#use-watch-when-you-need-to) * Monitor existing resources in a cluster * Perform long-running operations without timeout constraints * React to resource lifecycle events (creation, updates, deletion) ## Code Snippet Examples [Section titled “Code Snippet Examples”](#code-snippet-examples) This example watches Secret resources and logs different lifecycle events: create, update, delete, and create-or-update. > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-watch/capabilities/watch.ts)** ### Watch Create Events [Section titled “Watch Create Events”](#watch-create-events) ```typescript When(a.Secret) .IsCreated() .InNamespace(name) .WithName("create-me") .Watch((instance) => { Log.info(`Watched ${instance.metadata?.name}: create`); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"Watched create-me: create"} ``` ### Watch Create or Update Events [Section titled “Watch Create or Update Events”](#watch-create-or-update-events) ```typescript When(a.Secret) .IsCreatedOrUpdated() .InNamespace(name) .WithName("create-or-update-me") .Watch((instance, phase) => { Log.info(`Watched ${instance.metadata?.name}: ${phase}`); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-1) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"Watched create-or-update-me: ADDED"} ``` ### Watch Update Events [Section titled “Watch Update Events”](#watch-update-events) ```typescript When(a.Secret) .IsUpdated() .InNamespace(name) .WithName("update-me") .Watch((instance) => { Log.info(`Watched ${instance.metadata?.name}: update`); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-2) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"Watched create-or-update-me: update"} ``` ### Watch Delete Events [Section titled “Watch Delete Events”](#watch-delete-events) ```typescript When(a.Secret) .IsDeleted() .InNamespace(name) .WithName("delete-me") .Watch((instance) => { Log.info(`Watched ${instance.metadata?.name}: delete`); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-3) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"Watched create-or-update-me: delete"} ``` > \[!TIP] Learn more about the [Watch Action](https://docs.pepr.dev/actions/watch/) > **Source:** [hello-pepr-watch](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-watch) # Alias > Alias # Alias [Section titled “Alias”](#alias) The `Alias()` function allows you to label actions with meaningful identifiers that appear in logs. This is particularly useful in modules with multiple actions of the same type, making it easier to trace which specific action executed. ## When to Use [Section titled “When to Use”](#when-to-use) Use `Alias()` when you need to: * Distinguish between multiple similar actions * Improve log filtering and searchability * Debug complex modules with many actions * Track action execution in production ## Code Examples [Section titled “Code Examples”](#code-examples) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-alias/capabilities/alias.ts)** **Reconcile with Alias:** ```typescript When(a.ConfigMap) .IsCreated() .InNamespace("hello-pepr-alias-create") .WithName("cm-reconcile-create") .Alias("alias:create:reconcile") .Reconcile(function reconcileCreate(cm, phase, logger) { logger.info( cm, "external api call (reconcile-create-alias): reconcile/callback", ); }) .Finalize(function finalizeCreate(cm, logger) { logger.info( cm, "external api call (reconcile-create-alias): reconcile/finalize", ); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","alias":"alias:create:reconcile","msg":"external api call (reconcile-create-alias): reconcile/callback"} {"level":30,"time":,"pid":,"hostname":"pepr-","alias":"alias:create:reconcile","msg":"external api call (reconcile-create-alias): reconcile/finalize"} ``` **Watch with Alias:** ```typescript When(a.ConfigMap) .IsCreated() .InNamespace("hello-pepr-alias-create") .WithName("cm-watch-create") .Alias("alias:create:watch") .Watch(function watchCreate(cm, phase, logger) { logger.info(cm, "external api call (watch-create-alias): watch/callback"); }) .Finalize(function finalizeCreate(cm, logger) { logger.info(cm, "external api call (watch-create-alias): watch/finalize"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-1) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","alias":"alias:create:watch","msg":"external api call (watch-create-alias): watch/callback"} {"level":30,"time":,"pid":,"hostname":"pepr-","alias":"alias:create:watch","msg":"external api call (watch-create-alias): watch/finalize"} ``` **Validate with Alias:** ```typescript When(a.ConfigMap) .IsCreated() .InNamespace("hello-pepr-alias-create") .WithName("cm-validate-create") .Alias("alias:create:validate") .Validate(function validateCreate(cm, logger) { logger.info( cm, "external api call (validate-create-alias): validate/callback", ); return cm.Approve(); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-2) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","alias":"alias:create:validate","msg":"external api call (validate-create-alias): validate/callback"} ``` > \[!TIP] Learn more about the [Alias function](https://docs.pepr.dev/actions/using-alias-child-logger) > **Source:** [hello-pepr-alias](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-alias) # Module Settings > Module Settings # Configuration: Module Settings [Section titled “Configuration: Module Settings”](#configuration-module-settings) Pepr modules can be customized through configuration options in `package.json` under the `pepr` section. These module-level settings apply globally across all capabilities, allowing you to customize labels, environment variables, namespace exclusions, and webhook behavior. ## Use module configuration for: [Section titled “Use module configuration for:”](#use-module-configuration-for) * Setting global behavior that applies to all capabilities in your module * Adding custom labels for organizational or monitoring purposes * Injecting environment variables needed across multiple capabilities * Excluding system namespaces from processing * Adjusting webhook timeouts for cluster-specific performance needs ## Code Examples [Section titled “Code Examples”](#code-examples) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-config)** **Module Configuration in package.json:** ```json { "pepr": { "uuid": "aac63ece-b202-5b18-b3c8-fff5b631a4f1", "onError": "reject", "webhookTimeout": 15, "customLabels": { "namespace": { "super": "secret", "special": "sauce" } }, "alwaysIgnore": { "namespaces": [ "hello-pepr-config-ignore" ] }, "env": { "ITS": "a bird, a plane, superman" } } } ``` **Accessing Environment Variables in Code:** ```typescript When(a.ConfigMap) .IsCreated() .InNamespace(name) .WithName("noop") .Mutate(async function mutateEnv() { Log.info({ ITS: process.env.ITS }, "env"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","ITS":"a bird, a plane, superman","msg":"env"} ``` > \[!TIP] Learn more about [Configuration](https://docs.pepr.dev/user-guide/customization/#packagejson-configurations-table) > **Source:** [hello-pepr-config](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-config) # Controller-Specific Namespace Exclusions > Controller-Specific Namespace Exclusions # Configuration: Controller-Specific Namespace Exclusions [Section titled “Configuration: Controller-Specific Namespace Exclusions”](#configuration-controller-specific-namespace-exclusions) Controller-specific namespace ignoring provides fine-grained control over which controllers process resources in specific namespaces. While `alwaysIgnore.namespaces` excludes namespaces from ALL Pepr processing, you can configure controller-specific ignores to exclude namespaces from only admission webhooks or only watch controllers. ## When to Use [Section titled “When to Use”](#when-to-use) Use controller-specific namespace ignoring for: * Allowing admission control (Mutate/Validate) without watch processing * Allowing watch processing without admission control * Fine-grained control over namespace processing per controller type * Separating concerns between admission and watch behaviors ## Configuration Options [Section titled “Configuration Options”](#configuration-options) * **`admission.alwaysIgnore.namespaces`** - Excludes namespaces from admission webhooks (Mutate/Validate) only * **`watch.alwaysIgnore.namespaces`** - Excludes namespaces from watch controllers only ## Code Examples [Section titled “Code Examples”](#code-examples) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-config-ignored-controller-ns)** **Module Configuration in package.json:** ```json { "pepr": { "admission": { "alwaysIgnore": { "namespaces": ["mutate-ignored"] } }, "watch": { "alwaysIgnore": { "namespaces": ["watch-ignored"] } } } } ``` **Admission and Watch Actions:** ```typescript When(a.ConfigMap) .IsCreated() .WithNameRegex(/^mutate-ignored/) .Mutate(po => po.SetLabel("not", "seen")); When(a.ConfigMap) .IsCreated() .WithNameRegex(/^mutate-ignored/) .Watch(async cm => { await K8s(kind.ConfigMap).Apply({ metadata: { name: cm.metadata.name, namespace: cm.metadata.namespace, annotations: { been: "seen", }, }, data: { "cm-uid": cm.metadata.uid, }, }); }); ``` ## Behavior [Section titled “Behavior”](#behavior) **Admission Ignore (`mutate-ignored` namespace):** * Mutate actions: NOT called * Watch actions: CALLED **Watch Ignore (`watch-ignored` namespace):** * Mutate actions: CALLED * Watch actions: NOT called > \[!TIP] Learn more about [Configuration](https://docs.pepr.dev/user-guide/customization/#packagejson-configurations-table) > **Source:** [hello-pepr-config-ignored-controller-ns](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-config-ignored-controller-ns) # Namespace Exclusions > Namespace Exclusions # Configuration: Namespace Exclusions [Section titled “Configuration: Namespace Exclusions”](#configuration-namespace-exclusions) When you configure namespaces in `alwaysIgnore.namespaces`, Pepr will not process any resources in those namespaces for both admission webhooks (mutate/validate) and watch controllers. This completely excludes specific namespaces from all Pepr processing. ## When to Use [Section titled “When to Use”](#when-to-use) Use `alwaysIgnore.namespaces` for: * Completely excluding certain namespaces from all Pepr processing * System namespaces where you don’t want any Pepr intervention * Namespaces that should bypass all admission control and watch operations * Reducing processing overhead by ignoring irrelevant namespaces > \[!IMPORTANT] This applies to ALL actions in the module, regardless of capability or filters. Both admission webhooks and watch controllers will ignore these namespaces. ## Code Examples [Section titled “Code Examples”](#code-examples) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-config-ignored-ns)** **Module Configuration in package.json:** ```json { "pepr": { "alwaysIgnore": { "namespaces": [ "ignored" ] } } } ``` **Actions That Will Be Ignored:** ```typescript When(a.ConfigMap) .IsCreated() .WithNameRegex(/^invisible/) .Watch(async cm => { await K8s(kind.ConfigMap).Apply({ metadata: { name: cm.metadata.name, namespace: cm.metadata.namespace, annotations: { not: "seen", }, }, data: { "cm-uid": cm.metadata.uid, }, }); }); When(a.ConfigMap) .IsCreated() .WithNameRegex(/^invisible/) .Mutate(cm => cm.SetAnnotation("not", "seen")); ``` ## Behavior [Section titled “Behavior”](#behavior) **ConfigMap named “invisible-\*” created in “ignored” namespace:** * Mutate action: NOT called (namespace is in `alwaysIgnore`) * Watch action: NOT called (namespace is in `alwaysIgnore`) * Result: ConfigMap is created without any Pepr modifications or annotations **ConfigMap named “invisible-\*” created in any other namespace:** * Mutate action: CALLED (adds annotation `not: "seen"`) * Watch action: CALLED (applies annotation `not: "seen"` and data field `cm-uid`) * Result: ConfigMap gets Pepr modifications > \[!TIP] Learn more about [Configuration](https://docs.pepr.dev/user-guide/customization/#packagejson-configurations-table) > **Source:** [hello-pepr-config-ignored-ns](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-config-ignored-ns) # RBAC Permissions > RBAC Permissions # Configuration: RBAC Permissions [Section titled “Configuration: RBAC Permissions”](#configuration-rbac-permissions) Pepr modules can be configured with scoped RBAC permissions that follow the principle of least privilege instead of using cluster-admin access. By default, Pepr modules use admin RBAC mode which grants full cluster access. This example shows how to configure custom, scoped RBAC permissions that follow the principle of least privilege. ## RBAC Modes [Section titled “RBAC Modes”](#rbac-modes) Pepr supports two RBAC modes: **Admin Mode (default):** ```bash npx pepr build --rbac-mode admin ``` Grants full cluster-admin permissions. Use only for development or when necessary. **Scoped Mode (recommended):** ```bash npx pepr build --rbac-mode scoped ``` Grants only the permissions needed for specific actions. This example uses scoped mode with custom ClusterRole definitions. ## When to Use [Section titled “When to Use”](#when-to-use) Use scoped RBAC mode when you need to: * Follow the principle of least privilege in production environments * Restrict Pepr module permissions to specific resources and operations * Comply with security policies that prohibit cluster-admin access * Audit and control exactly which Kubernetes resources a module can access * Deploy multiple Pepr modules with different permission requirements ## Code Examples [Section titled “Code Examples”](#code-examples) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-custom-rbac/capabilities/rbac.ts)** This module demonstrates various Pepr features requiring different RBAC permissions: ### Watch Action with K8s SSA [Section titled “Watch Action with K8s SSA”](#watch-action-with-k8s-ssa) ```typescript When(a.Namespace) .IsCreated() .WithName("pepr-demo-2") .Watch(async ns => { Log.info("Namespace pepr-demo-2 was created."); try { // Apply the ConfigMap using K8s server-side apply await K8s(kind.ConfigMap).Apply({ metadata: { name: "pepr-ssa-demo", namespace: "pepr-demo-2", }, data: { "ns-uid": ns.metadata.uid, }, }); } catch (error) { // You can use the Log object to log messages to the Pepr controller pod Log.error(error, "Failed to apply ConfigMap using server-side apply."); } // You can share data between actions using the Store, including between different types of actions Store.setItem("watch-data", "This data was stored by a Watch Action."); }); ``` #### Example pod log output (success): [Section titled “Example pod log output (success):”](#example-pod-log-output-success) ```json {"level":30,"time":,"pid":,"hostname":"pepr-watcher-","msg":"Namespace pepr-demo-2 was created."} ``` ### Simple Mutate Action [Section titled “Simple Mutate Action”](#simple-mutate-action) ```typescript When(a.ConfigMap) .IsCreated() .Validate(request => { if (request.HasAnnotation("evil")) { return request.Deny("No evil CM annotations allowed.", 400); } return request.Approve(); }); ``` #### Example pod log output (approved): [Section titled “Example pod log output (approved):”](#example-pod-log-output-approved) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"ConfigMap approved"} ``` ### Mutate & Validate Actions [Section titled “Mutate & Validate Actions”](#mutate--validate-actions) ```typescript When(a.ConfigMap) .IsCreated() .WithName("example-2") .Mutate(request => { // This Mutate Action will mutate the request before it is persisted to the cluster // Use `request.Merge()` to merge the new data with the existing data request.Merge({ metadata: { labels: { pepr: "was-here", }, annotations: { "pepr.dev": "annotations-work-too", }, }, }); }) .Validate(request => { // This Validate Action will validate the request before it is persisted to the cluster // Approve the request if the ConfigMap has the label 'pepr' if (request.HasLabel("pepr")) { return request.Approve(); } // Otherwise, deny the request with an error message (optional) return request.Deny("ConfigMap must have label 'pepr'"); }) .Watch((cm, phase) => { // This Watch Action will watch the ConfigMap after it has been persisted to the cluster Log.info(cm, `ConfigMap was ${phase} with the name example-2`); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output) ```json {"level":30,"time":,"pid":,"hostname":"pepr-watcher-","msg":"ConfigMap was ADDED with the name example-2"} ``` > \[!TIP] Learn more about [RBAC Configuration](https://docs.pepr.dev/user-guide/rbac/) > **Source:** [hello-pepr-custom-rbac](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-custom-rbac) # Deletion Timestamp > Deletion Timestamp # Filter: Deletion Timestamp [Section titled “Filter: Deletion Timestamp”](#filter-deletion-timestamp) The `WithDeletionTimestamp()` filter targets Kubernetes resources that are in the process of being deleted. When a Kubernetes resource is deleted, it may not be removed immediately if it has [finalizers](https://docs.pepr.dev/actions/finalize). During this time, the resource has a `deletionTimestamp` set in its metadata. The `WithDeletionTimestamp()` filter allows you to target resources in this state. ## When to Use `WithDeletionTimestamp()` [Section titled “When to Use WithDeletionTimestamp()”](#when-to-use-withdeletiontimestamp) Use `WithDeletionTimestamp()` when you need to: * Implement custom cleanup logic before resource deletion * Work with finalizers that require additional processing * Monitor or log resources entering deletion state * Perform validation or mutation on resources being deleted (during UPDATE events) ## Important Notes [Section titled “Important Notes”](#important-notes) **Admission Controllers (Mutate/Validate):** * `WithDeletionTimestamp()` does **not** work with with `.IsDeleted()` through `Mutate` or `Validate` because the Kubernetes Admission Process does not fire the DELETE event with a deletion timestamp. * `WithDeletionTimestamp()` **will** match on `.IsUpdated()` events during Admission when pending-deletion changes occur (like removing a finalizer). **Watch Controllers:** * `WithDeletionTimestamp()` works with both `.IsUpdated()` and `.IsDeleted()` events * Watch actions can observe the full deletion lifecycle including the final DELETE event ## Code Examples [Section titled “Code Examples”](#code-examples) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-deletion-timestamp/capabilities/deletion.ts)** ### Admission Controller (Mutate) [Section titled “Admission Controller (Mutate)”](#admission-controller-mutate) Triggers when a Pod is updated and has a deletion timestamp (e.g., when finalizers are being removed): ```typescript When(a.Pod) .IsUpdated() .InNamespace(namespace1) .WithName("ns1-admission") .WithDeletionTimestamp() .Mutate(po => { logSeen(po.Raw.metadata.name); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"DTS: Saw a pod ns1-admission."} ``` ### Watch Controller [Section titled “Watch Controller”](#watch-controller) Monitors Pods with deletion timestamps in the Watch controller: ```typescript When(a.Pod) .IsUpdated() .InNamespace(namespace1) .WithName("ns1-watch") .WithDeletionTimestamp() .Watch(po => logSeen(po.metadata.name)); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-1) ```json {"level":30,"time":,"pid":,"hostname":"pepr-watcher-","msg":"DTS: Saw a pod ns1-watch."} ``` ### Watch Controller with Delete Event [Section titled “Watch Controller with Delete Event”](#watch-controller-with-delete-event) Responds to Pod deletion events that have deletion timestamps (Watch only): ```typescript When(a.Pod) .IsDeleted() .InNamespace(namespace2) .WithName("ns2-delete") .WithDeletionTimestamp() .Watch(po => logSeen(po.metadata.name)); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-2) ```json {"level":30,"time":,"pid":,"hostname":"pepr-watcher-","msg":"DTS: Saw a pod ns2-delete."} ``` ### Complete Finalizer Workflow [Section titled “Complete Finalizer Workflow”](#complete-finalizer-workflow) This example shows the complete lifecycle: creating a finalizer, deleting the pod, and removing the finalizer: ```typescript When(a.Pod) .IsCreated() .InNamespace(namespace1) .WithName("ns1-admission") .Watch(async po => { await addFinalizer(po.metadata.name); await deleteObj(po); await removeFinalizer(po.metadata.name); }); ``` > \[!TIP] Learn more about [Pepr Filters](https://docs.pepr.dev/user-guide/filters/) > **Source:** [hello-pepr-deletion-timestamp](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-deletion-timestamp) # Resource Name Patterns > Resource Name Patterns # Filter: Resource Name Patterns [Section titled “Filter: Resource Name Patterns”](#filter-resource-name-patterns) `.WithNameRegex()` filters resources by their name using regular expressions instead of namespace names. This is useful for matching resources based on naming conventions without explicitly listing each name. Note that this matches the resource name itself, not the namespace name. ## Usage [Section titled “Usage”](#usage) Use `.WithNameRegex()` to match resource names, not namespace names. Do **not** use to match resource namespace names. To match with namespace names, see [regex namespace-all](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-regex-ns-all/README.md). ## Code Examples [Section titled “Code Examples”](#code-examples) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-regex-name/capabilities/regex-name.ts)** ### Match Resource Names Starting with “default” [Section titled “Match Resource Names Starting with “default””](#match-resource-names-starting-with-default) ```typescript When(a.ConfigMap) .IsCreated() .WithNameRegex(/^default/) .Mutate(function mutateDef(request) { request.SetAnnotation("def", "seen"); }); ``` ### Match Resource Names Ending with “-default” [Section titled “Match Resource Names Ending with “-default””](#match-resource-names-ending-with--default) ```typescript When(a.ConfigMap) .IsCreated() .WithNameRegex(/-default$/) .Mutate(function mutateNs(request) { request.SetAnnotation("obviously", "seen"); }); ``` > \[!TIP] Learn more about [Filters](https://docs.pepr.dev/user-guide/filters/#filters) > **Source:** [hello-pepr-regex-name](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-regex-name) # Namespace Name Patterns > Namespace Name Patterns # Filter: Namespace Name Patterns [Section titled “Filter: Namespace Name Patterns”](#filter-namespace-name-patterns) When a Capability defines `namespaces: []`, you can use `.InNamespaceRegex()` to match namespace names using regular expressions. This is useful for filtering namespaces by naming patterns without explicitly listing them. ## When to Use `.InNamespaceRegex()` [Section titled “When to Use .InNamespaceRegex()”](#when-to-use-innamespaceregex) * Matching namespaces based on naming conventions or patterns * Applying policies across multiple namespaces that follow a naming standard * Filtering namespaces without explicitly listing each one * Enforcing governance on dynamically created namespaces **Important**: This filter matches namespace names, not resource names. To match resource names, see [regex name](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-regex-name/README.md). The Capability must define `namespaces: []` to enable cluster-wide namespace matching with regex patterns. ## Code Example [Section titled “Code Example”](#code-example) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-regex-ns-all/capabilities/namespace.ts)** ### Match Namespaces Starting with “default” [Section titled “Match Namespaces Starting with “default””](#match-namespaces-starting-with-default) ```typescript When(a.ConfigMap) .IsCreated() .InNamespaceRegex(/^default/) .Mutate(function mutateDef(request) { request.SetAnnotation("def", "seen"); }); ``` ### Match Namespaces Ending with “-ns-all” [Section titled “Match Namespaces Ending with “-ns-all””](#match-namespaces-ending-with--ns-all) ```typescript When(a.ConfigMap) .IsCreated() .InNamespaceRegex(/-ns-all$/) .Watch(async cm => { await K8s(kind.ConfigMap).Apply({ metadata: { name: cm.metadata.name, namespace: cm.metadata.namespace, annotations: { ns: "seen", }, }, data: { "cm-uid": cm.metadata.uid, }, }); }); ``` ### Process All Namespaces Without Filter [Section titled “Process All Namespaces Without Filter”](#process-all-namespaces-without-filter) ```typescript When(a.ConfigMap) .IsCreated() .Mutate(function mutateNon(request) { request.SetAnnotation("non", "seen"); }); ``` > \[!TIP] Learn more about [Filters](https://docs.pepr.dev/user-guide/filters/#filters) > **Source:** [hello-pepr-regex-ns-all](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-regex-ns-all) # Namespace Patterns (Bounded) > Namespace Patterns (Bounded) # Filter: Namespace Patterns (Bounded) [Section titled “Filter: Namespace Patterns (Bounded)”](#filter-namespace-patterns-bounded) When a Capability defines specific namespaces, `.InNamespaceRegex()` only matches within those bounded namespaces. The regex pattern is applied to the namespace list defined in the Capability, not to all cluster namespaces. This provides both the security of namespace boundaries and the flexibility of pattern matching. ## When to Use [Section titled “When to Use”](#when-to-use) Use `.InNamespaceRegex()` with bounded namespaces for: * Applying regex patterns within a restricted set of namespaces * Combining namespace security boundaries with flexible pattern matching * Multi-tenant environments where capabilities should only operate in specific namespaces * Matching subsets of namespaces that follow naming patterns within a defined scope > \[!TIP] You can combine `.InNamespaceRegex()` with `.WithNameRegex()` to filter both namespace and resource names for more precise targeting. ## Code Example [Section titled “Code Example”](#code-example) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-regex-ns-bounded/capabilities/namespace.ts)** ### Match Namespaces Ending with “-alpha” [Section titled “Match Namespaces Ending with “-alpha””](#match-namespaces-ending-with--alpha) ```typescript When(a.ConfigMap) .IsCreated() .InNamespaceRegex(/-alpha$/) .Mutate(function mutateAlpha(request) { request.SetAnnotation("a", "alpha"); }); ``` ### Combine Namespace and Resource Name Regex [Section titled “Combine Namespace and Resource Name Regex”](#combine-namespace-and-resource-name-regex) ```typescript When(a.ConfigMap) .IsCreated() .InNamespaceRegex(/-bravo$/) .WithNameRegex(/^two/) .Mutate(function mutateBravo(request) { request.SetAnnotation("b", "bravo"); }); ``` ### Process All Bounded Namespaces [Section titled “Process All Bounded Namespaces”](#process-all-bounded-namespaces) ```typescript When(a.ConfigMap) .IsCreated() .Mutate(function mutateCharlie(request) { request.SetAnnotation("c", "charlie"); }); ``` > \[!TIP] Learn more about [Filters](https://docs.pepr.dev/user-guide/filters/#filters) > **Source:** [hello-pepr-regex-ns-bounded](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-regex-ns-bounded) # Namespace Pattern Mismatch > Namespace Pattern Mismatch # Filter: Namespace Pattern Mismatch [Section titled “Filter: Namespace Pattern Mismatch”](#filter-namespace-pattern-mismatch) Demonstrates incorrect usage where `.InNamespaceRegex()` is used but the regex pattern does not match any of the namespaces defined in the Capability’s namespace list. When a Capability defines specific namespaces, `.InNamespaceRegex()` patterns must match at least one namespace in the Capability’s list. If the regex pattern does not match any bounded namespaces, the action will never trigger. This example demonstrates what **not** to do. When using `.InNamespaceRegex()` with bounded namespaces: * Always verify your regex pattern matches at least one namespace in the Capability’s list * Test your regex patterns before deployment to ensure they match intended namespaces * Consider using simple namespace filters if exact matching is needed * Use this pattern as a debugging reference when actions aren’t triggering as expected ## Code Example [Section titled “Code Example”](#code-example) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-regex-ns-wrong/capabilities/namespace.ts)** ### Capability Bound to Specific Namespace [Section titled “Capability Bound to Specific Namespace”](#capability-bound-to-specific-namespace) ```typescript export const HelloPeprNamespace = new Capability({ name: "hello-pepr-namespace", description: "hello-pepr-namespace", namespaces: ["hello-pepr-namespace"], }); ``` ### Regex Pattern That Does Not Match [Section titled “Regex Pattern That Does Not Match”](#regex-pattern-that-does-not-match) ```typescript When(a.ConfigMap) .IsCreated() .InNamespaceRegex(/^wrong/) .Validate(async function validateWrong(request) { return request.Approve(); }); ``` #### This action will never trigger because: [Section titled “This action will never trigger because:”](#this-action-will-never-trigger-because) 1. The Capability is bound to `["hello-pepr-namespace"]` 2. The action uses regex pattern `/^wrong/` which matches namespaces starting with “wrong” 3. The namespace “hello-pepr-namespace” does not start with “wrong”, so no match occurs > \[!TIP] Learn more about [Filters](https://docs.pepr.dev/user-guide/filters/#filters) > **Source:** [hello-pepr-regex-ns-wrong](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-regex-ns-wrong) # GenericKind > GenericKind # GenericKind [Section titled “GenericKind”](#generickind) `GenericKind` is the base class for working with any Kubernetes resource without requiring generated types. When working with Custom Resources (CRDs) that don’t have pre-generated TypeScript types, you can use `a.GenericKind` with the resource’s GVK (Group, Version, Kind) to create Pepr actions. This is especially useful for quick iteration or when type definitions aren’t available yet. ## When to Use [Section titled “When to Use”](#when-to-use) Use `GenericKind` when: * Working with CRDs without TypeScript types * Testing CRDs during development * Handling dynamic or third-party Custom Resources For production use with well-defined CRDs, consider generating TypeScript types using `kubernetes-fluent-client` for better type safety. ## Code Example [Section titled “Code Example”](#code-example) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-generic-kind/capabilities/genericKind.ts)** ```typescript When(a.GenericKind, { group: "pepr.dev", version: "v1", kind: "Unicorn", }) .IsCreated() .WithName("create") .Mutate(() => { Log.info("GenericKind created and mutated"); }) .Validate(instance => { Log.info("GenericKind created and validated"); return instance.Approve(); }) .Watch(() => { Log.info("GenericKind created and watched"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"GenericKind deleted mutated"} {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"GenericKind deleted validated"} {"level":30,"time":,"pid":,"hostname":"pepr-watcher-","msg":"GenericKind deleted watched"} ``` ```typescript When(a.GenericKind, { group: "pepr.dev", version: "v1", kind: "Unicorn", }) .IsDeleted() .WithName("delete") .Mutate(() => { Log.info("GenericKind deleted mutated"); }) .Validate(instance => { Log.info("GenericKind deleted validated"); return instance.Approve(); }) .Watch(() => { Log.info("GenericKind deleted watched"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-1) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"GenericKind deleted mutated"} {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"GenericKind deleted validated"} {"level":30,"time":,"pid":,"hostname":"pepr-watcher-","msg":"GenericKind deleted watched"} ``` > \[!TIP] Learn more about [Custom Resources](https://docs.pepr.dev/user-guide/custom-resources/) > **Source:** [hello-pepr-generic-kind](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-generic-kind) # Hooks > Hooks # Hooks [Section titled “Hooks”](#hooks) Pepr actions (Mutate and Validate) automatically translate to Kubernetes admission webhooks. This example demonstrates which webhook types are triggered for different action configurations. When you define Pepr actions, they automatically register as Kubernetes admission webhooks: * `Mutate()` actions → MutatingWebhookConfiguration * `Validate()` actions → ValidatingWebhookConfiguration ## When to Use [Section titled “When to Use”](#when-to-use) Use this pattern when you need to: * Understand how Pepr actions map to Kubernetes webhooks * Debug admission webhook configurations * Verify which webhook types are registered for your module * Troubleshoot webhook ordering and execution ## Code Examples [Section titled “Code Examples”](#code-examples) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-hooks/capabilities/hooks.ts)** **Mutate and Validate:** Creates both **MutatingWebhook** and **ValidatingWebhook** entries for the same resource. ```typescript When(a.Secret) .IsCreated() .InNamespace(name) .WithName("mutate-and-validate") .Mutate(function mutateMutVal() {}); When(a.Secret) .IsCreated() .InNamespace(name) .WithName("mutate-and-validate") .Validate(function validateMutVal(request) { return request.Approve(); }); ``` > \[!TIP] Learn more about [Pepr Actions](https://docs.pepr.dev/actions/) > **Source:** [hello-pepr-hooks](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-hooks) # Load Test Module > Load Test Module # Load Test Module [Section titled “Load Test Module”](#load-test-module) This module isn’t intended as a Module Author example: it has been designed to act as a load generator for use in [Pepr Load Testing](https://github.com/defenseunicorns/pepr/blob/main/hack/load.cli.ts). ## Usage [Section titled “Usage”](#usage) 1. Assuming this workstation project folder layout: ```bash ├── pepr │ ├── hack │ │ ├── load.cli.ts │ ├── [...] ├── pepr-excellent-examples │ ├── hello-pepr-load │ ├── [...] ``` 2. Move into Pepr project directory: ```bash cd ``` 3. Prepare Pepr artifacts for test: ```bash npx ts-node hack/load.cli.ts prep ./ ``` 4. Bring up test cluster: ```bash npx ts-node hack/load.cli.ts cluster up ``` 5. Deploy Pepr artifacts to test cluster: ```bash npx ts-node hack/load.cli.ts deploy ./pepr-0.0.0-development.tgz ./pepr-dev.tar ../pepr-excellent-examples/hello-pepr-load ``` 6. Run load test against test cluster (using `configmap.yaml` load): ```bash npx ts-node hack/load.cli.ts run ../pepr-excellent-examples/hello-pepr-load capabilities/configmap.yaml ``` 7. Post-process load test data and generate summary: ```bash npx ts-node hack/load.cli.ts post ``` 8. Create a graph of post-processed load test data: ```bash npx ts-node hack/load.cli.ts graph ``` 9. Bring down test cluster: ```bash npx ts-node hack/load.cli.ts cluster down ``` > **Source:** [hello-pepr-load](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-load) # Module > Module # Module [Section titled “Module”](#module) This is a Pepr Module. [Pepr](https://github.com/defenseunicorns/pepr) is a type-safe Kubernetes middleware system. The `capabilities` directory contains all the capabilities for this module. By default, a capability is a single typescript file in the format of `capability-name.ts` that is imported in the root `pepr.ts` file as `import { HelloPepr } from "./capabilities/hello-pepr";`. Because this is typescript, you can organize this however you choose, e.g. creating a sub-folder per-capability or common logic in shared files or folders. Example Structure: ```plaintext Module Root ├── package.json ├── pepr.ts └── capabilities ├── example-one.ts ├── example-three.ts └── example-two.ts ``` > \[!TIP] Learn more about [Modules](https://docs.pepr.dev/user-guide/pepr-modules/) > **Source:** [hello-pepr](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr) # Cluster-Wide Binding > Cluster-Wide Binding # Namespace: Cluster-Wide Binding [Section titled “Namespace: Cluster-Wide Binding”](#namespace-cluster-wide-binding) When a Capability defines `namespaces: []`, the module can process resources in any namespace cluster-wide. You can use `.InNamespace()` to filter for specific namespaces, or omit namespace filters entirely to process all namespaces. ## When to Use [Section titled “When to Use”](#when-to-use) Use `namespaces: []` for: * Cluster-wide policies that need to apply everywhere * Platform-level capabilities that monitor or modify resources across all namespaces * Cross-namespace operations and reporting * Dynamic environments where namespaces are frequently created and destroyed **Important**: With `namespaces: []`, you have maximum flexibility but also maximum scope. Use `.InNamespace()` filters when you need to target specific namespaces to limit the scope of your actions. ## Code Example [Section titled “Code Example”](#code-example) View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-ns-all/capabilities/namespace.ts) ### Filter for Specific Namespace [Section titled “Filter for Specific Namespace”](#filter-for-specific-namespace) ```typescript When(a.ConfigMap) .IsCreated() .InNamespace("default") .Mutate(function mutateDef(request) { request.SetAnnotation("def", "seen"); }); ``` ### Process All Namespaces [Section titled “Process All Namespaces”](#process-all-namespaces) ```typescript When(a.ConfigMap) .IsCreated() .Mutate(function mutateNon(request) { request.SetAnnotation("non", "seen"); }); ``` > \[!TIP] Learn more about [Pepr Filters](https://docs.pepr.dev/user-guide/filters) > **Source:** [hello-pepr-ns-all](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-ns-all) # Bounded > Bounded # Namespace: Bounded [Section titled “Namespace: Bounded”](#namespace-bounded) When a capability defines specific namespaces like `namespaces: ["alpha", "bravo"]`, the module only processes resources in those namespaces. This provides an important security boundary, ensuring actions only trigger within the defined namespace scope. ## When to Use [Section titled “When to Use”](#when-to-use) Use bounded namespaces for: * Security-sensitive operations that should only affect specific namespaces * Multi-tenant environments where capabilities must be isolated to certain tenants * Development/staging/production namespace separation * Limiting the blast radius of mutations or validations **Important**: All actions within the capability, even those without `.InNamespace()` filters, will only trigger for resources in the bounded namespaces. You can use `.InNamespace()` to further filter within those bounds. ## Code Example [Section titled “Code Example”](#code-example) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-ns-bounded/capabilities/namespace.ts)** ### Filter for Alpha Namespace [Section titled “Filter for Alpha Namespace”](#filter-for-alpha-namespace) ```typescript When(a.ConfigMap) .IsCreated() .InNamespace(alpha) .Mutate(function mutateAlpha(request) { request.SetAnnotation("a", "alpha"); }); ``` ### Filter for Bravo Namespace [Section titled “Filter for Bravo Namespace”](#filter-for-bravo-namespace) ```typescript When(a.ConfigMap) .IsCreated() .InNamespace(bravo) .Mutate(function mutateBravo(request) { request.SetAnnotation("b", "bravo"); }); ``` ### Process All Bounded Namespaces [Section titled “Process All Bounded Namespaces”](#process-all-bounded-namespaces) ```typescript When(a.ConfigMap) .IsCreated() .Mutate(function mutateCharlie(request) { request.SetAnnotation("c", "charlie"); }); ``` > \[!TIP]Learn more about [Pepr Filters](https://docs.pepr.dev/user-guide/filters) > **Source:** [hello-pepr-ns-bounded](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-ns-bounded) # Filter Mismatch > Filter Mismatch # Namespace: Filter Mismatch [Section titled “Namespace: Filter Mismatch”](#namespace-filter-mismatch) When a Capability defines specific namespaces, actions cannot use `.InNamespace()` with namespaces outside that list. The action will never trigger because the namespace filter references a namespace the Capability is not bound to. This example demonstrates what **not** to do. When using `.InNamespace()` with bounded namespaces: * Always verify the namespace in your `.InNamespace()` filter exists in the Capability’s namespace list * Double-check your Capability’s `namespaces` array includes all namespaces you’re filtering for * Use `namespaces: []` if you need to operate across all cluster namespaces * Use this pattern as a debugging reference when actions aren’t triggering as expected Always ensure `.InNamespace()` filters reference namespaces that are either in the Capability’s namespace list or use `namespaces: []` to allow all namespaces. ## Code Example [Section titled “Code Example”](#code-example) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-ns-wrong/capabilities/namespace.ts)** ### Capability Bound to Specific Namespace [Section titled “Capability Bound to Specific Namespace”](#capability-bound-to-specific-namespace) ```typescript export const HelloPeprNamespace = new Capability({ name: "hello-pepr-namespace", description: "hello-pepr-namespace", namespaces: ["hello-pepr-namespace"], }); ``` ### Action Using Wrong Namespace [Section titled “Action Using Wrong Namespace”](#action-using-wrong-namespace) ```typescript When(a.ConfigMap) .IsCreated() .InNamespace("wrong") .Validate(async function validateWrong(request) { return request.Approve(); }); ``` #### This action will never trigger because: [Section titled “This action will never trigger because:”](#this-action-will-never-trigger-because) 1. The Capability is bound to `["hello-pepr-namespace"]` 2. The action tries to filter for namespace `"wrong"` 3. Since `"wrong"` is not in the Capability’s namespace list, this action can never process any resources > \[!TIP] Learn more about [Pepr Filters](https://docs.pepr.dev/user-guide/filters) > **Source:** [hello-pepr-ns-wrong](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-ns-wrong) # OnSchedule Automated Task Execution > OnSchedule Automated Task Execution # OnSchedule Automated Task Execution [Section titled “OnSchedule Automated Task Execution”](#onschedule-automated-task-execution) This example demonstrates the Pepr `OnSchedule` API for executing code at predefined intervals, providing a simple alternative to Kubernetes CronJobs. `OnSchedule` enables automated code execution at regular intervals without complex CronJob configuration. It runs at the top level within a Capability and is backed by PeprStore to safeguard against schedule loss following pod restarts. ## When to Use [Section titled “When to Use”](#when-to-use) Use `OnSchedule` for: * Day 2 operations and maintenance tasks * Periodic health checks or status updates * Token or credential refresh operations * Scheduled data cleanup or archival **Important**: Intervals should be 30 seconds or longer due to the storage mechanism. Jobs should be idempotent. ## Code Examples [Section titled “Code Examples”](#code-examples) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-onschedule/capabilities/onschedule.ts)** **Run Forever:** ```typescript OnSchedule({ name: "forever", every: 10, unit: "seconds", run: async () => { Log.info({ schedule: "forever" }); }, }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","schedule":"forever"} {"level":30,"time":,"pid":,"hostname":"pepr-","schedule":"forever"} {"level":30,"time":,"pid":,"hostname":"pepr-","schedule":"forever"} ``` **Run N Times:** ```typescript OnSchedule({ name: "n-times", every: 10, unit: "seconds", completions: 2, run: async () => { Log.info({ schedule: "n-times" }); }, }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-1) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","schedule":"n-times"} {"level":30,"time":,"pid":,"hostname":"pepr-","schedule":"n-times"} ``` **Run Once with Delay:** ```typescript const want = new Date(Date.now()); OnSchedule({ name: "once-delayed", every: 10, unit: "seconds", startTime: want, completions: 1, run: async () => { Log.info({ schedule: "once-delayed", want: want.valueOf(), }); }, }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-2) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","schedule":"once-delayed"} ``` > \[!TIP] Learn more about [OnSchedule](https://docs.pepr.dev/user-guide/onschedule/) > **Source:** [hello-pepr-onschedule](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-onschedule) # Soak Test > Soak Test # Soak Test [Section titled “Soak Test”](#soak-test) * [Background](#background) * [Cluster Setup](#cluster-setup) * [k3d](#k3d) * [Kind](#kind) * [Get Started](#get-started) * [install istio](#install-istio) * [install kubeshark](#install-kubeshark) ## Background [Section titled “Background”](#background) Pepr watches for `Pods` with labels `api` and `bug` and for `Secrets` with label `deletedeletedelete` in `pepr-demo` namespace. A successful soak should result in: 1. No pods in the `pepr-demo` namespace 2. No secrets in the `pepr-demo` namespace The Watcher deployment is running at `LOG_LEVEL` debug while the admission deployment is on info to keep the irrelevant noise down. ## Cluster Setup [Section titled “Cluster Setup”](#cluster-setup) #### k3d [Section titled “k3d”](#k3d) Create a k3d cluster with audit logging enabled ```yaml cat < audit-policy.yaml apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata EOF k3d cluster create auditer \ --k3s-arg '--kube-apiserver-arg=audit-policy-file=/etc/kubernetes/policies/audit-policy.yaml@server:*' \ --k3s-arg '--kube-apiserver-arg=audit-log-path=/var/log/kubernetes/audit.log@server:*' \ --k3s-arg '--kube-apiserver-arg=audit-log-format=json@server:*' \ --volume $(pwd)/audit-policy.yaml:/etc/kubernetes/policies/audit-policy.yaml ``` View audit logs ```bash docker exec -it k3d-auditer-server-0 cat /var/log/kubernetes/audit.log ``` #### Kind [Section titled “Kind”](#kind) Create a kind cluster with auditing. ```yaml cat < audit-policy.yaml apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata EOF cat < kind-config.yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane kubeadmConfigPatches: - | kind: ClusterConfiguration apiServer: # enable auditing flags on the API server extraArgs: audit-log-path: /var/log/kubernetes/kube-apiserver-audit.log audit-policy-file: /etc/kubernetes/policies/audit-policy.yaml # mount new files / directories on the control plane extraVolumes: - name: audit-policies hostPath: /etc/kubernetes/policies mountPath: /etc/kubernetes/policies readOnly: true pathType: "DirectoryOrCreate" - name: "audit-logs" hostPath: "/var/log/kubernetes" mountPath: "/var/log/kubernetes" readOnly: false pathType: DirectoryOrCreate # mount the local file on the control plane extraMounts: - hostPath: ./audit-policy.yaml containerPath: /etc/kubernetes/policies/audit-policy.yaml readOnly: true EOF kind create cluster --config kind-config.yaml ``` Make sure you got audit logs ```bash docker exec kind-control-plane cat /var/log/kubernetes/kube-apiserver-audit.log ``` Troubleshoot ```bash docker exec kind-control-plane ls /etc/kubernetes/policies ``` expected ```bash audit-policy.yaml ``` API Server contain the mounts and arugments? ```bash docker exec kind-control-plane cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep audit ``` expected ```yaml - --audit-log-path=/var/log/kubernetes/kube-apiserver-audit.log - --audit-policy-file=/etc/kubernetes/policies/audit-policy.yaml name: audit-logs name: audit-policies name: audit-logs name: audit-policies ``` ## Get Started [Section titled “Get Started”](#get-started) #### Install Istio [Section titled “Install Istio”](#install-istio) Download [istioctl](https://istio.io/latest/docs/setup/getting-started/#download) ```bash curl -L https://istio.io/downloadIstio | sh - sudo mv istio-*/bin/istioctl /usr/local/bin/ ``` ```bash istioctl install --set profile=demo -y ``` Deploy the module and watch logs in one terminal ```yaml istioctl install --set profile=demo -y kubectl apply -f dist kubectl apply -f -< **Source:** [hello-pepr-soak](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-soak) # Persistent Key-Value Storage > Persistent Key-Value Storage # Store: Persistent Key-Value Storage [Section titled “Store: Persistent Key-Value Storage”](#store-persistent-key-value-storage) Pepr `Store` is a built-in key-value storage mechanism that persists and shares data within the cluster. It solves the challenge of sharing data between different actions (Mutate, Validate, and Watch) without manually creating and managing Kubernetes resources. The Store is backed by a single Custom Resource Definition (CRD) per Pepr Module, automatically created at deployment. Data is shared automatically across all replicas of the admission and watch controllers.. ## When should you use Store? [Section titled “When should you use Store?”](#when-should-you-use-store) Use `Store` when you need to share data between admission controllers and watch operations. The Store provides: * Asynchronous key-value storage backed by Kubernetes Custom Resources * Data persistence that survives pod restarts * Multiple operation modes: async (`setItem`/`removeItem`) and sync (`setItemAndWait`/`removeItemAndWait`) * Reactive updates through `subscribe()` for monitoring store changes * Initialization hooks via `onReady()` for setup before watches begin processing ## Code Snippet Examples [Section titled “Code Snippet Examples”](#code-snippet-examples) This example demonstrates the core Store API patterns: initialization in `onReady`, asynchronous operations, and synchronous operations with guaranteed persistence. View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-store/capabilities/store.ts) ### Store Initialization [Section titled “Store Initialization”](#store-initialization) ```typescript Store.onReady(async () => { const [key, val] = ["https://onReady", "yep"]; // Ensure Store.clear() in an empty store creates no errors Store.clear(); await Store.setItemAndWait(key, val); // Ensure you can set the same key again await Store.setItemAndWait(key, val); const value = Store.getItem(key); Log.info({ key, value }, "onReady"); Store.clear(); await untilTrue(gone(key)); Log.info({ key }, "onReady"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","key":"https://onReady","value":"yep","msg":"onReady"} {"level":30,"time":,"pid":,"hostname":"pepr-","key":"https://onReady","msg":"onReady"} ``` ### Asynchronous Store Operations [Section titled “Asynchronous Store Operations”](#asynchronous-store-operations) ```typescript When(a.ConfigMap) .IsCreated() .WithName("async") .Watch(async function asyncWatch() { const [key, val] = ["https://async", "yep"]; Store.setItem(key, val); await untilTrue(found(key)); Log.info({ key, value: val }, "setItem"); const value = Store.getItem(key); Log.info({ key, value }, "getItem"); Store.removeItem(key); await untilTrue(gone(key)); Log.info({ key }, "removeItem"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-1) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","key":"https://async","value":"yep","msg":"setItem"} {"level":30,"time":,"pid":,"hostname":"pepr-","key":"https://async","value":"yep","msg":"getItem"} {"level":30,"time":,"pid":,"hostname":"pepr-","key":"https://async","msg":"removeItem"} ``` ### Synchronous Store Operations [Section titled “Synchronous Store Operations”](#synchronous-store-operations) ```typescript When(a.ConfigMap) .IsCreated() .WithName("sync") .Watch(async function syncWatch() { const [key, val] = ["https://sync", "yep"]; await Store.setItemAndWait(key, val); Log.info({ key, val }, "setItemAndWait"); const value = Store.getItem(key); Log.info({ key, value }, "getItem"); await Store.removeItemAndWait(key); Log.info({ key }, "removeItemAndWait"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output-2) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","key":"https://sync","val":"yep","msg":"setItemAndWait"} {"level":30,"time":,"pid":,"hostname":"pepr-","key":"https://sync","value":"yep","msg":"getItem"} {"level":30,"time":,"pid":,"hostname":"pepr-","key":"https://sync","msg":"removeItemAndWait"} ``` > **Source:** [hello-pepr-store](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-store) # Redacting Sensitive Values in Logs > Redacting Sensitive Values in Logs # Store: Redacting Sensitive Values in Logs [Section titled “Store: Redacting Sensitive Values in Logs”](#store-redacting-sensitive-values-in-logs) When storing sensitive information like passwords or secrets in the Pepr Store, you should avoid logging the raw values. The Store itself does not automatically redact values in logs. This example shows how to retrieve sensitive data from the Store and transform it before logging to prevent exposing secrets in pod logs. ## Code Example [Section titled “Code Example”](#code-example) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-store-redact-logs/capabilities/store.ts)** ```typescript Store.onReady(async () => { const [key, val] = ["password", "***SECRET***"]; Store.clear(); await Store.setItemAndWait(key, val); const value = Store.getItem(key); Log.info( { password: `${value.replace("SECRET", "DECLASSIFIED")}` }, "getItem", ); Store.clear(); await untilTrue(gone(key)); Log.info("DONE"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output) ```json {"level":30,"time":,"pid":,"hostname":"pepr-","password":"***DECLASSIFIED***","msg":"getItem"} {"level":30,"time":,"pid":,"hostname":"pepr-","msg":"DONE"} ``` > \[!TIP] Learn more about [Pepr Store](https://docs.pepr.dev/user-guide/store/) > **Source:** [hello-pepr-store-redact-logs](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-store-redact-logs) # Resources > Resources # Store: Resources [Section titled “Store: Resources”](#store-resources) Pepr Store is backed by Kubernetes Custom Resources that can be accessed directly as `PeprStore` objects. Behind the scenes, the Pepr Store API (`Store.setItem()`, `Store.getItem()`, etc.) uses a Custom Resource Definition called `PeprStore` to persist data in the cluster. This example demonstrates how Store operations create and update PeprStore resources in Kubernetes. ## Key Points [Section titled “Key Points”](#key-points) * All Store operations for a capability are persisted in a single PeprStore CRD * The PeprStore resource lives in the `pepr-system` namespace * Store keys can include special characters like `://` (e.g., URLs) * Store data is stored as key-value string pairs in the `data` field * Store updates are batched and sent to the Kubernetes API at 5-second intervals * The Store is eventually consistent, backed by etcd through Kubernetes ## When to Use [Section titled “When to Use”](#when-to-use) Use this pattern when you need to: * Inspect Store data directly using kubectl for debugging * Understand how Store operations map to Kubernetes resources * Verify that Store data is being persisted correctly * Access Store data from outside your Pepr module * Troubleshoot Store-related issues by examining the underlying CRD ## Code Example [Section titled “Code Example”](#code-example) > **View full example on [Github](https://github.com/defenseunicorns/pepr-excellent-examples/blob/main/hello-pepr-store-resources/capabilities/store.ts)** ```typescript When(a.ConfigMap) .IsCreated() .WithName("observe") .Watch(async function observeWatch() { Store.setItem("a", "1"); Store.setItem("b", "2"); Store.setItem("c", "3"); await Store.setItemAndWait("https://observed", "yay"); Log.info("observed"); }); ``` #### Example pod log output: [Section titled “Example pod log output:”](#example-pod-log-output) ```json {"level":30,"time":1726798504518,"pid":16,"hostname":"pepr-watcher-6dc69654c9-5ql6b","msg":"observed"} ``` ## Viewing the PeprStore Resource [Section titled “Viewing the PeprStore Resource”](#viewing-the-peprstore-resource) You can inspect the PeprStore Custom Resources using kubectl: ```bash # List all PeprStore resources kubectl get peprstores -n pepr-system # Get detailed YAML output for a specific store kubectl get peprstore -n pepr-system -o yaml ``` ### Example PeprStore YAML: [Section titled “Example PeprStore YAML:”](#example-peprstore-yaml) After the Store operations execute, the PeprStore resource will contain: ```yaml apiVersion: pepr.dev/v1 kind: PeprStore metadata: name: pepr--store-hello-pepr-store-resources namespace: pepr-system data: a: "1" b: "2" c: "3" https://observed: "yay" ``` > \[!TIP] Learn more about the [Pepr Store](https://docs.pepr.dev/user-guide/store/) > **Source:** [hello-pepr-store-resources](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/hello-pepr-store-resources) # Pepr Best Practices > Pepr Best Practices # Pepr Best Practices [Section titled “Pepr Best Practices”](#pepr-best-practices) ## Table of Contents [Section titled “Table of Contents”](#table-of-contents) * [Pepr Best Practices](#pepr-best-practices) * [High Availability](#high-availability) * [Table of Contents](#table-of-contents) * [Mutating Webhook Errors](#mutating-webhook-errors) * [Core Development](#core-development) * [Debugging](#debugging) * [Deployment](#deployment) * [Keep Modules Small](#keep-modules-small) * [Monitoring](#monitoring) * [Multiple Modules or Multiple Capabilities](#multiple-modules-or-multiple-capabilities) * [OnSchedule](#onschedule) * [Reconcile](#reconcile) * [Security](#security) * [Pepr Store](#pepr-store) * [Watch](#watch) ## High Availability [Section titled “High Availability”](#high-availability) Pepr is designed to be deployed in a high availability (HA) configuration. This means that you can run multiple instances of the Pepr Admission Controller in different availability zones or regions to ensure that your Pepr deployment is resilient to failures. By default, the Admission Controller is deployed with two replicas. For modules that have a failurePolicy of `Fail`, it is recommended to opt into `podAntiAffinity` in the helm chart by setting `admission.antiAffinity` to true in the values.yaml of the helm chart generated after running `npx pepr build`. This ensures the pods are scheduled on different nodes, providing better fault tolerance for when nodes are unavailable. Modules running on edge devices or single node environments should keep `admission.antiAffinity` set to false to avoid scheduling issues. ```yaml admission: antiAffinity: true ``` Due to the nature of having two different instances of the Admission Controller, it is important to ensure that the modules are idempotent and do not store ephemeral data since there is no guarantee which replica will serve the request. When state must be saved, use the [`PeprStore`](../../user-guide/store). ## Mutating Webhook Errors [Section titled “Mutating Webhook Errors”](#mutating-webhook-errors) When developing mutating admission policies, it is essential to include a validation step immediately after applying mutations. This ensures that the changes made by the mutating admission policy were applied correctly and do not introduce unintended inconsistencies or invalid configurations into your Kubernetes cluster. ### Why Validate After Mutating? [Section titled “Why Validate After Mutating?”](#why-validate-after-mutating) 1. Detect Misconfigurations Early: Mutating admission policies modify incoming resource configurations dynamically. Without validation, you risk introducing invalid configurations into your cluster if the mutation logic contains bugs, unintended side effects, or runs too long and causes a [Webhook Timeout](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#timeouts). 2. Maintain Cluster Integrity: By validating the mutated resource, you ensure it adheres to expected formats, standards, and constraints, maintaining the health and stability of your cluster. 3. Catch Logic Errors in Mutations: A mutation may not always produce the intended output due to edge cases, unexpected inputs, or incorrect assumptions in the mutation logic. Validation helps catch such issues early and becomes *particularly* important if your Module is [configured](../../user-guide/customization#admission-and-watcher-subparameters) to use a Webhook [failurePolicy](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#failure-policy) of `ignore`. In that case, admission requests failures *won’t prevent further processing and/or acceptance* of mutate-failed requests and could result in undesirable resources getting into your cluster! 1. Comply with Kubernetes Best Practices: Kubernetes resources must meet specific structural and functional requirements. Validating ensures compliance, preventing the risk of deployment failures or runtime errors. ### How to implement a Validate-After-Mutate Pattern [Section titled “How to implement a Validate-After-Mutate Pattern”](#how-to-implement-a-validate-after-mutate-pattern) 1. Apply the desired transformations to the resource in the `Mutate` block. 2. Validate the mutated resource in the `Validate` block to ensure it adheres to the expected structure. 3. If the validation fails, reject the resource with a descriptive message explaining the issue. ```typescript When(a.Pod) .IsCreated() .InNamespace("my-app") .WithName("database") .Mutate(po => po.SetLabel("pepr", "true")) .Validate(po => { if (po.Raw.metadata?.labels["pepr"] !== "true") { return po.Approve(); } return po.Deny("Needs pepr label set to true") }); ``` ## Core Development [Section titled “Core Development”](#core-development) When developing new features in Pepr Core, it is recommended to use `npx pepr deploy --image pepr:dev`, which will deploy Pepr’s Kubernetes manifests to the cluster with the development image. This will allow you to test your changes without having to build a new image and push it to a registry. The workflow for developing features in Pepr is: 1. Run `npm test` which will create a k3d cluster and build a development image called `pepr:dev` 2. Deploy development image into the cluster with `npx pepr deploy --image pepr:dev` ## Debugging [Section titled “Debugging”](#debugging) * [Debugging During Module Development](#debugging-during-module-development) * [Logging](#logging) * [Internal Error Occurred](#internal-error-occurred) * [Pepr Store](#pepr-store) Pepr is composed of `Modules`, `Capabilities`, and `Actions`: * [Actions](../../actions) are the blocks of code containing filters, `Mutate`, `Validate`, `Watch`, `Reconcile`, and `OnSchedule`. * [Capabilities](../../user-guide/capabilities) such as `hello-pepr.ts`. * `Modules` are the result of `npx pepr init`. You can have as many Capabilities as you would like in a Module. Pepr is a webhook-based system, meaning it is event-driven. When a resource is created, updated, or deleted, Pepr is called to perform the actions you have defined in your Capabilities. It’s common for multiple webhooks to exist in a cluster, not just Pepr. When there are multiple webhooks, the order in which they are called is not guaranteed. The only guarantee is that all of the `MutatingWebhooks` will be called before all of the `ValidatingWebhooks`. After the admission webhooks are called, the `Watch` and `Reconcile` are called. The `Reconcile` and `Watch` create a watch on the resources specified in the `When` block and are watched for changes after admission. The difference between reconcile and watch is that `Reconcile` processes events in a queue to guarantee that the events are processed in order where as watch does not. Considering that many webhooks may be modifying the same resource, it is best practice to validate the resource after mutations are made to ensure that the resource is in a valid state if it has been changed since the last mutation. ```typescript When(a.Pod) .IsCreated() .InNamespace("my-app") .WithName("database") .Mutate(pod => { pod.metadata.labels["pepr"] = "true"; return pod; }) // another mutating webhook could removed labels .Validate(pod => { if (pod.metadata.labels["pepr"] !== "true") { return pod.Approve("Label 'pepr' must be 'true'"); } return pod.Deny("Needs pepr label set to true") }); ``` *If you think your Webhook is not being called for a given resource, check the `*WebhookConfiguration`.* ### Debugging During Module Development [Section titled “Debugging During Module Development”](#debugging-during-module-development) Pepr supports breakpoints in the VSCode editor. To use breakpoints, run `npx pepr dev` in the root of a Pepr module using a JavaScript Debug Terminal. This command starts the Pepr development server running at `localhost:3000` with the `*WebhookConfiguration` configured to send `AdmissionRequest` objects to the local address. This allows you to set breakpoints in `Mutate()`, `Validate()`, `Reconcile()`, `Watch()` or `OnSchedule()` and step through module code. Note that you will need a cluster running: ```bash k3d cluster create pepr-dev --k3s-arg '--debug@server:0' --wait ``` ```typescript When(a.Pod) .IsCreated() .InNamespace("my-app") .WithName("database") .Mutate(pod => { // Set a breakpoint here pod.metadata.labels["pepr"] = "true"; return pod; }) .Validate(pod => { // Set a breakpoint here if (pod.metadata.labels["pepr"] !== "true") { return ["Label 'pepr' must be 'true'"]; } }); ``` ### Logging [Section titled “Logging”](#logging) Pepr can deploy two types of controllers: Admission and Watch. The controllers deployed are dictated by the [Actions](../../actions) called for by a given set of Capabilities (Pepr only deploys what is necessary). Within those controllers, the default log level is `info` but that can be changed to `debug` by setting the `LOG_LEVEL` environment variable to `debug`. To pull logs for all controller pods: ```bash kubectl logs -l app -n pepr-system ``` #### Admission Controller [Section titled “Admission Controller”](#admission-controller) If the focus of the debug is on a `Mutate()` or `Validate()`, the relevant logs will be from pods with label `pepr.dev/controller: admission`. ```bash kubectl logs -l pepr.dev/controller=admission -n pepr-system ``` More refined admission logs – which can be optionally filtered by the module UUID – can be obtained with [`npx pepr monitor`](#monitoring) ```bash npx pepr monitor ``` #### Watch Controller [Section titled “Watch Controller”](#watch-controller) If the focus of the debug is a `Watch()`, `Reconcile()`, or `OnSchedule()`, look for logs from pods containing label `pepr.dev/controller: watcher`. ```bash kubectl logs -l pepr.dev/controller=watcher -n pepr-system ``` ### Internal Error Occurred [Section titled “Internal Error Occurred”](#internal-error-occurred) ```bash Error from server (InternalError): Internal error occurred: failed calling webhook "pepr.dev": failed to call webhook: Post ... ``` When an internal error occurs, check the deployed `*WebhookConfiguration` resources’ timeout and failurePolicy configurations. If the failurePolicy is set to `Fail`, and a request cannot be processed within the timeout, that request will be rejected. If the failurePolicy is set to `Ignore`, given the same timeout conditions, the request will (perhaps surprisingly) be allowed to continue. If you have a validating webhook, the recommended is to set the failurePolicy to `Fail` to ensure that the request is rejected if the webhook fails. ```yaml failurePolicy: Fail matchPolicy: Equivalent timeoutSeconds: 3 ``` The failurePolicy and timeouts can be set in the Module’s `package.json` file, under the `pepr` configuration key. If changed, the settings will be reflected in the `*WebhookConfiguration` after the next build: ```json "pepr": { "uuid": "static-test", "onError": "ignore", "webhookTimeout": 10, } ``` Read [more on customization](../../user-guide/customization). ### Pepr Store Custom Resource [Section titled “Pepr Store Custom Resource”](#pepr-store-custom-resource) If you need to read all store keys, or you think the PeprStore is malfunctioning, you can check the PeprStore CR: ```bash kubectl get peprstore -n pepr-system -o yaml ``` You should run in `npx pepr dev` mode to debug the issue, see the [Debugging During Module Development](#debugging-during-module-development) section for more information. ## Deployment [Section titled “Deployment”](#deployment) Production environment deployments should be `declarative` in order to avoid mistakes. The Pepr modules should be generated with `npx pepr build` and moved into the appropriate location. Development environment deployments can use `npx pepr deploy` to deploy Pepr’s Kubernetes manifests into the cluster or `npx pepr dev` to active debug the Pepr module with breakpoints in the code editor. ## Keep Modules Small [Section titled “Keep Modules Small”](#keep-modules-small) Modules are minified and built JavaScript files that are stored in a Kubernetes Secret in the cluster. The Secret is mounted in the Pepr Pod and is processed by Pepr Core. Due to the nature of the module being packaged in a Secret, it is recommended to keep the modules as small as possible to avoid hitting the [1MB limit](https://kubernetes.io/docs/concepts/configuration/secret/#restriction-data-size) of secrets. Recommendations for keeping modules small are: * Don’t repeat yourself * Only import the part of the library modules that you need It is suggested to lint and format your modules using your own linting and formatting tools (e.g., ESLint and Prettier). ## Monitoring [Section titled “Monitoring”](#monitoring) Pepr can monitor Mutations and Validations from Admission Controller the through the `npx pepr monitor [module-uuid]` command. This command will display neatly formatted log showing approved and rejected Validations as well as the Mutations. If `[module-uuid]` is not supplied, then it uses all Pepr admission controller logs as the data source. If you are unsure of what modules are currently deployed, issue `npx pepr uuid` to display the modules and their descriptions. ```plaintext ✅ MUTATE pepr-demo/pepr-demo (50c5d836-335e-4aa5-8b56-adecb72d4b17) ✅ VALIDATE pepr-demo/example-2 (01c1d044-3a33-4160-beb9-01349e5d7fea) ❌ VALIDATE pepr-demo/example-evil-cm (8ee44ca8-845c-4845-aa05-642a696b51ce) [ 'No evil CM annotations allowed.' ] ``` ## Multiple Modules or Multiple Capabilities [Section titled “Multiple Modules or Multiple Capabilities”](#multiple-modules-or-multiple-capabilities) Each module can have its own MutatingWebhookConfiguration, ValidatingWebhookConfiguration, Admission Controller, Watch Controller, and Store depending on the actions it contains. This allows for each module to be deployed independently of each other. However, creating multiple modules creates overhead on the kube-apiserver, and the cluster. Due to the overhead costs, it is recommended to deploy multiple capabilities that share the same resources (when possible). This will simplify analysis of which capabilities are responsible for changes on resources. However, there are some cases where multiple modules makes sense. For instance different teams owning separate modules, or one module for Validations and another for Mutations. If you have a use-case where you need to deploy multiple modules it is recommended to separate concerns by operating in different namespaces. ## OnSchedule [Section titled “OnSchedule”](#onschedule) `OnSchedule` is supported by a `PeprStore` to safeguard against schedule loss following a pod restart. It is utilized at the top level, distinct from being within a `Validate`, `Mutate`, `Reconcile` or `Watch`. Recommended intervals are 30 seconds or longer, and jobs are advised to be idempotent, meaning that if the code is applied or executed multiple times, the outcome should be the same as if it had been executed only once. A major use-case for `OnSchedule` is day 2 operations. ## Security [Section titled “Security”](#security) To enhance the security of your Pepr Controller, we recommend following these best practices: * Regularly update Pepr to the latest stable release. * Secure Pepr through RBAC using [scoped mode](../../user-guide/rbac#scoped) taking into account access to the Kubernetes API server needed in the callbacks. * Practice the principle of least privilege when assigning roles and permissions and avoid giving the service account more permissions than necessary. * Use NetworkPolicy to restrict traffic from Pepr Controllers to the minimum required. * Limit calls from Pepr to the Kubernetes API server to the minimum required. * Set webhook failure policies to `Fail` to ensure that the request is rejected if the webhook fails. More Below\.. * Verify the source and build identity of Pepr npm releases by following the [release provenance guidance](../provenance). When using Pepr as a `Validating` Webhook, it is recommended to set the Webhook’s `failurePolicy` to `Fail`. This can be done in your Pepr module in the`values.yaml` file of the helm chart by setting `admission.failurePolicy` to `Fail` or in the `package.json` under `pepr` by setting the `onError` flag to `reject`, then running `npx pepr build` again. By following these best practices, you can help protect your Pepr Controller from potential security threats. ## Reconcile [Section titled “Reconcile”](#reconcile) Fills a similar niche to `.Watch()` – and runs in the Watch Controller – but it employs a Queue to force sequential processing of resource states once they are returned by the Kubernetes API. This allows things like operators to handle bursts of events without overwhelming the system or the Kubernetes API. It provides a mechanism to back off when the system is under heavy load, enhancing overall stability and maintaining the state consistency of Kubernetes resources, as the order of operations can impact the final state of a resource. For example, creating and then deleting a resource should be processed in that exact order to avoid state inconsistencies. ```typescript When(WebApp) .IsCreatedOrUpdated() .Validate(validator) .Reconcile(async instance => { // Do work here ``` ## Pepr Store [Section titled “Pepr Store”](#pepr-store) The store is backed by ETCD in a `PeprStore` resource, and updates happen at 5-second intervals when an array of patches is sent to the Kubernetes API Server. The store is intentionally not designed to be `transactional`; instead, it is built to be eventually consistent, meaning that the last operation within the interval will be persisted, potentially overwriting other operations. Changes to the data are made without a guarantee that they will occur simultaneously, so caution is needed in managing errors and ensuring consistency. ## Watch [Section titled “Watch”](#watch) Pepr streamlines the process of receiving timely change notifications on resources by employing the `Watch` mechanism. It is advisable to opt for `Watch` over `Mutate` or `Validate` when dealing with more extended operations, as `Watch` does not face any [timeout limitations](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#timeouts). Additionally, `Watch` proves particularly advantageous for monitoring previously existing resources within a cluster. One compelling scenario for leveraging `Watch` is when there is a need to chain API calls together, allowing `Watch` operations to be sequentially executed following `Mutate` and `Validate` actions. ```typescript When(a.Pod) .IsCreated() .InNamespace("my-app") .WithName("database") .Mutate(pod => // .... ) .Validate(pod => // .... ) .Watch(async (pod, phase) => { Log.info({pod}, `Pod was ${phase}.`); // do consecutive api calls ``` # Frequently Asked Questions > Frequently Asked Questions # Frequently Asked Questions [Section titled “Frequently Asked Questions”](#frequently-asked-questions) ## What is the `PEPR_WATCH_MODE` environment variable? [Section titled “What is the PEPR\_WATCH\_MODE environment variable?”](#what-is-the-pepr_watch_mode-environment-variable) The `PEPR_WATCH_MODE` environment variable is used to determine if Pepr binds Admission Endpoints and runs as an Admission Controller or sets up the watch processor and runs as a Kubernetes Controller. For greater availability the Admission Controller runs with two replicas. The variable is set during `npx pepr build` when the manifests are generated. This variable should not be set by the user. ## Difference between `.Watch()` vs. `.Reconcile()` in Pepr [Section titled “Difference between .Watch() vs. .Reconcile() in Pepr”](#difference-between-watch-vs-reconcile-in-pepr) `.Watch()` and `.Reconcile()` are two distinct mechanisms in Pepr used to handle Kubernetes events. The core difference is **when** and **how** they process events: * **`.Watch()`** processes events **immediately** as they arrive, without any ordering guarantees. * **`.Reconcile()`** processes events **sequentially** through a queue, based on a user-defined configuration. ### Why This Matters [Section titled “Why This Matters”](#why-this-matters) If you’re performing multiple operations on the same resource, using `.Watch()` can lead to **out-of-order event handling**. **Example:**\ Suppose your controller updates a resource and then deletes it. With `.Watch()`, the delete event might be processed **before** the update if the update takes longer to complete. This could result in the resource being deleted before it was properly updated — leading to **unexpected behavior**. ### How `.Reconcile()` Solves This [Section titled “How .Reconcile() Solves This”](#how-reconcile-solves-this) `.Reconcile()` ensures **event order is preserved** by placing events into a queue. Events are then processed **in the order they were received**, regardless of their processing time. This reduces race conditions and increases predictability. ### `Reconcile()` Queue Configurations [Section titled “Reconcile() Queue Configurations”](#reconcile-queue-configurations) You can configure `.Reconcile()` with different queue granularities depending on your desired level of isolation: | Option | Description | Example | | ------------ | ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `kind` | One queue per resource kind. | All Deployments share a queue, all Pods share a queue, etc. | | `kindNs` | One queue per kind **within each namespace**. | Deployments in `default` share a queue; Pods in `default` share a different one. | | `kindNsName` | One queue **per resource**, keyed by kind + namespace + name. *(Recommended)* | Events for `Deployment/default/my-deployment` are processed in their own dedicated queue. | | `global` | A **single global queue** for all events. | All events across all kinds and namespaces are funneled into one queue. | ### Summary [Section titled “Summary”](#summary) * Use **`.Watch()`** when immediate reaction is acceptable and event order doesn’t matter. * Use **`.Reconcile()`** when consistency and **event order are critical**, especially for updates, deletes, or resource coordination. * Prefer `kindNsName` mode for most real-world workloads to ensure per-resource consistency. ## What is a module author? [Section titled “What is a module author?”](#what-is-a-module-author) A module author is someone who creates a [Pepr Module](../../user-guide/pepr-modules) , which defines capabilities to enforce or apply desired state in a Kubernetes cluster. If you’ve run `npx pepr init` to create a module, congratulations — you’re a module author. ## How do I remove the punycode warning? [Section titled “How do I remove the punycode warning?”](#how-do-i-remove-the-punycode-warning) By default, warnings are removed. You can allow warnings by setting the `PEPR_NODE_WARNINGS` environment variable. ```bash PEPR_NODE_WARNINGS="true" ``` If you allow warnings, you can disable the specific punycode warning by: ```bash export NODE_OPTIONS="--disable-warning=DEP0040" ``` or ```bash npx --node-options="--disable-warning=DEP0040" pepr [command] ``` ## How does Pepr compare to Operator SDK? [Section titled “How does Pepr compare to Operator SDK?”](#how-does-pepr-compare-to-operator-sdk) Pepr and Operator SDK are both frameworks used for building Kubernetes operators and admission controllers. While they share a common goal of simplifying the creation of Kubernetes operators and enhancing Kubernetes functionality, they have different approaches and features. Similarities: * **Scaffolding**: Automatically generate boilerplate code for new operators and Kubernetes manifests for building controllers. * **Helper Functions**: Provide utility functions to interact with Kubernetes resources and manage the lifecycle of Kubernetes resources. * **Admission Webhooks and Kubernetes Controllers**: Both support building admission and Kubernetes controllers by reacting to changes in the cluster in an automated way. Differences: * **Main Goals**: Operator SDK is mainly focused on building operators and later included support for Webhooks. In contrast, Pepr started out as a framework for building Webhooks and later added support for building operators via [Kubernetes-Fluent-Client](https://github.com/defenseunicorns/kubernetes-fluent-client) through [Watch](../../actions/watch) and [Reconcile](../../actions/reconcile). * **Language Support**: Operator SDK supports Go, Ansible, and Helm, while Pepr is written in TypeScript and designed with an English style fluent API for simplicity. * **Lifecycle Management**: Operator SDK provides tools for managing the lifecycle of operators through OLM (Operator Lifecycle Manager), while Pepr relies on [Helm](../../user-guide/customization) for upgrades. * **Complexity**: Operator SDK uses native Kubernetes Go libraries for deep integration with Kubernetes resources, while Pepr exposes a high-level abstraction allowing users to focus on business logic. * **Easy Setup**: While both make it easy to initialize a new project, Pepr comes with an out-of-the-box `hello-pepr.ts` example which demonstrates how to use Pepr effectively. ## How does Pepr compare to Kyverno? [Section titled “How does Pepr compare to Kyverno?”](#how-does-pepr-compare-to-kyverno) Although Pepr and Kyverno have similarities, Pepr is very different than Kyverno. They have very different mission statements. Pepr focuses on making operators as easy as possible. Kyverno focuses on reporting, not building operators. Similarities: * Both have Mutating Webhooks that can dynamically change resources before admission * Both have Validating Webhooks to configure what can/cannot go through admission * Both provide a way to react to changes to pre-existing cluster resources (i.e., resources that have already gone through admission) Differences: * Pepr is more like a “framework” than a tool. In Pepr you create a Pepr [Module](../../user-guide/pepr-modules). In the Pepr module you define [capabilities](../../user-guide/capabilities) that enforce / apply desired cluster state. * Pepr is written in TypeScript. Kyverno is written in Go. * Pepr provides the flexibility of a full-fledged, strongly typed programming language to decide what decisions to make based on events happening in the cluster. With Kyverno, you are limited to the constraints of YAML. * Pepr can be used to reconcile events in order, similar to Kube-Builder or Operator SDK. * Pepr can apply a CustomResourceDefinition and control cluster state based on that custom resource. Both Pepr and Kyverno are great tools. Which one to use for your project depends on your use case. ## How do I add custom labels to Pepr’s Kubernetes manifests? [Section titled “How do I add custom labels to Pepr’s Kubernetes manifests?”](#how-do-i-add-custom-labels-to-peprs-kubernetes-manifests) During the build process, custom labels can be added the `pepr-system` namespace based on the `package.json`. Checkout the [Customizing with package.json](../../user-guide/customization#packagejson-configurations-table). The following example shows how to add custom namespace labels. ```json "pepr": { "name": "new-release", ... "customLabels": { "namespace": { "istio-injection": "enabled", "app.kubernetes.io/name": "new-release" } }, ... } ``` The resulting namespace will be generated after `npx pepr build`. ```yaml apiVersion: v1 kind: Namespace metadata: name: pepr-system labels: istio-injection: enabled app.kubernetes.io/name: new-release ``` ## My Pepr version is not the latest [Section titled “My Pepr version is not the latest”](#my-pepr-version-is-not-the-latest) If you notice your Pepr version does not correspond to the latest release in GitHub after doing `npx pepr -V`, clearing the NPX cache can often resolve the issue. Run the cache clearing command ```bash npx clear-npx-cache ``` If you want to ensure the cache has been cleared, you can check the cache directory. The location of this directory varies based on your operating system and configuration. However, you can generally find it in your system’s home directory under `.npm`. > \[!NOTE] If you are inside of the Pepr Core repo (), then it is normal for `npx pepr -V` to return `0.0.0-development`. ## I’ve found a bug, what should I do? [Section titled “I’ve found a bug, what should I do?”](#ive-found-a-bug-what-should-i-do) Please report it by opening an issue in the [Pepr GitHub repository](https://github.com/defenseunicorns/pepr/issues). Please include as much information as possible in your bug report, including: * The version of Pepr you are using * The version of Kubernetes you are using ## I’ve found a security issue, what should I do? [Section titled “I’ve found a security issue, what should I do?”](#ive-found-a-security-issue-what-should-i-do) Security issues should be reported privately, via [email](mailto:security@defenseunicorns.com). You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. ## I have a feature request, what should I do? [Section titled “I have a feature request, what should I do?”](#i-have-a-feature-request-what-should-i-do) Please let us know by opening an issue in the [Pepr GitHub repository](https://github.com/defenseunicorns/pepr/issues). ## How do I get help with Pepr? [Section titled “How do I get help with Pepr?”](#how-do-i-get-help-with-pepr) If you have a question about Pepr, please open an issue in the [Pepr GitHub repository](https://github.com/defenseunicorns/pepr/issues) or contact us through the Pepr channel on the Kubernetes Slack. # Module Examples > Module Examples # Module Examples [Section titled “Module Examples”](#module-examples) We maintain a repo of Pepr examples at [Pepr Excellent Examples](https://github.com/defenseunicorns/pepr-excellent-examples). Each example is a complete module that can be deployed to a Pepr instance. # Pepr Release Provenance > Pepr Release Provenance # Pepr Release Provenance [Section titled “Pepr Release Provenance”](#pepr-release-provenance) Current Pepr releases publish provenance for the npm package. Consumers can use it to verify the package’s source repository, workflow, commit, and digest. ## Scope [Section titled “Scope”](#scope) This guidance applies to the `pepr-.tgz` npm package. It does not cover Pepr container images, source archives, or other release artifacts. Pepr publishes two records for the package: | Record | Subject | Digest | Verify with | | --------------------------- | ------------------------ | ------- | ----------------------- | | GitHub Artifact Attestation | `pepr-.tgz` | SHA-256 | `gh attestation verify` | | npm provenance | `pkg:npm/pepr@` | SHA-512 | `npm audit signatures` | The subject names and digest algorithms differ, but both records identify the same published package bytes. ## Verify a Release [Section titled “Verify a Release”](#verify-a-release) Always verify an explicit version instead of a mutable tag such as `latest`. These examples require recent versions of the GitHub CLI, npm, and `jq`. Download the package and verify its GitHub attestation: ```bash version="2.0.2" artifact=$(npm pack "pepr@$version" --json | jq -r '.[0].filename') gh attestation verify "$artifact" \ --repo defenseunicorns/pepr \ --signer-workflow defenseunicorns/pepr/.github/workflows/release.yml ``` The command fails if the file’s digest has no valid attestation from the expected repository and workflow. To inspect the provenance details: ```bash gh attestation verify "$artifact" \ --repo defenseunicorns/pepr \ --format json \ --jq '.[].verificationResult | { subject: .statement.subject, predicateType: .statement.predicateType, certificate: .signature.certificate, provenance: .statement.predicate, timestamps: .verifiedTimestamps }' ``` In a project where Pepr is installed and recorded in the package lock, verify the npm registry signature and provenance with: ```bash npm audit signatures ``` Use `npm audit signatures --json --include-attestations` to inspect the full verified Sigstore bundles. ## Understand the Result [Section titled “Understand the Result”](#understand-the-result) The verified fields answer different questions about the package: | Field | Expected value | Meaning | | ----------------------- | ---------------------------------------------------- | ----------------------------------------------------- | | Subject | The downloaded tarball’s name and digest | Identifies the exact bytes covered by the attestation | | Source repository | `defenseunicorns/pepr` | Identifies the source repository used by the build | | Signer workflow | `defenseunicorns/pepr/.github/workflows/release.yml` | Identifies the workflow that issued the attestation | | Source commit | The commit for the selected release tag | Identifies the source revision used by the build | | Builder, ref, and event | GitHub-hosted runner, `refs/heads/main`, and `push` | Describes the expected release context | Verification proves that the attestation is authentic and covers the downloaded bytes. Consumers must still decide whether the repository, workflow, commit, and build context satisfy their policy. A valid signature from an unexpected workflow should not be accepted as a Pepr release. > \[!CAUTION] Provenance links an artifact to its source and build instructions. It does not prove that the artifact is free of vulnerabilities or malicious code. ## SLSA Assurance [Section titled “SLSA Assurance”](#slsa-assurance) The `https://slsa.dev/provenance/v1` predicate identifies the provenance format. The `v1` in that URL is a schema version, not a SLSA Build level. Pepr’s npm tarball has signed provenance generated on a GitHub-hosted runner. GitHub classifies this standard Artifact Attestation model as SLSA v1.0 Build Level 2. It protects against undetected changes after the build and provides a verifiable link to the source and build instructions. Pepr does not currently claim Build Level 3 for this artifact. Build Level 3 is an additional property of the build platform’s isolation and resistance to tampering during the build; it cannot be inferred from the predicate format or a successful verification result. ## How Pepr Preserves Artifact Identity [Section titled “How Pepr Preserves Artifact Identity”](#how-pepr-preserves-artifact-identity) Pepr’s release workflow maintains one package identity across jobs: 1. The `slsa` job builds one npm tarball and records its SHA-256 digest. 2. The attestation job downloads the tarball, checks that digest, and attests the verified file with `actions/attest`. 3. The publish job downloads the same tarball, checks the same digest, and publishes that file through npm trusted publishing. 4. The provenance verification job downloads the published version from npm, compares its digest with the build output, verifies its GitHub attestation, and runs `npm audit signatures`. 5. The OCI package job repeats the digest check before pushing the tarball to the downstream registry. This prevents later jobs from silently rebuilding or substituting the package. npm trusted publishing adds the npm provenance record, while `actions/attest` adds the separate GitHub Artifact Attestation. Release workflow changes must preserve these properties: * Build the npm tarball once and use the same file for attestation and publication. * Record its digest after the build and verify it after each cross-job transfer. * Do not rebuild or modify the package after attestation. * Publish the verified tarball path rather than running another pack or build. * Limit OIDC and attestation permissions to the jobs that need them. * Keep third-party actions pinned and retain npm trusted publishing. * Treat the repository, signer workflow, subject name, and release trigger as part of the public verification contract. The provenance verification job runs after npm publication and in parallel with the OCI package and controller image jobs. Its failure marks the release workflow unsuccessful but cannot undo a completed npm publication. ## Historical Releases [Section titled “Historical Releases”](#historical-releases) | Release | npm provenance | GitHub Artifact Attestation | | ------- | ---------------------------------------------------------- | --------------------------------------------------------- | | `2.0.0` | SLSA provenance v0.2 from the former SLSA GitHub generator | Not available through the Pepr repository attestation API | | `2.0.2` | SLSA provenance v1 from npm trusted publishing | SLSA provenance v1 for the npm tarball | A missing GitHub attestation for an older release does not mean that npm provenance is also missing. Release `2.0.2` is the baseline for the current two-record verification process. ## References [Section titled “References”](#references) * [GitHub Artifact Attestations](https://docs.github.com/en/actions/concepts/security/artifact-attestations) * [GitHub guidance for SLSA Build Level 3](https://docs.github.com/en/actions/how-tos/secure-your-work/use-artifact-attestations/increase-security-rating) * [npm provenance](https://docs.npmjs.com/generating-provenance-statements/) * [npm signature verification](https://docs.npmjs.com/cli/v11/commands/npm-audit/#audit-signatures) * [SLSA security levels](https://slsa.dev/spec/v1.2/build-track-basics) # Roadmap for Pepr > Roadmap for Pepr # Roadmap for Pepr [Section titled “Roadmap for Pepr”](#roadmap-for-pepr) *2025 Roadmap* ## Phase 1: Code Quality - Experimentation [Section titled “Phase 1: Code Quality - Experimentation”](#phase-1-code-quality---experimentation) * **Q1**: * **Turn on eslint enforcement and configure settings and see no warnings**: * Eliminate circular dependencies, complexity, return statements, etc. * **Metric and Performance Baselining**: * Establish a baseline for performance and resource utilization metrics. Use this data to make informed decisions about the direction of the project in terms of Deno2 * **Nightly Release**: * Establish a nightly release process. This will help us to catch bugs early and ensure that the project is always in a releasable state. ## Phase 2: Test Reliability & CRD generation [Section titled “Phase 2: Test Reliability & CRD generation”](#phase-2-test-reliability--crd-generation) * **Q2**: * **Professional Dashboard displaying metrics and performance tests originating from CI** * **Improve integration testing with Pepr, UDS Core, and Kubernetes Fluent Client** * This focused on a more reliable release process to reduce the frequency of regressions. * \*\*Migrate old `journey/` style tests to Pepr-Excellent-Examples or newer CLI integration tests. See [pepr/#1597](https://github.com/defenseunicorns/pepr/issues/1597). * **Define Governance with Pepr** * Add support for compliance/governance validation within Pepr. See [pepr/#1823](https://github.com/defenseunicorns/pepr/issues/1823). * As an alpha feature, support the generation of CRDs within Pepr. See [pepr/#1062](https://github.com/defenseunicorns/pepr/issues/1062). ## Phase 3: TBD [Section titled “Phase 3: TBD”](#phase-3-tbd) * **Q3**: * **Support Governance with Pepr** * Improve initial support added in Q2. * **Transactional PeprStore Implementation**: * Begin integrating transactional functionality into PeprStore. The implementation will emphasize robust testing and clear documentation to support fast and reliable data operations in a transactional manner. ## Phase 4: TBD [Section titled “Phase 4: TBD”](#phase-4-tbd) * **Q4**: * **Prepare for Project Donation** * Consider Pepr project donation after resolving the open topics of Governance and performance at scale * **Experimentation with Deno2**: * Experiment with Deno2 through Dash Days and see if it can be used in the project. Look into the performance improvements and new features that Deno2 brings to the table. ## Deferred or Unprioritized Work [Section titled “Deferred or Unprioritized Work”](#deferred-or-unprioritized-work) This section contains work we’ve considered, but have not slotted in to the roadmap or have had to deprioritize. * **Pepr v1.0.0** * Determine if Pepr is stable enough to be at `v1.0.0` and release it! * **OTEL Preparation**: * Come up with a plan to implement Open Telemetry. Specifically distributed tracing, metrics, logs and events. Use this data to make debugging easier from a UDS Core perspective. There will be documentation work on how to use an OTEL collector with a Pepr Module. * **Deno2 Implementation**: * If determined to be advisable, move forward with migrating the project to Deno2 (starting with the kubernetes-fluent-client..?). This phase will focus on adapting the codebase, conducting extensive testing, and creating comprehensive documentation to ensure a seamless transition. * **Determine if a Transactional PeprStore makes sense**: * Sus out details involved with having a transactional Pepr Store. What are the implications of this? What are the benefits? What are the drawbacks? What are the use-cases? What are the technologies that can be used to implement this? *2024 Roadmap* ## Phase 1: Preparation - Testing and Docs [Section titled “Phase 1: Preparation - Testing and Docs”](#phase-1-preparation---testing-and-docs) * **Q1**: * **Establish Medium for Communication with Community**: * Establish communication channel for community members and contributors. Easy/discoverable “how to contribute” guide. * **Site/Documentation**: * Improve information architecture, nail killer use-cases, and make it obvious how to get started and find your way around. * **Automated Testing**: * Focus on stories that increase confidence in protection of features and functionality. Simplify hello-pepr and bring e2e test against external repo with examples. Make sure that contributions are well-tested. ## Phase 2: Community Building, Competitive Analysis, Instrumentation and Feature Development [Section titled “Phase 2: Community Building, Competitive Analysis, Instrumentation and Feature Development”](#phase-2-community-building-competitive-analysis-instrumentation-and-feature-development) * **Q2**: * **Community Engagement**: * Begin engaging with potential contributors and users through social media, Kubernetes/Cloud Native Computing Foundation (CNCF) meetups, and other channels. Monitor and participate in our Slack channel. * **Feature Development**: * Based on company feedback, continuously improve and add features. Add feature parity with other tools in the Kubernetes ecosystem where it makes sense. Chip away at the backlog. * **Documentation Improvements**: * Continue to improve documentation and add more examples like Doom, find scaling limitations * **Competitive Analysis**: * Understand the competitive landscape and how/where Pepr can/does differentiate itself. Have it in the docs. * **Instrumentation**: * Outfit Pepr with the necessary instrumentation to collect metrics and logs. Use this data to make informed about the watch direction. ## Phase 3: Informer Iterations, Watch Analysis, Metric Collection, Development/Stabilization/Improvement of New Features [Section titled “Phase 3: Informer Iterations, Watch Analysis, Metric Collection, Development/Stabilization/Improvement of New Features”](#phase-3-informer-iterations-watch-analysis-metric-collection-developmentstabilizationimprovement-of-new-features) * **Q3**: * **Informer Iterations**: * Tune informer based on feedback from UDS Core and delivery so events will be reconciled through eventual consistency. * [Evaluate](https://github.com/defenseunicorns/pepr/blob/main/adr/0015-kfc-watch-undici.md) other underlying technologies for informer. * **Feature Development**: * `.WithNameRegex()/.InNamespaceRegex()` for Kubernetes Controller development against resources that could match a variety of names or namespaces. * `.WithDeletionTimestamp()` for Kubernetes Controller Development against resources that are pending deletion * Create a sharded queue that enables the Module Author to define queueing strategies based on `kind`, `kind/namespace`, `kind/namespace/name`, `global`. * **Community Building**: * Grow the contributor base, establish a governance model, and encourage community-led initiatives. Look to drive conversation in our Slack Channel. * Based on community feedback, continuously improve and add features. Rigorously test and document and review code. * **Project Advocation**: * Publicly advocate for the project and encourage adoption. * **Stability**: * Ensure that the project is stable and reliable. Make sure that the project is well-tested and documented. * Identify new areas of project improvement and work on them. ## Phase 4: Feature Development, Stabilization, Code and Testing Quality Improvements [Section titled “Phase 4: Feature Development, Stabilization, Code and Testing Quality Improvements”](#phase-4-feature-development-stabilization-code-and-testing-quality-improvements) * **Q4**: * **Features**: * Improve DevEx overrides in Pepr section of package.json for customized builds of Modules * `.Finalize()` for Kubernetes Controller Development to controlling downstream resources through finalizers * Scaffolding to validate images from a registry through cosign/sigstore * Replace node-fetch with Undici in the KFC project for performance improvements * **Removal of Circular Dependencies**: Identify and remove circular dependencies in the codebase. * **Strong Typings**: * Identify where we can make Pepr/KFC stronger by adding typings. * **Work to reduce code complexity** * Monitor code complexity through eslint, work to drive down complexity * **Robust E2E Tests in KFC**: * Create a strong e2e suite in KFC, ensure tests are robust and cover all the features of KFC. * **Documentation**: * Ensure that the documentation is up-to-date and accurate. Add more examples and use-cases. * Onboarding and contribution guides should be clear and easy to follow. * **Load/Stress Testing**: * Load test Pepr/KFC to identify bottlenecks and areas of improvement. * Ensure that Pepr/KFC can handle a large number of resources and events over a sustained period of time (nightly). # Create a Pepr Dashboard > Create a Pepr Dashboard # Create a Pepr Dashboard [Section titled “Create a Pepr Dashboard”](#create-a-pepr-dashboard) ## Introduction [Section titled “Introduction”](#introduction) This tutorial will walk you through the process of creating a dashboard to display your Pepr metrics. This dashboard will present data such as the number of validation requests processed, the number of mutation requests that were allowed, the number of errors that were processed, the number of alerts that were processed, the status of the Pepr pods, and the scrape duration of the Pepr pods. This dashboard will be created using [Grafana](https://grafana.com/). The dashboard will display data from [Prometheus](https://prometheus.io/), which is a monitoring system that Pepr uses to collect metrics. This tutorial is not intended for production, but instead is intended to show how to quickly scrape Pepr metrics. The [Kube Prometheus Stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) provides a starting point for a more production suitable way of deploying Prometheus in prod. An example of what the dashboard will look like is shown below:  ***Note:*** *The dashboard shown above is an example of what the dashboard will look like. The dashboard will be populated with data from your Pepr instance.* ## Steps [Section titled “Steps”](#steps) ### Step 1: Get Cluster Running With Your Pepr Module Deployed [Section titled “Step 1: Get Cluster Running With Your Pepr Module Deployed”](#step-1-get-cluster-running-with-your-pepr-module-deployed) You can learn more about how to create a Pepr module and deploy it in the [Create a Pepr Module](../create-pepr-module) tutorial. The short version is: ```bash #Create your cluster k3d cluster create #Create your module npx pepr init #Change directory to your module that was created using `npx pepr init` npx pepr dev kubectl apply -f capabilities/hello-pepr.samples.yaml #Deploy your module to the cluster npx pepr deploy ``` ### Step 2: Create and Apply Our Pepr Dashboard to the Cluster [Section titled “Step 2: Create and Apply Our Pepr Dashboard to the Cluster”](#step-2-create-and-apply-our-pepr-dashboard-to-the-cluster) Create a new file called grafana-dashboard.yaml and add the following content: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: pepr-dashboard namespace: default data: pepr-dashboard.json: | { "__inputs": [ { "name": "DS_PROMETHEUS", "label": "Prometheus", "description": "", "type": "datasource", "pluginId": "prometheus", "pluginName": "Prometheus" } ], "__elements": {}, "__requires": [ { "type": "grafana", "id": "grafana", "name": "Grafana", "version": "9.1.6" }, { "type": "datasource", "id": "prometheus", "name": "Prometheus", "version": "1.0.0" }, { "type": "panel", "id": "stat", "name": "Stat", "version": "" }, { "type": "panel", "id": "timeseries", "name": "Time series", "version": "" } ], "annotations": { "list": [ { "builtIn": 1, "datasource": { "type": "grafana", "uid": "-- Grafana --" }, "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "target": { "limit": 100, "matchAny": false, "tags": [], "type": "dashboard" }, "type": "dashboard" } ] }, "editable": true, "fiscalYearStartMonth": 0, "graphTooltip": 0, "id": null, "links": [], "liveNow": false, "panels": [ { "collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }, "id": 18, "panels": [], "title": "Pepr Status", "type": "row" }, { "datasource": { "type": "prometheus", "uid": "prometheus" }, "description": "Pepr pod status by pod", "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "mappings": [ { "options": { "0": { "color": "red", "index": 1, "text": "Down" }, "1": { "color": "green", "index": 0, "text": "Up" } }, "type": "value" }, { "options": { "match": "empty", "result": { "color": "blue", "index": 2, "text": "?" } }, "type": "special" } ], "min": 0, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 12, "x": 0, "y": 1 }, "id": 14, "options": { "colorMode": "value", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": [ "last" ], "fields": "", "values": false }, "text": { "titleSize": 16, "valueSize": 70 }, "textMode": "auto" }, "pluginVersion": "9.1.6", "targets": [ { "datasource": { "type": "prometheus", "uid": "prometheus" }, "editorMode": "builder", "expr": "up{container=\"server\"}", "legendFormat": "{{instance}}", "range": true, "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "prometheus" }, "editorMode": "builder", "expr": "up{container=\"watcher\"}", "hide": false, "legendFormat": "{{instance}}", "range": true, "refId": "B" } ], "title": "Pepr Status", "type": "stat" }, { "datasource": { "type": "prometheus", "uid": "prometheus" }, "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 0, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 12, "x": 12, "y": 1 }, "id": 12, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "prometheus" }, "editorMode": "builder", "expr": "scrape_duration_seconds{container=\"server\"}", "legendFormat": "{{instance}}", "range": true, "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "prometheus" }, "editorMode": "builder", "expr": "scrape_duration_seconds{container=\"watcher\"}", "hide": false, "legendFormat": "{{instance}}", "range": true, "refId": "B" } ], "title": "Scrape Duration Seconds", "type": "timeseries" }, { "collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 9 }, "id": 6, "panels": [], "title": "Error, Alert, Validate and Mutate Counts", "type": "row" }, { "datasource": { "type": "prometheus", "uid": "prometheus" }, "fieldConfig": { "defaults": { "color": { "fixedColor": "dark-red", "mode": "fixed" }, "mappings": [], "min": 0, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 6, "x": 0, "y": 10 }, "id": 16, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": [ "last" ], "fields": "", "values": false }, "text": { "titleSize": 16, "valueSize": 70 }, "textMode": "auto" }, "pluginVersion": "9.1.6", "targets": [ { "datasource": { "type": "prometheus", "uid": "prometheus" }, "editorMode": "builder", "expr": "count by(instance) (rate(pepr_errors{container=\"server\"}[$__rate_interval]))", "legendFormat": "{{instance}}", "range": true, "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "prometheus" }, "editorMode": "builder", "expr": "count by(instance) (rate(pepr_errors{container=\"watcher\"}[$__rate_interval]))", "hide": false, "legendFormat": "{{instance}}", "range": true, "refId": "B" } ], "title": "Pepr: Error Count", "type": "stat" }, { "datasource": { "type": "prometheus", "uid": "prometheus" }, "description": "Count of Pepr Alerts by pod", "fieldConfig": { "defaults": { "color": { "fixedColor": "dark-yellow", "mode": "fixed" }, "mappings": [], "min": 0, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 6, "x": 6, "y": 10 }, "id": 10, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": [ "last" ], "fields": "", "values": false }, "text": { "titleSize": 16, "valueSize": 70 }, "textMode": "auto" }, "pluginVersion": "9.1.6", "targets": [ { "datasource": { "type": "prometheus", "uid": "prometheus" }, "editorMode": "builder", "expr": "pepr_alerts{container=\"server\"}", "legendFormat": "{{instance}}", "range": true, "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "prometheus" }, "editorMode": "builder", "expr": "pepr_alerts{container=\"watcher\"}", "hide": false, "legendFormat": "{{instance}}", "range": true, "refId": "B" } ], "title": "Pepr: Alert Count", "type": "stat" }, { "datasource": { "type": "prometheus", "uid": "prometheus" }, "description": "Count of Pepr Validate actions by pod", "fieldConfig": { "defaults": { "color": { "fixedColor": "dark-purple", "mode": "fixed" }, "mappings": [], "min": 0, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 6, "x": 12, "y": 10 }, "id": 4, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": [ "last" ], "fields": "", "values": false }, "text": { "titleSize": 16, "valueSize": 66 }, "textMode": "auto" }, "pluginVersion": "9.1.6", "targets": [ { "datasource": { "type": "prometheus", "uid": "prometheus" }, "editorMode": "builder", "exemplar": false, "expr": "pepr_validate_count{container=\"server\"}", "instant": false, "legendFormat": "{{instance}}", "range": true, "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "prometheus" }, "editorMode": "builder", "expr": "pepr_validate_sum{container=\"watcher\"}", "hide": false, "legendFormat": "{{instance}}", "range": true, "refId": "B" } ], "title": "Pepr: Validate Count", "type": "stat" }, { "datasource": { "type": "prometheus", "uid": "prometheus" }, "description": "Count of Pepr mutate actions applied by pod.", "fieldConfig": { "defaults": { "color": { "fixedColor": "dark-blue", "mode": "fixed" }, "mappings": [], "min": 0, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 6, "x": 18, "y": 10 }, "id": 2, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": [ "last" ], "fields": "", "values": false }, "text": { "titleSize": 16, "valueSize": 70 }, "textMode": "value_and_name" }, "pluginVersion": "9.1.6", "targets": [ { "datasource": { "type": "prometheus", "uid": "prometheus" }, "editorMode": "builder", "expr": "pepr_mutate_count{container=\"server\"}", "legendFormat": "{{instance}}", "range": true, "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "prometheus" }, "editorMode": "builder", "expr": "rate(pepr_mutate_count{container=\"watcher\"}[24h])", "hide": false, "legendFormat": "{{instance}}", "range": true, "refId": "B" } ], "title": "Pepr: Mutate Count", "type": "stat" } ], "schemaVersion": 37, "style": "dark", "tags": [], "templating": { "list": [] }, "time": { "from": "now-24h", "to": "now" }, "timepicker": {}, "timezone": "", "title": "Pepr Dashboard", "uid": "j7BjgMpIk", "version": 17, "weekStart": "" } ``` Now, apply the grafana-dashboard.yaml file to the cluster: ```bash kubectl apply -f grafana-dashboard.yaml ``` ### Step 3: Install Prometheus and Grafana using the kube-prometheus-stack Helm Chart [Section titled “Step 3: Install Prometheus and Grafana using the kube-prometheus-stack Helm Chart”](#step-3-install-prometheus-and-grafana-using-the-kube-prometheus-stack-helm-chart) First, create a values.yaml file to add our endpoints to Prometheus and allow us to see our dashboard. ```yaml prometheus: enabled: true additionalServiceMonitors: - name: admission selector: matchLabels: pepr.dev/controller: admission namespaceSelector: matchNames: - pepr-system endpoints: - targetPort: 3000 scheme: https tlsConfig: insecureSkipVerify: true - name: watcher selector: matchLabels: pepr.dev/controller: watcher namespaceSelector: matchNames: - pepr-system endpoints: - targetPort: 3000 scheme: https tlsConfig: insecureSkipVerify: true additionalClusterRoleBindings: - name: scrape-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: scrape-resources subjects: - kind: ServiceAccount name: prometheus-operator namespace: default grafana: enabled: true adminUser: admin adminPassword: secret defaultDashboardsTimezone: browser extraVolumeMounts: - mountPath: /var/lib/grafana/dashboards name: pepr-dashboard extraVolumes: - name: pepr-dashboard configMap: name: pepr-dashboard dashboardProviders: dashboardproviders.yaml: apiVersion: 1 providers: - name: 'default' isDefault: true orgId: 1 folder: '' type: file disableDeletion: false editable: true options: path: /var/lib/grafana/dashboards/default dashboardsConfigMaps: default: pepr-dashboard ``` Now, install the kube-prometheus-stack Helm Chart using the values.yaml file we created. ```bash helm install -f values.yaml monitoring prometheus-community/kube-prometheus-stack ``` ### Step 4: Check on Services [Section titled “Step 4: Check on Services”](#step-4-check-on-services) ```bash kubectl get svc ``` You should see something similar to the following services: ```bash $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.43.0.1 443/TCP 99m monitoring-kube-prometheus-prometheus ClusterIP 10.43.116.49 9090/TCP,8080/TCP 81s monitoring-kube-state-metrics ClusterIP 10.43.232.84 8080/TCP 81s monitoring-grafana ClusterIP 10.43.82.67 80/TCP 81s monitoring-kube-prometheus-operator ClusterIP 10.43.197.97 443/TCP 81s monitoring-kube-prometheus-alertmanager ClusterIP 10.43.40.24 9093/TCP,8080/TCP 81s monitoring-prometheus-node-exporter ClusterIP 10.43.152.179 9100/TCP 81s alertmanager-operated ClusterIP None 9093/TCP,9094/TCP,9094/UDP 81s prometheus-operated ClusterIP None 9090/TCP 81s ``` ### Step 5: Port Forward Prometheus and Grafana Services [Section titled “Step 5: Port Forward Prometheus and Grafana Services”](#step-5-port-forward-prometheus-and-grafana-services) ```bash kubectl port-forward service/prometheus-operated 9090 ``` ```bash kubectl port-forward service/monitoring-grafana 3000:80 ``` ### Step 6: View Prometheus Metrics Targets [Section titled “Step 6: View Prometheus Metrics Targets”](#step-6-view-prometheus-metrics-targets) You should be able to see the Pepr targets in the Prometheus UI by visiting the following URL: ```plaintext http://localhost:9090/targets ``` The targets should look something like this:   ### Step 7: Test the Prometheus Connection in Grafana [Section titled “Step 7: Test the Prometheus Connection in Grafana”](#step-7-test-the-prometheus-connection-in-grafana) You should be able to test the Prometheus connection in the Grafana UI by visiting the following URL: The login information for Grafana was set in the values.yaml file: username: admin password: secret By clicking on the Prometheus data source, you should be able to test the connection to Prometheus by clicking the “Test” button at the bottom of the screen. > \[!NOTE] The Prometheus server URL should be something like: You should now be able to select the Pepr Dashboard from the Grafana UI in the “Dashboards” section. > \[!NOTE] The dashboard may take a few minutes to populate with data. ## Summary [Section titled “Summary”](#summary) This tutorial demonstrated how to use Prometheus and Grafana to display metrics from your Pepr instance. If you have questions about Pepr metrics or dashboards, please reach out to us on [Slack](https://kubernetes.slack.com/archives/C06DGH40UCB) or [GitHub Issues](https://github.com/defenseunicorns/pepr/issues) # Create a Pepr Module > Create a Pepr Module # Create a Pepr Module [Section titled “Create a Pepr Module”](#create-a-pepr-module) ## Introduction [Section titled “Introduction”](#introduction) This tutorial will walk you through the process of creating a Pepr module. Each Pepr Module is it’s own Typescript project, produced by [`npx pepr init`](/user-guide/pepr-cli#pepr-init). Typically a module is maintained by a unique group or system. For example, a module for internal [Zarf](https://zarf.dev/) mutations would be different from a module for [Big Bang](https://p1.dso.mil/products/big-bang). An important idea with modules is that they are *wholly independent of one another*. This means that 2 different modules can be on completely different versions of Pepr and any other dependencies; their only interaction is through the standard K8s interfaces like any other webhook or controller. ## Prerequisites [Section titled “Prerequisites”](#prerequisites) ## Steps [Section titled “Steps”](#steps) ### Step 1: Create the module [Section titled “Step 1: Create the module”](#step-1-create-the-module) Use [`npx pepr init`](../../user-guide/pepr-cli#pepr-init) to generate a new module. ### Step 2: Quickly validate system setup [Section titled “Step 2: Quickly validate system setup”](#step-2-quickly-validate-system-setup) Every new module includes a sample Pepr Capability called `HelloPepr`. By default, this capability is deployed and monitoring the `pepr-demo` namespace. There is a sample yaml also included you can use to see Pepr in your cluster. Here’s the quick steps to do that after `npx pepr init`: ```bash # cd to the newly-created Pepr module folder cd my-module-name # If you don't already have a local K8s cluster, you can set one up with k3d npm run k3d-setup # Launch pepr dev mode # If using another local K8s distro instead of k3d, use `npx pepr dev --host host.docker.internal` npx pepr dev # From another terminal, apply the sample yaml kubectl apply -f capabilities/hello-pepr.samples.yaml # Verify the configmaps were transformed using kubectl, k9s or another tool ``` ### Step 3: Create your custom Pepr Capabilities [Section titled “Step 3: Create your custom Pepr Capabilities”](#step-3-create-your-custom-pepr-capabilities) Now that you have confirmed Pepr is working, you can now create new [capabilities](../../user-guide/capabilities). You’ll also want to disable the `HelloPepr` capability in your module (`pepr.ts`) before pushing to production. You can disable by commenting out or deleting the `HelloPepr` variable below: ```typescript new PeprModule(cfg, [ // Remove or comment the line below to disable the HelloPepr capability HelloPepr, // Your additional capabilities go here ]); ``` *Note: if you also delete the `capabilities/hello-pepr.ts` file, it will be added again on the next [`npx pepr update`](../../user-guide/pepr-cli#pepr-update) so you have the latest examples usages from the Pepr SDK. Therefore, it is sufficient to remove the entry from your `pepr.ts` module config.* ### Step 4: Build and deploy the Pepr Module [Section titled “Step 4: Build and deploy the Pepr Module”](#step-4-build-and-deploy-the-pepr-module) Most of the time, you’ll likely be iterating on a module with `npx pepr dev` for real-time feedback and validation Once you are ready to move beyond the local dev environment, Pepr provides deployment and build tools you can use. `npx pepr deploy` - you can use this command to build your module and deploy it into any K8s cluster your current `kubecontext` has access to. This setup is ideal for CI systems during testing, but is not recommended for production use. See [`npx pepr deploy`](../../user-guide/pepr-cli#pepr-deploy) for more info. ## Additional Information [Section titled “Additional Information”](#additional-information) By default, when you run `npx pepr init`, the module is not configured with any additional options. Currently, there are 3 options you can configure: * `deferStart` - if set to `true`, the module will not start automatically. You will need to call `start()` manually. This is useful if you want to do some additional setup before the module controller starts. You can also use this to change the default port that the controller listens on. * `beforeHook` - an optional callback that will be called before every request is processed. This is useful if you want to do some additional logging or validation before the request is processed. * `afterHook` - an optional callback that will be called after every request is processed. This is useful if you want to do some additional logging or validation after the request is processed. You can configure each of these by modifying the `pepr.ts` file in your module. Here’s an example of how you would configure each of these options: ```typescript const module = new PeprModule( cfg, [ // Your capabilities go here ], { deferStart: true, beforeHook: req => { // Any actions you want to perform before the request is processed, including modifying the request. }, afterHook: res => { // Any actions you want to perform after the request is processed, including modifying the response. }, } ); // Do any additional setup before starting the controller module.start(); ``` ## Summary [Section titled “Summary”](#summary) Checkout some examples of Pepr modules in the [excellent examples repo](https://github.com/defenseunicorns/pepr-excellent-examples). If you have questions after that, please reach out to us on [Slack](https://kubernetes.slack.com/archives/C06DGH40UCB) or [GitHub Issues](https://github.com/defenseunicorns/pepr/issues) # Building a Kubernetes Operator with Pepr > Building a Kubernetes Operator with Pepr # Building a Kubernetes Operator with Pepr [Section titled “Building a Kubernetes Operator with Pepr”](#building-a-kubernetes-operator-with-pepr) ## Introduction [Section titled “Introduction”](#introduction) This tutorial guides you through building a Kubernetes Operator using Pepr. You’ll create a WebApp Operator that manages custom WebApp resources in your Kubernetes cluster. If you get stuck at any point, you can reference the [complete example code in the Pepr Excellent Examples repository](https://github.com/defenseunicorns/pepr-excellent-examples/tree/main/pepr-operator). ## What You’ll Build [Section titled “What You’ll Build”](#what-youll-build) The WebApp Operator will: 1. Deploy a custom `WebApp` resource definition (CRD) 2. Watch for WebApp instances and reconcile them with the actual cluster state 3. For each WebApp instance, manage: * A `Deployment` with configurable replicas * A `Service` to expose the application * A `ConfigMap` containing configurable HTML with language and theme options All resources will include `ownerReferences`, triggering cascading deletion when a WebApp is removed. The operator will also automatically restore any managed resources that are deleted externally. [Back to top](#introduction) ## Prerequisites [Section titled “Prerequisites”](#prerequisites) * A Kubernetes cluster (local or remote) * Access to the `curl` command * Basic understanding of Kubernetes concepts * Familiarity with TypeScript * Node.js ≥ 20.0 [Back to top](#introduction) ## Steps [Section titled “Steps”](#steps) ### Step 1: Create a new Pepr Module [Section titled “Step 1: Create a new Pepr Module”](#step-1-create-a-new-pepr-module) First, create a new Pepr module for your operator: ```bash npx pepr init \ --name operator \ --uuid my-operator-uuid \ --description "Kubernetes Controller for WebApp Resources" \ --error-behavior reject \ --yes && cd operator # set working directory as the new pepr module ``` To track your progress in this tutorial, let’s treat it as a `git` repository: ```bash git init && git add --all && git commit -m "npx pepr init" ``` [Back to top](#introduction) ### Step 2: Create CRD [Section titled “Step 2: Create CRD”](#step-2-create-crd) The WebApp Custom Resource Definition (CRD) specifies the structure and validation for your custom resource. Create the necessary directory structure: ```bash mkdir -p capabilities/crd/generated capabilities/crd/source ``` Generate a class based on the WebApp CRD using [kubernetes-fluent-client](https://github.com/defenseunicorns/kubernetes-fluent-client). This allows us to react to the CRD fields in a type-safe manner. Create a CRD named `crd.yaml` for the WebApp that includes: * Theme selection (dark/light) * Language selection (en/es) * Configurable replica count * Status tracking ```bash curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/crd/source/crd.yaml \ -o capabilities/crd/source/crd.yaml ``` Examine the contents of `capabilities/crd/source/crd.yaml`. Note that the status should be listed under `subresources` to make it writable. We provide descriptions for each property to clarify their purpose. Enums are used to restrict the values that can be assigned to a property. Create an interface for the CRD spec with the following command: ```bash curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/crd/generated/webapp-v1alpha1.ts \ -o capabilities/crd/generated/webapp-v1alpha1.ts ``` Examine the contents of `capabilities/crd/generated/webapp-v1alpha1.ts`. Create a TypeScript file that contains the WebApp CRD named `webapp.crd.ts`. This will enable the controller to automatically create the CRD on startup. Use the command: ```bash curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/crd/source/webapp.crd.ts \ -o capabilities/crd/source/webapp.crd.ts ``` Take a moment to commit your changes for CRD creation: ```bash git add capabilities/crd/ && git commit -m "Create WebApp CRD" ``` The `webapp.crd.ts` file defines the structure of our CRD, including the validation rules and schema that Kubernetes will use when WebApp resources are created or modified. Create a file that will automatically register the CRD on startup named `capabilities/crd/register.ts`: ```bash curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/crd/register.ts \ -o capabilities/crd/register.ts ``` The `register.ts` file contains logic to ensure our CRD is created in the cluster when the operator starts up, avoiding the need for manual CRD installation. Create a file to validate that WebApp instances are in valid namespaces and have a maximum of `7` replicas. Create a `validator.ts` file with the command: ```bash curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/crd/validator.ts \ -o capabilities/crd/validator.ts ``` The `validator.ts` file implements validation logic for our WebApp resources, ensuring they meet our requirements before they’re accepted by the cluster. In this section, we’ve generated the CRD class for WebApp, created a function to automatically register the CRD, and added a validator to ensure WebApp instances are in valid namespaces and don’t exceed 7 replicas. Commit your changes for CRD registration & validation: ```bash git add capabilities/crd/ && git commit -m "Create CRD handling logic" ``` [Back to top](#introduction) ### Step 3: Create Helpers [Section titled “Step 3: Create Helpers”](#step-3-create-helpers) Now, let’s create helper functions that will generate the Kubernetes resources managed by our operator. These helpers will simplify the creation of Deployments, Services, and ConfigMaps for each WebApp instance. Create a `controller` folder in the `capabilities` folder and create a `generators.ts` file. This file will contain functions that generate Kubernetes objects for the operator to deploy (with the ownerReferences automatically included). Since these resources are owned by the WebApp resource, they will be deleted when the WebApp resource is deleted. ```bash mkdir -p capabilities/controller ``` Create `generators.ts` with the following command: ```bash curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/controller/generators.ts \ -o capabilities/controller/generators.ts ``` The `generators.ts` file contains functions to create all the necessary Kubernetes resources for our WebApp, including properly configured Deployments, Services, and ConfigMaps with appropriate labels and selectors. Our goal is to simplify WebApp deployment. Instead of requiring users to manage multiple Kubernetes objects, track versions, and handle revisions manually, they can focus solely on the `WebApp` instance. The controller reconciles WebApp instances against the actual cluster state to achieve the desired configuration. The controller deploys a `ConfigMap` based on the language and theme specified in the WebApp resource and sets the number of replicas according to the WebApp specification. Commit your changes for deployment, service, and configmap generation: ```bash git add capabilities/controller/ && git commit -m "Add generators for WebApp deployments, services, and configmaps" ``` [Back to top](#introduction) ### Step 4: Create Reconciler [Section titled “Step 4: Create Reconciler”](#step-4-create-reconciler) Now, create the function that reacts to changes in WebApp instances. This function will be called and placed into a queue, guaranteeing ordered and synchronous processing of events, even when the system is under heavy load. In the base of the `capabilities` folder, create a `reconciler.ts` file and add the following: ```bash curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/reconciler.ts \ -o capabilities/reconciler.ts ``` The `reconciler.ts` file contains the core logic of our operator, handling the creation, updating, and deletion of WebApp resources and ensuring the cluster state matches the desired state. Finally, create the `index.ts` file in the `capabilities` folder and add the following: ```bash curl -s https://raw.githubusercontent.com/defenseunicorns/pepr-excellent-examples/main/pepr-operator/capabilities/index.ts \ -o capabilities/index.ts ``` The `index.ts` file contains the WebAppController capability and the functions that are used to watch for changes to the WebApp resource and corresponding Kubernetes resources. * When a WebApp is created or updated, validate it, store the name of the instance and enqueue it for processing. * If an “owned” resource (ConfigMap, Service, or Deployment) is deleted, redeploy it. * Always redeploy the WebApp CRD if it was deleted as the controller depends on it In this section we created a `reconciler.ts` file that contains the function responsible for reconciling the state of WebApp instances with the cluster and updating their status. The `index.ts` file contains the WebAppController capability and functions that watch for changes to WebApp resources and their corresponding Kubernetes objects. The `Reconcile` action processes callbacks in a queue, guaranteeing ordered and synchronous processing of events. Commit your changes with: ```bash git add capabilities/ && git commit -m "Create reconciler for webapps" ``` [Back to top](#introduction) #### Add capability to Pepr Module [Section titled “Add capability to Pepr Module”](#add-capability-to-pepr-module) Ensure that the PeprModule in `pepr.ts` uses `WebAppController`. The implementation should look something like this: ```typescript new PeprModule(cfg, [WebAppController]); ``` Using `sed`, replace the contents of the file to use our new `WebAppController`: ```bash # Use the WebAppController Module sed -i '' -e '/new PeprModule(cfg, \[/,/\]);/c\ new PeprModule(cfg, [WebAppController]);' ./pepr.ts # Update Imports if [[ "$OSTYPE" == "darwin"* ]]; then sed -i '' 's|import { HelloPepr } from "./capabilities/hello-pepr";|import { WebAppController } from "./capabilities";|' ./pepr.ts else sed -i 's|import { HelloPepr } from "./capabilities/hello-pepr";|import { WebAppController } from "./capabilities";|' ./pepr.ts fi ``` Commit your changes now that the WebAppController is part of the Pepr Module: ```bash git add pepr.ts && git commit -m "Register WebAppController with pepr module" ``` ### Step 5: Build and Deploy Your Operator [Section titled “Step 5: Build and Deploy Your Operator”](#step-5-build-and-deploy-your-operator) #### Preparing Your Environment [Section titled “Preparing Your Environment”](#preparing-your-environment) Create an ephemeral cluster with `k3d`. ```bash k3d cluster delete pepr-dev && k3d cluster create pepr-dev --k3s-arg '--debug@server:0' --wait && kubectl rollout status deployment -n kube-system ``` **What is an ephemeral cluster?**\ An ephemeral cluster is a temporary Kubernetes cluster that exists only for testing purposes. Tools like Kind (Kubernetes in Docker) and k3d let you quickly create and destroy clusters without affecting your production environments. #### Update and Prepare Pepr [Section titled “Update and Prepare Pepr”](#update-and-prepare-pepr) Make sure Pepr is updated to the latest version: ```bash npx pepr update --skip-template-update ``` > \[!CAUTION] Be cautious when updating Pepr in an existing project as it could potentially override custom configurations. The `--skip-template-update` flag helps prevent this. #### Building the Operator [Section titled “Building the Operator”](#building-the-operator) Build the Pepr module by running: ```bash npx pepr build ``` Commit your changes after the build completes: ```bash git add capabilities/ package*.json && git commit -m "Build pepr module" ``` **The build process explained** The `pepr build` command performs three critical steps: 1. **Compile TypeScript**: Converts your TypeScript code to JavaScript using the settings in tsconfig.json 2. **Bundle the Operator**: Packages everything into a deployable format using esbuild 3. **Generate Kubernetes Manifests**: Creates all necessary YAML files in the `dist` directory, including: * Custom Resource Definitions (CRDs) * The controller deployment * ServiceAccounts and RBAC permissions * Any other resources needed for your operator This process creates a self-contained deployment unit that includes everything needed to run your operator in a Kubernetes cluster. ```text ┌─────────────────────┐ ┌──────────────────┐ ┌───────────────────┐ │ │ │ │ │ │ │ Your Pepr Code │────────▶ │ pepr build │────────▶ │ dist/ │ │ (TypeScript) │ │ (Build Process) │ │(Deployment Files) │ │ │ │ │ │ │ └─────────────────────┘ └──────────────────┘ └───────────────────┘ │ │ ▼ ┌───────────────────┐ │ │ │ Kubernetes │ │ Cluster │ │ │ └───────────────────┘ ``` #### Deploy to Kubernetes [Section titled “Deploy to Kubernetes”](#deploy-to-kubernetes) To deploy your operator to a Kubernetes cluster: ```bash kubectl apply -f dist/pepr-module-my-operator-uuid.yaml; # Wait for pods to have a status so we can check for readiness sleep 10; kubectl wait --for=condition=Ready pods -l app -n pepr-system --timeout=120s ``` **What’s happening during deployment?** 1. The first command applies all the Kubernetes resources defined in the YAML file, including: * The WebApp CRD (Custom Resource Definition) * A Deployment that runs your operator code * The necessary RBAC permissions for your operator to function 2. The second command waits for the operator pod to be ready before proceeding. This ensures your operator is running before you attempt to create WebApp resources. #### Troubleshooting Deployment Issues [Section titled “Troubleshooting Deployment Issues”](#troubleshooting-deployment-issues) **Troubleshooting operator startup problems** If your operator doesn’t start properly, check these common issues: 1. **Check pod logs**: ```bash kubectl logs -n pepr-system -l app --tail=100 ``` 2. **Verify permissions**: ```bash kubectl describe deployment -n pepr-system ``` Look for permission-related errors in the events section. Verify the deployment was successful by checking if the CRD has been properly registered: ```bash # Wait for crd to be registered sleep 10; kubectl get crd | grep webapp ``` You should see `webapps.pepr.io` in the output, which confirms your Custom Resource Definition was created successfully. #### Understanding the WebApp Resource [Section titled “Understanding the WebApp Resource”](#understanding-the-webapp-resource) You can use `kubectl explain` to see the structure of your custom resource. It may take a moment for the cluster to recognize this resource before the following command will work: ```bash # This command takes a lot of time to succeed the first time after CRD creation until kubectl explain wa.spec >/dev/null 2>&1; do sleep 2; done kubectl explain wa.spec ``` Expected Output: ```text GROUP: pepr.io KIND: WebApp VERSION: v1alpha1 FIELD: spec