close

Articles Home » Containers » Kubernetes Custom Resource Definitions (CRDs) Implementation

Kubernetes Custom Resource Definitions (CRDs) Implementation

Kubernetes has taken the tech world by storm, thanks to its robust platform for managing containerized applications in a clustered environment. It’s a hit among developers and system administrators for its ability to ensure that cloud-native applications run smoothly and reliably. A key player in this orchestration magic is the Custom Resource Definition (CRD), a potent feature that allows users to create their own customized extensions of the Kubernetes API.

This guide aims to shed light on how to implement CRDs in a Kubernetes setup. And also to empower you with the knowledge and skills to design, deploy, and manage your Custom Resource Definitions like a pro. We’re targeting individuals with a decent grasp on Kubernetes basics, who want to explore the custom resources to tailor Kubernetes to their needs.

Before we hit the road, let’s make sure we’ve got our gear in check. You’ll need:

  • A working knowledge of Kubernetes basics, including Pods, Deployments, and Services.
  • Familiarity with YAML syntax and basic scripting.
  • A Kubernetes cluster where you can flex your CRD muscles – you can set one up on your machine using Minikube or use a cloud-based solution like Google Kubernetes Engine (GKE).
  • A comfy seat, a cup of coffee (or tea, if that’s your jam), and an explorer’s spirit.

Understanding Custom Resource Definitions (CRDs)

Definition and Use Cases of CRDs

Custom Resource Definitions (CRDs) are an exciting extension of the Kubernetes API that empower users to create their own bespoke Kubernetes objects. Imagine you’re crafting a complex application that has unique operational needs, something the standard Kubernetes objects like Pods or Deployments can’t quite handle. This is where CRDs come into play, offering a customizable framework to define your objects, mirroring the behavior of native Kubernetes objects.

Some compelling use cases of CRDs include:

  • Configuring Custom Software: CRDs allow for the configuration of custom software components running within your Kubernetes cluster.
  • Managing Operational Processes: CRDs can encapsulate operational logic, helping manage processes like backups, restores, or updates in a Kubernetes-native way.
  • Abstracting Infrastructure Setups: Through CRDs, users can abstract away complex infrastructure setups, providing a simplified interface to manage underlying resources.

Comparison of CRDs with Built-in Kubernetes Objects

Now, let’s draw a comparison between CRDs and built-in Kubernetes objects to better understand their distinctions:

  • Bespoke Versus Standard: Built-in objects like Pods, Services, and Deployments are standard across all Kubernetes installations. They cater to common orchestration tasks and are supported out-of-the-box. On the flip side, CRDs are tailor-made, designed by users to serve specific needs within their Kubernetes environments.
  • API Extensibility: CRDs extend the Kubernetes API, allowing for the creation of new types of objects with custom schemas. Built-in objects, however, are predefined within the Kubernetes API and adhere to fixed schemas.
  • Maintenance: Built-in objects are maintained by the Kubernetes community, ensuring stability and support across versions. Conversely, the maintenance of CRDs rests on the shoulders of the users or teams that create them. This includes ensuring compatibility with different Kubernetes versions and handling upgrades or deprecations.
  • Tooling: Kubernetes’ built-in tooling like kubectl and dashboard work seamlessly with built-in objects. For CRDs, you might need to develop or adapt tooling to manage and visualize your custom resources effectively.

Setting Up The Environment

To begin we need a conducive environment. In this section, we’ll set up our toolkit, spin up a Kubernetes cluster, and ensure everything is in tip-top shape.

Tools and Resources Required

  • Kubernetes Cluster: You can either set one up locally using Minikube or Kind, or opt for a cloud-based solution like Google Kubernetes Engine (GKE), Azure Kubernetes Service (AKS), or Amazon Elastic Kubernetes Service (EKS).
  • Kubectl: This is your trusty command-line tool for interacting with your Kubernetes cluster.
  • Code Editor: Any text editor or Integrated Development Environment (IDE) you’re comfortable with. Visual Studio Code, or IntelliJ IDEA are solid choices.
  • Version Control System: A tool like Git to track changes to your configuration files and code.
  • Docker: As Kubernetes is a container orchestrator, you’ll need Docker to create and manage your container images.

Setting up a Kubernetes Cluster

  1. Local Setup using Minikube:
    • Install Minikube.
    • Start a Minikube instance with the command minikube start.
    • Once the instance is up, you can interact with it using kubectl.
  2. Cloud-based Setup (e.g., using GKE):
    • Create an account or log in to the Google Cloud Platform.
    • Navigate to the Kubernetes Engine and create a new cluster.
    • Connect to your cluster using kubectl by following the provided instructions.

Verifying the Setup

Now that our Kubernetes cluster is up and humming, let’s ensure everything’s operational:

Check Cluster Status:

kubectl cluster-infoCode language: Bash (bash)

Verify Nodes:

kubectl get nodesCode language: Bash (bash)

Check System Pods:

kubectl get pods --namespace kube-systemCode language: Bash (bash)

If all the nodes are in a Ready state and system pods are running fine, your Kubernetes environment is set, you’re ready for the next step.

Creating Your First Custom Resource Definition

The magic of Custom Resource Definitions (CRDs) unfurls as you design your own Kubernetes objects. Let’s roll up our sleeves and create our first CRD, shall we?

Designing a CRD Schema

A CRD schema is the blueprint of your custom resource. It defines the structure, types, and validation of data. For our example, let’s create a CRD for a fictional “Planet” resource, where each planet has a name, a type (gas or rocky), and a number of moons.

  1. Identify the Properties:
    • Name: string
    • Type: enum (gas, rocky)
    • MoonCount: integer
  2. Set Validation Rules:
    • Name: 1-40 characters, required
    • Type: one of the predefined values, required
    • MoonCount: non-negative integer, optional

Writing a CRD Manifest

Create a file named planet-crd.yaml and pop in the following content:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: planets.astronomy.example.com
spec:
  group: astronomy.example.com
  versions:
    - name: v1
      served: true
      storage: true
  scope: Namespaced
  names:
    plural: planets
    singular: planet
    kind: Planet
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            name:
              type: string
              minLength: 1
              maxLength: 40
            type:
              type: string
              enum:
                - gas
                - rocky
            moonCount:
              type: integer
              minimum: 0Code language: YAML (yaml)

Applying the CRD Manifest to the Kubernetes Cluster:

Time to bring our CRD to life in the Kubernetes cluster:

kubectl apply -f planet-crd.yamlCode language: Bash (bash)

Verifying the CRD Creation:

Now let’s ensure our new CRD is alive and kicking:

List CRDs:

kubectl get crdsCode language: Bash (bash)

Describe CRD:

kubectl describe crd planets.astronomy.example.comCode language: Bash (bash)

Upon successful verification, you’ll see the planets.astronomy.example.com CRD listed, and the describe command will provide detailed information about the CRD schema and other configurations.

Custom Resource (CR) Operations

With our CRD for planets set up, we can now perform typical CRUD (Create, Read, Update, Delete) operations on custom resource (CR) instances. Additionally, we’ll explore how to watch for changes to our custom resources using the Kubernetes API.

Creating Instances of the Custom Resource:

Creating a Planet Resource: Create a file named planet-earth.yaml with the following content to represent planet Earth:

apiVersion: astronomy.example.com/v1
kind: Planet
metadata:
  name: earth
spec:
  name: Earth
  type: rocky
  moonCount: 1Code language: YAML (yaml)

Apply this manifest to create the Earth resource:

kubectl apply -f planet-earth.yamlCode language: Bash (bash)

Reading, Updating, and Deleting Custom Resource Instances:

Reading a Custom Resource:

kubectl get planet earthCode language: Bash (bash)

Updating a Custom Resource: Update the moonCount field in planet-earth.yaml to 2 (just for fun), then apply the change:

kubectl apply -f planet-earth.yamlCode language: Bash (bash)

Verify the update:

kubectl get planet earth -o yamlCode language: Bash (bash)

Deleting a Custom Resource:

kubectl delete planet earthCode language: Bash (bash)

Watching Changes to Custom Resources using Kubernetes API

Kubernetes provides a watch API to observe changes to resources in real-time. This is quite handy to track updates to our custom resources.

Watching CRs from the Command Line:

kubectl get planets --watchCode language: Bash (bash)

Now, in a separate terminal, create, update or delete a planet resource and observe the changes in the watch terminal.

Watching CRs Programmatically: For a more advanced and programmatic approach, you can use client libraries like client-go to watch for changes to your custom resources within your application code. Here’s a simplified example in Go:

import (
    "context"
    "fmt"
    "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
)

func watchPlanets() {
    config, err := rest.InClusterConfig()
    if err != nil {
        panic(err)
    }
    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        panic(err)
    }

    planetWatcher, err := clientset.ExtensionsV1beta1().CustomResourceDefinitions().Watch(context.TODO(), v1.ListOptions{})
    if err != nil {
        panic(err)
    }

    for event := range planetWatcher.ResultChan() {
        fmt.Printf("Event: %v\n", event.Type)
    }
}

func main() {
    watchPlanets()
}Code language: Bash (bash)

In this section, we’ve explored the basic operations you can perform on custom resources and how to keep an eye on changes to these resources.

Implementing Custom Controllers

Custom controllers are the driving force behind the operational logic of your CRDs. They observe the state of your resources and take actions to steer the system towards the desired state. Let’s see how we can create a custom controller for our Planet CRD.

Understanding the Controller Pattern in Kubernetes

In Kubernetes, controllers are control loops that watch the state of your system, and then make or request changes where necessary. They operate on a reconciliation model, continuously ensuring that the current state matches the desired state defined by the CRD.

Designing and Implementing a Custom Controller for the CRD

Creating a custom controller involves several steps:

Set Up Your Development Environment:

  • Install Go.
  • Set up the Kubebuilder toolchain, which provides scaffolding for your controller project.

Create a New Controller Project:

kubebuilder init --domain example.com --repo github.com/example/astronomyCode language: Bash (bash)

Create a Controller for the Planet CRD:

kubebuilder create api --group astronomy --version v1 --kind PlanetCode language: Bash (bash)

Implement Your Controller Logic:

  • Open the generated controller file located at controllers/planet_controller.go.
  • Implement the Reconcile method to add the logic for creating, updating, and deleting planets.
func (r *PlanetReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // Your reconciliation logic here...
}Code language: Go (go)

Register Your Controller: In the main.go file, ensure your controller is set up with the manager.

func main() {
    mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
        Scheme:             scheme,
        MetricsBindAddress: metricsAddr,
        Port:               9443,
        LeaderElection:     enableLeaderElection,
        LeaderElectionID:   "f1c5cede.astronomy.example.com",
    })
    if err != nil {
        setupLog.Error(err, "unable to start manager")
        os.Exit(1)
    }

    if err = (&controllers.PlanetReconciler{
        Client: mgr.GetClient(),
        Log:    ctrl.Log.WithName("controllers").WithName("Planet"),
        Scheme: mgr.GetScheme(),
    }).SetupWithManager(mgr); err != nil {
        setupLog.Error(err, "unable to create controller", "controller", "Planet")
        os.Exit(1)
    }

    // ...
}Code language: Go (go)

Verifying the Custom Controller’s Functionality:

Build and Deploy Your Controller:

make docker-build docker-push IMG=<your-image-tag>
make deploy IMG=<your-image-tag>Code language: Bash (bash)

Test Your Controller:

  • Create, update, and delete Planet resources to observe how your controller reacts to these changes.
  • Check the controller logs for any errors or information messages that can help verify its functionality.
kubectl logs -l control-plane=controller-manager -n system --tail=-1Code language: Bash (bash)

Verify Desired Behavior: Based on the logic implemented in your controller, verify if the system’s state transitions correctly according to the changes in the Planet resources.

With a custom controller in play, your Planet CRD is now powered with the operational logic to manage the lifecycle of planet instances, making your Kubernetes setup more intelligent and self-reliant.

Validating Custom Resources

Validation is a crucial step to ensure that the custom resources conform to the desired specifications. By enforcing a well-defined structure and rules, we can catch errors early and maintain a healthy state within our Kubernetes cluster. Let’s explore how to implement validation for our Planet CRD and test it with different custom resource instances.

Implementing Validation Logic in CRD Manifest

We’ve already set up basic validation in our Planet CRD manifest earlier, enforcing certain field types and value ranges. However, let’s expand the validation to ensure the name field is not empty and moonCount is non-negative.

Update your planet-crd.yaml with the following validation block:

  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            name:
              type: string
              minLength: 1
              maxLength: 40
              pattern: '^[a-zA-Z0-9]+(?:[ -][a-zA-Z0-9]+)*$'
            type:
              type: string
              enum:
                - gas
                - rocky
            moonCount:
              type: integer
              minimum: 0Code language: YAML (yaml)

In the above block:

  • We added a pattern to the name field to ensure it follows a specific regex pattern.
  • The enum for the type field ensures that only specified values are accepted.
  • The minimum constraint on moonCount ensures a non-negative value.

Apply the updated CRD manifest:

kubectl apply -f planet-crd.yamlCode language: Bash (bash)

Testing Validation with Different Custom Resource Instances:

Now, let’s test our validation logic by creating different Planet resource instances.

Valid Resource: Create a file named planet-jupiter.yaml with the following content:

apiVersion: astronomy.example.com/v1
kind: Planet
metadata:
  name: jupiter
spec:
  name: Jupiter
  type: gas
  moonCount: 79Code language: YAML (yaml)

Apply the manifest:

kubectl apply -f planet-jupiter.yamlCode language: Bash (bash)

You should see a success message indicating the resource has been created.

Invalid Resource – Negative Moon Count: Create a file named planet-negative-moon.yaml:

apiVersion: astronomy.example.com/v1
kind: Planet
metadata:
  name: negative-moon
spec:
  name: Negative Moon
  type: rocky
  moonCount: -5Code language: YAML (yaml)

Try to apply the manifest:

kubectl apply -f planet-negative-moon.yamlCode language: CSS (css)

You should see an error message indicating the moonCount cannot be negative.

Invalid Resource – Invalid Type: Create a file named planet-invalid-type.yaml:

apiVersion: astronomy.example.com/v1
kind: Planet
metadata:
  name: invalid-type
spec:
  name: Invalid Type
  type: unknown
  moonCount: 0Code language: YAML (yaml)

Try to apply the manifest:

kubectl apply -f planet-invalid-type.yamlCode language: Bash (bash)

You should see an error message indicating the type field has an invalid value.

Versioning and Updating CRDs

As your project evolves, you may need to change the structure of your custom resources. Versioning and updating CRDs in a backward-compatible manner is essential to ensure a smooth transition and to prevent disruptions in your system.

Strategies for Versioning CRDs

  1. Semantic Versioning: Adopt a semantic versioning scheme to clearly communicate the changes in different versions of your CRD.
  2. API Versioning: Utilize the apiVersion field to indicate the version of your CRD, e.g., v1, v2, etc.

Updating a CRD Schema

Non-breaking Changes:

  • Add new optional fields or new accepted values for existing fields.
  • Apply the updated CRD manifest using kubectl apply.

bashCopy code

kubectl apply -f updated-planet-crd.yaml

Breaking Changes:

  • Create a new version of your CRD, say v2, without removing the old version.
  • Update your controllers to handle the new version.
  • Encourage users to transition to the new version over time.

Managing CRD Backward Compatibility

Conversion Webhooks: Implement conversion webhooks to translate between different versions of your CRD. This way, you can support multiple versions at the same time.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
# ...
spec:
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        service:
          name: webhook-service
          namespace: default
          path: "/convert"
      conversionReviewVersions: ["v1", "v2"]
# ...Code language: YAML (yaml)

Deprecation Notices: If a field or an entire CRD version is being deprecated, communicate this clearly through deprecation notices in the CRD manifest and your project documentation.

Support Multiple Versions: Keep support for older versions for a reasonable period, allowing users to transition to the newer version.

# ...
spec:
  versions:
    - name: v1
      served: true
      storage: false
    - name: v2
      served: true
      storage: true
# ...Code language: YAML (yaml)

By carefully managing the versioning and updating of your CRDs, you ensure a stable and reliable system, even as your custom resources evolve over time. This practice is vital for the longevity and maintainability of your Kubernetes setup.

Debugging and Monitoring CRD Implementations

Inevitably, you may encounter issues or unexpected behavior with your CRD implementations. Being adept at debugging and monitoring is crucial to diagnose and resolve such issues efficiently.

Common Issues and Troubleshooting Steps:

CRD Validation Errors: Check the error messages for details about the validation failure. Verify the CRD manifest for any syntax errors or incorrect field specifications.

Custom Controller Failures: Inspect the controller logs for any errors or warnings.

kubectl logs -l control-plane=controller-manager -n system --tail=-1Code language: Bash (bash)

Ensure the controller is running and has the necessary RBAC permissions.

Resource Synchronization Issues: Check the status and events of the custom resource for any synchronization or reconciliation errors.

kubectl describe <kind> <name>Code language: Bash (bash)

Custom Resource Creation Failures: Ensure that the CRD is correctly installed and the API version matches the one specified in the custom resource manifest.

Monitoring Custom Resources and Controllers using Kubernetes Tools:

Logging: Configure logging levels in your controllers to capture necessary information. Use centralized logging systems like ELK Stack (Elasticsearch, Logstash, Kibana) or Splunk to aggregate and analyze logs.

Metrics: Expose custom metrics from your controllers using libraries like Prometheus. Monitor these metrics using Grafana or other monitoring tools to get insights into the performance and behavior of your CRD implementations.

Events: Emit Kubernetes events from your controllers to signal significant occurrences.

eventBroadcaster := record.NewBroadcaster()
eventBroadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: kubeClientset.CoreV1().Events("")})
recorder := eventBroadcaster.NewRecorder(scheme.Scheme, corev1.EventSource{Component: controllerName})
recorder.Event(customResource, corev1.EventTypeWarning, "FailedSync", "Failed to sync the custom resource")Code language: Go (go)

Dashboards: Utilize Kubernetes dashboards to monitor the state of your custom resources and controllers. Customize dashboards to include information specific to your CRD implementations.

Alerting: Set up alerting rules based on the metrics and events to get notified about critical issues or abnormal behavior.

Audit Logs: Utilize Kubernetes audit logs to track changes to your custom resources and monitor access to your CRD API.

Best Practices

Adhering to best practices is crucial for the maintainability, security, and performance of your CRD implementations. Here are some key areas:

Naming Conventions:

  1. Consistent Naming: Employ a consistent naming convention for your CRDs, custom resources, and controllers. This includes using descriptive names and following the camelCase or kebab-case conventions as appropriate.
  2. Domain-Specific Naming: Utilize a domain-specific naming scheme to avoid conflicts with other resources. For example, using a unique API group name for your CRDs.
  3. Version Naming: Stick to conventional version naming practices like v1, v2, etc., for clarity and consistency.

Security Considerations:

  1. RBAC: Implement Role-Based Access Control (RBAC) to define who has access to what in your Kubernetes cluster. This is crucial for protecting your CRD data and operations.
  2. Namespace Isolation: Where possible, use namespaces to isolate your CRDs and custom resources, especially in multi-tenant environments.
  3. Validation: Implement thorough validation in your CRD manifests to prevent erroneous or malicious data entry.
  4. Audit Logging: Enable audit logging to track who did what and when, which is crucial for post-incident analysis and compliance.

Performance Optimization Tips:

  1. Indexing: Utilize indexing in your controllers to efficiently lookup and list resources, which is especially important as the number of custom resources grows.
  2. Rate Limiting: Implement rate limiting in your controllers to avoid overwhelming the Kubernetes API server, especially in error scenarios.
  3. Resource Efficiency: Be mindful of the resources (CPU, memory) your controllers consume and optimize your code for efficiency.
  4. Parallelization: Where possible, parallelize operations in your controllers to improve performance, but do so with caution to avoid race conditions.

Real-World CRD Implementation

In this section, let’s explore a real-world use case of CRD implementation by looking at a simplified version of managing database clusters within a Kubernetes environment.

Overview of a Real-World Use Case:

Imagine a scenario where you are tasked with managing multiple database clusters in a Kubernetes environment. Each database cluster has its unique configuration, including the database version, the number of replicas, storage size, and credentials. A Custom Resource Definition (CRD) can be a perfect solution to model and manage these database clusters.

Step-by-step Walkthrough of the CRD Implementation:

Designing the CRD Schema: Identify the properties required for each database cluster, such as dbVersion, replicaCount, storageSize, username, and password.

Creating the CRD Manifest: Define a CRD with a schema that matches the identified properties.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: dbclusters.database.example.com
spec:
  group: database.example.com
  versions:
    - name: v1
      served: true
      storage: true
  scope: Namespaced
  names:
    plural: dbclusters
    singular: dbcluster
    kind: DbCluster
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            dbVersion:
              type: string
            replicaCount:
              type: integer
            storageSize:
              type: string
            username:
              type: string
            password:
              type: stringCode language: Go (go)

Applying the CRD Manifest: Apply the CRD manifest to the Kubernetes cluster using kubectl apply.

kubectl apply -f dbcluster-crd.yamlCode language: Bash (bash)

Implementing the Custom Controller: Create a custom controller that watches for changes to DbCluster resources. Implement reconciliation logic to create, update, or delete database clusters based on the DbCluster resource state.

Deploying the Custom Controller: Build and deploy your custom controller to the Kubernetes cluster.

Creating DbCluster Resources: Create DbCluster resources for each database cluster you want to manage. The custom controller should react to these resources, managing the database clusters accordingly.

apiVersion: database.example.com/v1
kind: DbCluster
metadata:
  name: production-db
spec:
  dbVersion: "12.4"
  replicaCount: 3
  storageSize: "10Gi"
  username: admin
  password: secretCode language: YAML (yaml)
kubectl apply -f production-db.yamlCode language: Bash (bash)

Monitoring and Management: Monitor the state of your database clusters through Kubernetes tools and custom metrics exposed by your controller. Utilize the Kubernetes API to manage the lifecycle of your database clusters through DbCluster resources.

Through this real-world example, you can see how CRDs and custom controllers enable a declarative approach to managing complex systems within a Kubernetes environment. This pattern can be extended to manage a wide variety of resources and configurations.

The knowledge and practices discussed in this guide equip you with the foundation to explore and implement CRDs in your own projects. Whether you’re managing database clusters, network configurations, or any other complex systems, CRDs provide a powerful and flexible framework to extend the Kubernetes ecosystem according to your needs.

Related posts: