What is a DaemonSet in Kubernetes? Best practices to manage them for peak efficiency

What is a DaemonSet in Kubernetes?

A DaemonSet in Kubernetes is a workload controller that ensures a specific pod is running onevery node (or specific nodes) in a cluster. As nodes are added or removed, the DaemonSet automatically ensures the Daemon Pods are created or deleted accordingly.

Think of it as the cluster-wide agent deployment strategy. Instead of manually creating individual pods per node, a DaemonSet automates this in a scalable, declarative way.

Use cases for DaemonSets in real-world Kubernetes environments

Here are some use cases to show how a DaemonSet powers critical Kubernetes infrastructure:

These workloads don't need replicas across the cluster. They need a presence on each node. DaemonSets make that possible.

How does a Kubernetes DaemonSet work?

Here's what happens when you deploy a DaemonSet:

• A pod is scheduled: One pod is scheduled per eligible node.
• A node joins: A new node triggers the automatic creation of a Daemon Pod on that node.
• A node leaves: When a node is removed, its Daemon Pod is automatically cleaned up.

DaemonSet update strategies: RollingUpdate vs. OnDelete

Kubernetes offers two updateStrategy options for DaemonSets, each with its own implications:

1. RollingUpdate

This is the default, recommended strategy. It updates DaemonSet Pods in a rolling fashion, replacing old pods with new ones gradually. This minimizes downtime and ensures that the workload remains available during the update.

Key parameters for RollingUpdate include:

• maxUnavailable: Specifies the maximum number of pods that can be unavailable during the update (e.g., a value of 1 means only one pod can be down at a time)
• maxSurge: Specifies the maximum number of extra pods that can be created during the update (e.g., a value of 1 means one additional pod can be created temporarily to maintain availability during the rollout)

2. OnDelete

With this strategy, pods are only updated when you manually delete the old pods. Kubernetes will then create new pods based on the updated DaemonSet specification. This strategy is less common as it requires manual intervention and can lead to downtime if not managed carefully.

Choosing the right strategy

• Use RollingUpdate for most scenarios where you want minimal disruptions.
• Consider OnDelete only when you have specific requirements that necessitate manual control over the update process and you are prepared to handle the potential downtime.

Basic example of a DaemonSet manifest

apiVersion: apps/v1 
kind: DaemonSet 
metadata: 
  name: node-monito 
spec: 
  selector: 
    matchLabels: 
      app: node-monitor 
  template: 
    metadata: 
      labels: 
        app: node-monitor 
    spec: 
    containers: 
     - name: monitor 
        image: your-monitoring-image    

Key features of a DaemonSet Kubernetes controller

• Cluster-wide distribution: Ensures one pod per node
• Selective scheduling: Uses nodeSelectors or affinity rules to target specific nodes
• Graceful updates: Supports rolling updates and surge control
• Integration with taints and tolerations: Enables running DaemonSets on tainted nodes (e.g., primary-only workloads)

Deployment vs. DaemonSet vs. StatefulSet: What are the differences?

To avoid confusion between Deployments, DaemonSets, and StatefulSets, here’s a side-by-side comparison:

When to use what?

• Use a Deployment to scale stateless applications horizontally across your cluster.
• Use a DaemonSet when you need to run a copy of a pod on all (or select) nodes, such as for log collection, monitoring, or networking.
• Use a StatefulSet when running stateful services that need stable network identities, persistent storage, or ordered deployment, like databases or clustered apps.

Best practices for managing DaemonSets

• Use tolerations and node affinity to target specific nodes (e.g., GPU nodes or tainted nodes).
• Monitor pod health using liveness and readiness probes.
• Avoid scheduling DaemonSets on critical control plane nodes, unless necessary.
• Limit resource consumption to prevent them from starving app workloads.
• Audit DaemonSets regularly to ensure they're not unintentionally consuming resources or overlapping with other system agents.

Security best practices for DaemonSets

DaemonSets, due to their cluster-wide nature, require careful security considerations:

• Pod Security Standards (PSSs): Pod Security Policies are deprecated as of Kubernetes version 1.21. Kubernetes now uses PSSs to define different isolation levels for pods. Consider applying appropriate PSS profiles (Privileged, Baseline, or Restricted) to your DaemonSet to limit its capabilities. Aim for the Restricted profile if possible, or Baseline as a starting point.
• The principle of least privilege: Grant the DaemonSet only the necessary permissions. Avoid using privileged containers unless absolutely required. If privileged access is unavoidable, carefully audit the DaemonSet's actions.
  • For example, instead of granting the DaemonSet cluster-admin privileges, only grant it the necessary permissions to read logs from the nodes.
  • You can achieve this by creating a Role and RoleBinding that grants the DaemonSet's service account the get, list, and watch permissions on pods and log resources in the desired namespaces.
• Service accounts: Use dedicated service accounts for your DaemonSets and grant them only the required RBAC permissions. Avoid using the default service account.
• Network policies: Implement network policies to restrict the DaemonSet's network access to only the necessary resources. This can prevent lateral movement in the event of compromise.
• Image security: Regularly scan the container images used in your DaemonSet for vulnerabilities. Use a trusted image registry and follow secure image building practices.

Monitoring DaemonSets with Site24x7

Site24x7 provides comprehensive monitoring capabilities for your Kubernetes DaemonSets, ensuring their optimal health and performance across your cluster.

• DaemonSet health monitoring: Monitor the availability and status of DaemonSet Pods on each node. Site24x7 can alert you to pod failures, restarts, or scheduling issues.
• Resource monitoring: Track the CPU, memory, disk, and network resource consumption of DaemonSet Pods on each node. Identify resource bottlenecks and prevent resource exhaustion.
• Tracking changes to DaemonSets over time: Track DaemonSet configuration changes in real time with the Kubernetes Change Tracker. Quickly identify and undo problematic changes.
• Dependency analysis: Understand how various Kubernetes resources are interconnected with the Dependency Tree view, making it easier to pinpoint the root cause of issues and optimize resource usage.
• Log monitoring: Aggregate and analyze logs from DaemonSet Pods to troubleshoot issues and identify potential problems.
• Alerting: Configure custom alerts based on various metrics, such as pod availability, resource utilization, or log patterns. For example, set up alerts for:
  • When the number of available pods in a DaemonSet falls below the number of nodes.
  • When the CPU or memory utilization of a DaemonSet Pod exceeds a certain threshold.
  • When specific error messages appear in the DaemonSet's logs.
• Integrations with incident management solutions: Integrate Site24x7 with your incident management system to automatically create incidents when DaemonSet issues are detected.

By leveraging Site24x7's monitoring capabilities, you can proactively identify and resolve issues with your DaemonSets, ensuring the optimal stability, security, and performance of your Kubernetes cluster.

Final thoughts: Embracing DaemonSets in your Kubernetes strategy

If you're managing Kubernetes clusters at scale, understanding what a DaemonSet is in Kubernetes is essential for building resilient infrastructure. From observability to storage, DaemonSets bridge the gap between node-level operations and cluster-wide automation.

Don't treat them as an afterthought—they are the backbone of Kubernetes' operational layer. Streamline monitoring across all nodes with DaemonSets—one pod per node, zero manual intervention.