Site24x7 Kubernetes Operator: Automated monitoring for clusters and applications

The site24x7-operator simplifies the deployment and management of Site24x7 monitoring agents in your Kubernetes cluster. It automates the configuration and installation process, eliminating the need to manually manage multiple YAML files across different monitoring agents.

Why use the site24x7-operator?

Traditionally, monitoring Kubernetes environments isn't straightforward. Teams often have to:

• Juggle multiple YAML files to collect complete metrics.
• Deal with different installation YAMLs for platforms like OpenShift or GKE Autopilot.
• Manually edit production YAMLs to configure or update application performance monitoring (APM) agents.
• Update Docker images to add monitoring components.

All of this makes the onboarding process slow, complex, and require additional manual effort.

The site24x7-operator automates the entire deployment process:

• Automatic platform configuration: Specify your Kubernetes environment type, and the site24x7-operator applies the appropriate configuration.
• Hands-free installation: Automatically schedules and deploys monitoring resources without manual intervention.
• No Docker image modifications: Monitoring agents are dynamically injected into pods, keeping your application deployment setup clean.
• Centralized configuration: Manage all monitoring settings through a single custom resource file.

How it works

The site24x7-operator manages the complete monitoring life cycle for both infrastructure and APM resources within a Kubernetes cluster through a single custom resource.

Users deploy the site24x7-operator into the cluster and create a Site24x7K8s resource describing their desired monitoring configuration, such as namespaces to watch, agent images to be used, proxy settings, and other monitoring preferences.

The site24x7-operator continuously reconciles the desired state of Site24x7 monitoring resources against the actual cluster state, provisioning and updating monitoring resources based on the selected Site24x7K8s configuration.

When the configuration changes, the site24x7-operator detects the updated generation, compares the user-provided configuration with the existing state of Site24x7 resources, and applies the required changes across all managed resources. No manual Helm upgrades or kubectl patch operations are required.

site24x7-operator: Key components

1. The custom resource (Site24x7K8s)

The Site24x7K8s custom resource serves as the central configuration point for monitoring within your Kubernetes cluster. Instead of manually deploying and configuring multiple monitoring components, you define your monitoring requirements in a single custom resource and the site24x7-operator manages the resources automatically.

The custom resource contains two primary sections:

• infrastructure for Kubernetes cluster and node monitoring
• apm for application performance monitoring

The site24x7-operator continuously watches this resource and automatically creates, updates, or removes child resources whenever the configuration changes. This declarative approach ensures that your monitoring configuration remains version-controlled, reproducible, and consistent across environments.

2. Operator (control plane)

The site24x7-operator acts as the central controller running in the site24x7 namespace. It continuously watches the Site24x7K8s custom resource and ensures that the state of Site24x7 resources matches the desired monitoring configuration.

When the custom resource is created or modified, the site24x7-operator determines which resources need to be deployed, updated, or removed and applies those changes. It also tracks the health of managed resources and retries failed deployments automatically.

The site24x7-operator hosts the webhook server responsible for automatic APM instrumentation. During custom resource deletion, a finalizer ensures that all cluster-wide resources created by the site24x7-operator are cleaned up safely.

3. Infrastructure monitoring

Site24x7 Agent (DaemonSet)

The Site24x7 Agent runs as a DaemonSet, ensuring that one agent pod is deployed on every node in the cluster.

Each agent pod contains:

• The primary node agent, which collects host-level metrics such as CPU, memory, disk, and network utilization.
• An auto-profiler that discovers running applications and determines their eligibility for APM instrumentation.

The agent communicates directly with the Kubernetes API server to gather cluster-state information and forwards collected metrics to Site24x7 for visualization and analysis.

Since the agent collects cluster-wide and node-level information, it requires elevated permissions to access Kubernetes resources.

Kube-State-Metrics (Deployment)

Kube-State-Metrics is a standard Kubernetes component that generates metrics about the state of Kubernetes objects.

It provides information such as:

• Deployment replica counts
• Pod states and phases
• Resource requests and limits
• Persistent Volume Claim (PVC) status

The Site24x7 Agent consumes these metrics to provide comprehensive visibility into cluster health and resource utilization.

This component is deployed automatically when infrastructure monitoring is enabled.

Cluster Agent (Deployment)

The Cluster Agent is an optional component that becomes active automatically in larger clusters.

In large environments, having every node agent query the Kubernetes API server independently can increase API server load. The Cluster Agent centralizes these cluster-wide queries, retrieves the required information once, and distributes it to node agents.

This approach reduces Kubernetes API server overhead and improves monitoring scalability.

The Cluster Agent also exposes health check endpoints for readiness and liveness monitoring.

4. APM

CSI Driver (DaemonSet)

The CSI Driver runs as a DaemonSet and is responsible for delivering APM agent binaries to application pods.

The CSI Driver architecture includes:

• A provisioner container that downloads APM agent packages for supported languages, such as Java, .NET, Node.js, and Python.
• A CSI server container that facilitates the required APM agents in application containers through CSI VolumeMounts.
• A driver registration component that integrates with kubelet.

Agent binaries are downloaded once per node and shared across instrumented workloads, reducing storage and network overhead.

Mutating webhook configuration and webhook server

The mutating configuration enables automatic APM instrumentation.

Whenever a new pod is created in a monitored namespace, the webhook server intercepts the pod creation request and injects the required monitoring components before the pod is scheduled.

Depending on the workload type, the webhook may:

• Add CSI ephemeral volumes containing APM agent binaries.
• Inject required environment variables.
• Configure agent-specific runtime settings.
• Inject license information from Kubernetes secrets.

The webhook supports namespace-level filtering and honors opt-out configurations, allowing administrators to control which workloads are instrumented.

Data Exporter

The Data Exporter receives collected metrics and data from APM agents running inside application workloads and forwards the collected data to Site24x7.

It can operate in two deployment modes:

• DaemonSet mode, where a single exporter serves workloads running on the same node.
• Sidecar mode, where a container is deployed alongside each monitored application pod.

DaemonSet mode provides better resource efficiency, while sidecar mode offers stronger workload isolation.

Supported platforms

• Standard Kubernetes
• OpenShift (Red Hat)
• Google Kubernetes Engine (GKE)
• Other managed Kubernetes services (AWS EKS, Azure AKS)

Site24x7K8s configuration options

The site24x7-operator supports extensive configuration options for monitoring and deployment customization. For detailed information about all supported configuration fields, refer to this help documentation.

Installation steps

Prerequisites

• Kubernetes v1.25+
• Helm 3.x
• Docker 17.03+ or compatible containerd version
• Linux Kernel 3.18+
• Cluster permissions required:
  Installing the Site24x7 Kubernetes Operator requires the user or CI system performing the installation to have the following cluster-level and namespace-level permissions to create the Site24x7 Operator and its associated resources.

          Cluster-scoped:

• •  namespaces (""): get, create, patch, update
  • customresourcedefinitions (apiextensions.k8s.io): get, create, patch, update
  • clusterroles (rbac.authorization.k8s.io): get, create, patch, update
  • clusterrolebindings (rbac.authorization.k8s.io): get, create, patch, update
  • priorityclasses (scheduling.k8s.io): get, create, patch, update

Namespaced (namespace site24x7):

• • serviceaccounts (""): get, create, patch, update
  • services (""): get, create, patch, update
  • deployments (apps): get, create, patch, update

The site24x7-operator supports two deployment methods: YAML-based installation and Helm-based installation. Follow the steps for your preferred deployment method.

Step one is common for both installation methods. Complete step two based on your preferred installation method (YAML or Helm), then proceed with the common steps three through six to configure and verify monitoring.

Installation method 1: YAML-based installation

Step 1: Create the Site24x7 device key secret

kubectl create namespace site24x7 && 
kubectl create secret generic site24x7-agent --namespace site24x7 --from-literal=s247_device_key=< device key >

Step 2: Deploy site24x7-operator

Download the site24x7-operator YAML file from the Site24x7 console and apply it to your cluster.

kubectl apply -f site24x7_operator.yaml

This creates the following resources:

• A dedicated namespace (site24x7)
• The Custom Resource Definition (CRD) required by the site24x7-operator
• A ClusterRole and ClusterRoleBinding to grant necessary permissions
• The site24x7-operator deployment pod that manages monitoring resources within the cluster

Installation method 2: Helm-based installation

Step 2: Deploy the site24x7-operator

i. Add the Site24x7 Helm repository:

helm repo add site24x7 https://site24x7.github.io/helm-charts

ii. Update the repository:

helm repo update

iii. Install the site24x7-operator:

helm install site24x7-operator site24x7/site24x7operator

For OpenShift environments:

helm install site24x7-operator site24x7/site24x7operator --set openshift=true

Step 3: Download the site24x7k8s_resource.yaml file and configure it

Download the Site24x7K8s custom resource YAML file (site24x7k8s_resource.yaml) and update it with your monitoring preferences:

apiVersion: site24x7.com/v1alpha1
kind: Site24x7K8s
metadata:
  name: site24x7k8s-resource
  namespace: site24x7
spec:
  openShift: false
  installationType:
    agentType: "fso" # options: fso | apm_insight | infra | custom
  proxy:
    http_proxy: "<http proxy url>"
    https_proxy: "<https proxy url>"
  infrastructure:
    managementAction: true
    clusterName: "PROD_CLUSTER_EKS_1"
    site24x7Agent:
      leaseAPI:
        enabled: false
      nodeSelector: {}
      imagePullSecrets: []
  apm:
    webhook:
      monitorAllNamespaces: true
      excludeNamespaces:
        - site24x7
        - kube-system
        - kube-public
        - kube-node-lease
        - cert-manager
        - ingress-nginx
        - external-dns
        - argocd
        - flux-system
        - velero
    monitorAllPods: false
    includeAppLabels:
      payment_namespace:
        - app.kubernetes.io/name: payments-api
          app.kubernetes.io/instance: payments-prod
          app.kubernetes.io/component: api
          app.kubernetes.io/part-of: payments-platform

Step 4: Apply the custom resource

kubectl apply -f site24x7-config.yaml

This step triggers the site24x7-operator to start acting on your configuration. Once applied, the site24x7-operator processes the site24x7k8s-resource and provisions the required monitoring resources.

The site24x7-operator will automatically apply required monitoring and data collection resources.

Step 5: Verify installation

Check that the site24x7-operator and monitoring resources are running:

kubectl get pods -n site24x7

You should see the following resources (based on enabled monitoring types):

• site24x7-operator deployment pod
• site24x7-agent DaemonSet pods (infrastructure monitoring)
• site24x7-kube-state-metrics deployment pod (infrastructure monitoring)
• site24x7-cluster-agent deployment (based on node count)
• site24x7-apminsight-dataexporter-daemonset pods (APM monitoring)
• site24x7-csi-driver DaemonSet pods (APM monitoring)

Step 6: Verify monitoring in the Site24x7 console

1. Log in to your Site24x7 console.
2. To view the APM metrics:
   • Navigate to APM > K8s Application Discovery to view Kubernetes application discovery details.
   • Navigate to APM > Applications to view monitored application instances and performance insights.
3. To view the infrastructure metrics:
   • Navigate to K8s to view detailed information about your cluster and node metrics.

Updating the configuration

If your monitoring needs change, such as enabling APM later, switching to full-stack observability, or updating which pods should be monitored, you can easily modify the configuration without reinstalling anything.

To modify your monitoring configuration after installation:

1. Edit the custom resource YAML file (site24x7k8s_resource.yaml).
2. Make your desired changes (e.g., enable or disable monitoring types, update pod labels).
3. Reapply the configuration:

   kubectl apply -f site24x7k8s_resource.yaml

The site24x7-operator's reconciliation loop automatically detects changes in the site24x7k8s-resource and updates the monitoring resources accordingly.

How do I switch from manual deployment to the Kubernetes site24x7-operator?

If you're currently using manual YAML-based deployment:

1. Install the site24x7-operator as described above.
2. You can configure the custom resource with your existing monitoring preferences.
3. Remove the old manual monitoring resources.
4. The site24x7-operator will automatically manage monitoring going forward.

Uninstalling the site24x7-operator

To remove the site24x7-operator and all monitoring resources, execute the following commands to delete the applied manifests and clean up the resources created in your cluster.

kubectl delete -f site24x7k8s_resource.yaml --wait=false

Make sure to restart any pods mutated by the site24x7-mutating-webhook so that the site24x7-apminsight CSI volume is removed before deleting the site24x7-operator deployment.

Use this command to fetch all the pods that have been injected with the Site24x7 volume:

kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(any(.spec.volumes[]?; .csi.driver == "site24x7-csi-driver")) | "\(.metadata.name)\t\(.metadata.namespace)"'

After removing the custom resource, delete the site24x7-operator deployment and the resources created by the operator using the following command:

kubectl delete -f site24x7_operator.yaml

Related articles

• Kubernetes monitoring
• Application performance monitoring
• Full-Stack agent