How to monitor Kafka in Huawei Cloud
Site24x7 offers monitoring support for Huawei Cloud Kafka. By monitoring the performance and health, you can manage distributed message streaming services based on Apache Kafka. Monitoring Kafka metrics ensures healthy producer and consumer throughput and prevents message accumulation that could cause processing delays.
Use cases
Prevent data loss: When message backlog keeps increasing over time, Site24x7 alerts teams early so they can fix slow consumers or broker issues before messages expire.
Avoid storage issues: When broker disk usage crosses safe limits, Site24x7 sends a warning, giving teams time to adjust retention settings or add storage before new messages are rejected.
Maintain pipeline performance: Alerts on rising processing times helps the team to rebalance workloads or scale brokers, keeping message flow smooth and preventing delays for downstream consumers.
Setup and configuration
Kafka resources are auto-discovered and monitored during the Huawei Cloud integration. To enable monitoring, follow the steps below:
- Navigate to Cloud > Huawei > Add Huawei Monitor. Learn how to add a Huawei Cloud monitor.
- While adding or editing a Huawei Cloud monitor, select Kafka from the Service/Resource Types drop-down menu and click Save.
- Navigate to Cloud > Huawei, select the created Huawei monitor, and then click Kafka.
Supported metrics
Instance
Metric name |
Description |
Units |
| Partitions | The total number of topic partitions across the Kafka instance. | Count |
| Topics | The total number of topics created in the Kafka instance. | Count |
| Accumulated Messages | The total number of messages waiting to be consumed across all consumer groups. | Count |
| Message Production | The rate at which messages are being produced to the Kafka instance across all topics. | Bytes/second |
| Message Consumption | The rate at which messages are being consumed from the Kafka instance across all topics. | Bytes/second |
| Partitions Usage | The percentage of the maximum allowed partitions currently in use. | Percentage |
Broker
Metric name |
Description |
Units |
| Message Creation Rate | The rate at which new messages are being written to the broker. | Bytes/second |
| Message Creation | The byte rate at which new message data is being ingested by the broker. | Bytes/second |
| Message Retrieval | The byte rate at which message data is being read from the broker by consumers. | Bytes/second |
| Disk Capacity Usage | The percentage of the total broker disk capacity currently consumed by message data. | Percentage |
| Memory Usage | The percentage of memory currently in use by the Kafka broker process. | Percentage |
| CPU Usage | The percentage of CPU resources currently consumed by the Kafka broker. | Percentage |
| Connections | The number of active client connections to the Kafka broker. | Count |
| JVM Heap Memory Usage | The percentage of Java Virtual Machine (JVM) heap memory currently in use by the Kafka broker. | Percentage |
| Message Size | The total size of messages currently stored on the broker disk. | Bytes |
Latency
Metric name |
Description |
Units |
| Average Message Creation Processing Duration | The average time taken by the broker to process and acknowledge a produce request from producers. | Milliseconds |
| Average Message Retrieval Processing Duration | The average time taken by the broker to process and fulfill a fetch request from consumers. | Milliseconds |
Threshold configuration
You can configure thresholds and alerts for all Kafka metrics to detect performance degradation proactively or connection issues.
- Go to Admin > Configuration Profiles > Threshold and Availability.
- Create or edit your Threshold Profile for Kafka.
- Assign the profile to the respective monitors to trigger alerts.
IT Automation
Use Site24x7's IT Automation to resolve common issues with Kafka performance:
- Go to Admin > IT Automation Templates. Then, click Add Automation Templates.
- Create an automation rule by selecting the automation Type (e.g., Server reboot, clear queue).
- Map the created rules to the Kafka, for automatic execution during alerts.
Configuration rules
Use Configuration Rules to simplify bulk setup across Kafka instances. Automatically assign Threshold Profiles, Notification Profiles, Tags, and Monitor Groups when new monitors are discovered.