Go to All Forums

Monitor the performance of your Kafka servers

Configure kafka plugin to monitor the performance metrics of your kafka server. Use these key indicators to ensure continuous functioning of your kafka Server. Take informed troubleshooting decisions by keeping track of critical metrics including:

  • Number of active connections 
  • Network Input/Output rate
  • Incoming byte rate
  • Outgoing byte rate
  • Average request latency
  • Request rate
  • Response rate
  • Average time for Input/Output thread spent waiting for a socket
Learn how to install Kafka plugin from Site24x7 Kafka GitHub repository  

Related resources:

Like (6) Reply
Replies (1)


We are happy to introduce Apache Kafka plugin integrations to monitor your entire Kafka stack. You will now be able to monitor critical Kafka producer, consumer, and broker metrics in real time and stay ahead of issues and bottlenecks.

Refer to the below GitHub link for the Kafka monitoring plugin details and setup steps:


You can monitor the following components with our Kafka plugins: 

1. Kafka consumer

Install the Kafka Consumer plugin and monitor key consumer metrics.


      Supported metrics: 

  • Records Lag Max(All Partitions)
  • Bytes Consumed Rate(All Topics)
  • Bytes Consumed Rate(Topic Specific)
  • Records Consumed Rate(All Topics)
  • Records Consumed Rate(Topic Specific)
  • Fetch Rate
  • Records Lag
  • Records Lag Max
  • Records Per Request Avg
  • Fetch Throttle Time Avg
  • Fetch Throttle Time Max
  • Topic Name
  • Partition No

2. Kafka producer

Install the Apache Kafka Producer plugin to monitor key Kafka producer metrics and ensure a steady flow of data.


      Supported metrics:

  • Compression Rate Avg
  • Response Rate
  • Request Rate
  • Request Latency Avg
  • Outgoing Byte Rate
  • IO Wait Time NS Avg
  • Batch Size Avg

3. Kafka broker


      Supported metrics:

  • Under Replicated Partitions
  • ISR Shrinks Per Sec
  • ISR Expands Per Sec
  • Active Controller Count
  • Offline Partitions Count
  • Leader Election Rate And Time Ms
  • Unclean Leader Elections Per Sec
  • Total Time ms
  • Purgatory Size
  • Bytes In Per Sec
  • Network Request Rate
  • Network Error Rate
  • Total Broker Partitions
  • Young Generation GC Count
  • Young Generation GC Time
  • Old Generation GC Count
  • Old Generation GC Time
  • Log End Offset

Prerequisites for installing the plugins:

  • Install the jmxquery module for Python3.

           pip install jmxquery

Plugin Installation:

If you want to install the Kafka consumer monitoring plugin,

  • Create a directory named "kafka_consumer_monitoring" under the Site24x7 Linux Agent plugin directory:

    Linux    ->   /opt/site24x7/monagent/plugins/kafka_consumer_monitoring
  • Download all the files in the "kafka_consumer_monitoring" folder and place it under the "kafka_consumer_monitoring" directory.

wget https://raw.githubusercontent.com/site24x7/plugins/master/Kafka_Complete_Monitoring/Kafka_Consumer_Monitoring/kafka_consumer_monitoring.py


  • Provide appropriate configuration in the kafka_consumer_monitoring.cfg file.

To test the plugin manually, execute the below command:

 python3 kafka_consumer_monitoring.py --kafka_consumer_host=<KAFKA_CONSUMER_HOST_NAME> kafka_consumer_jmx_port=<KAFKA_CONSUMER_PORT_NO> --kafka_consumer_partition=<KAFKA_CONSUMER_PARTITION_NO> --kafka_topic_name=<KAFKA_TOPIC_NAME> --kafka_consumer_client_id=<KAFKA_CONSUMER_CLIENT_ID> --logs_enabled=False --log_type_name=None --log_file_path=None

The Kafka consumer plugin will be added for monitoring during the next data collection event of the agent.

Repeat the respective processes for producer and broker plugins as well.

Happy monitoring.

Like (0) Reply

Was this post helpful?