Greetings!

We are happy to introduce Apache Kafka plugin integrations to monitor your entire Kafka stack. You will now be able to monitor critical Kafka producer, consumer, and broker metrics in real time and stay ahead of issues and bottlenecks.

Refer to the below GitHub link for the Kafka monitoring plugin details and setup steps:

https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring

You can monitor the following components with our Kafka plugins: 

1. Kafka consumer

Install the Kafka Consumer plugin and monitor key consumer metrics.

https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring/Kafka_Consumer_Monitoring

      Supported metrics: 

• Records Lag Max(All Partitions)
• Bytes Consumed Rate(All Topics)
• Bytes Consumed Rate(Topic Specific)
• Records Consumed Rate(All Topics)
• Records Consumed Rate(Topic Specific)
• Fetch Rate
• Records Lag
• Records Lag Max
• Records Per Request Avg
• Fetch Throttle Time Avg
• Fetch Throttle Time Max
• Topic Name
• Partition No

2. Kafka producer

Install the Apache Kafka Producer plugin to monitor key Kafka producer metrics and ensure a steady flow of data.

https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring/Kafka_Producer_Monitoring

      Supported metrics:

• Compression Rate Avg
• Response Rate
• Request Rate
• Request Latency Avg
• Outgoing Byte Rate
• IO Wait Time NS Avg
• Batch Size Avg

3. Kafka broker

 https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring/Kafka_Broker_Monitoring

      Supported metrics:

• Under Replicated Partitions
• ISR Shrinks Per Sec
• ISR Expands Per Sec
• Active Controller Count
• Offline Partitions Count
• Leader Election Rate And Time Ms
• Unclean Leader Elections Per Sec
• Total Time ms
• Purgatory Size
• Bytes In Per Sec
• Network Request Rate
• Network Error Rate
• Total Broker Partitions
• Young Generation GC Count
• Young Generation GC Time
• Old Generation GC Count
• Old Generation GC Time
• Log End Offset

Prerequisites for installing the plugins:

• Install the jmxquery module for Python3.

           pip install jmxquery

• Download and install the latest version of the [Site24x7 Linux agent] (https://www.site24x7.com/app/client#/admin/inventory/add-monitor) in the server where you plan to run the plugin.

Plugin Installation:

If you want to install the Kafka consumer monitoring plugin,

• Create a directory named "kafka_consumer_monitoring" under the Site24x7 Linux Agent plugin directory:
  
  Linux    ->   /opt/site24x7/monagent/plugins/kafka_consumer_monitoring
  
• Download all the files in the "kafka_consumer_monitoring" folder and place it under the "kafka_consumer_monitoring" directory.

wget https://raw.githubusercontent.com/site24x7/plugins/master/Kafka_Complete_Monitoring/Kafka_Consumer_Monitoring/kafka_consumer_monitoring.py

wget
https://raw.githubusercontent.com/site24x7/plugins/master/Kafka_Complete_Monitoring/Kafka_Consumer_Monitoring/kafka_consumer_monitoring.cfg

• Provide appropriate configuration in the kafka_consumer_monitoring.cfg file.

To test the plugin manually, execute the below command:

 python3 kafka_consumer_monitoring.py --kafka_consumer_host=<KAFKA_CONSUMER_HOST_NAME> kafka_consumer_jmx_port=<KAFKA_CONSUMER_PORT_NO> --kafka_consumer_partition=<KAFKA_CONSUMER_PARTITION_NO> --kafka_topic_name=<KAFKA_TOPIC_NAME> --kafka_consumer_client_id=<KAFKA_CONSUMER_CLIENT_ID> --logs_enabled=False --log_type_name=None --log_file_path=None

The Kafka consumer plugin will be added for monitoring during the next data collection event of the agent.

Repeat the respective processes for producer and broker plugins as well.

Happy monitoring.