We would like to have a plugin to shows the lag for the provided consumer group.
Currently, I need to monitor Kafka consumer and topic message lag. we are not using confluent platform to use the existing plugin. Could you please help us create plugin for this propose which would offer following metrics specific to consumer, topic and partitions used.
(GROUP NAME, TOPIC, PARTITION, CURRENT-OFFSET, LOG-END-OFFSET, LAG, CONSUMER-ID, HOST and CLIENT-ID)
By executing below command could get the result i wanted but plugin for this would be helpful.
bin/kafka-consumer-groups --bootstrap-server <kafka-broker>:9092 --describe --group sample-consumer-group
Hi Guest,
Thanks for your request.
We will analyze the Kafka consumer metrics and add them to our Plugins Roadmap. We will update in this thread, once we are done.
Regards,
Muralikrishnan
Product Manager
Hi,
Greeting!
We are happy to inform you that a full stack of Kafka's performance monitoring has been implemented. The following GitHub link contains the plugin details.
https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring
The following components have been monitored using our plugins.
1. Kafka consumer monitoring:
https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring/Kafka_Consumer_Monitoring
Metrics:
- Records Lag Max(All Partitions) - Maximum number of messages consumer is behind producer across all partitions
- Bytes Consumed Rate(All Topics) - Average number of bytes consumed per second across all topics.
- Bytes Consumed Rate(Topic Specific) - Average number of bytes consumed per second for a specific topic
- Records Consumed Rate(All Topics) - Average number of records consumed per second across all topics
- Records Consumed Rate(Topic Specific) - Average number of records consumed per second for a specific topic
- Fetch Rate - Number of fetch requests per second from the consumer
- Records Lag - Number of messages consumer is behind producer on this partition
- Records Lag Max - Maximum number of messages consumer is behind producer, for a specific partition
- Records Per Request Avg - Average number of records in each request
- Fetch Throttle Time Avg - Average throttle time milliseconds
- Fetch Throttle Time Max - Maximum throttle time in milliseconds
- Topic Name - Name of the Topic Specified to collect the metrics
- Partition No - Partition no specified to collect the metrics
2. Kafka producer monitoring:
https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring/Kafka_Producer_Monitoring
Metrics:
- Compression Rate Avg - Average compression rate of batches sent
- Response Rate - Average number of bytes consumed per second across all topics.
- Request Rate - Average number of requests sent per second
- Request Latency Avg - Average request latency
- Outgoing Byte Rate - Average number of outgoing/incoming bytes per second
- IO Wait Time NS Avg - Average length of time the I/O thread spent waiting for a socket
- Batch Size Avg - The average number of bytes sent per partition per request
3. Kafka broker monitoring:
https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring/Kafka_Broker_Monitoring
Metrics:
- Under Replicated Partitions - Number of unreplicated partitions
- ISR Shrinks Per Sec - The rate at which the pool of in-sync replicas (ISRs) shrinks
- ISR Expands Per Sec - The rate at which the pool of in-sync replicas (ISRs) expands
- Active Controller Count - Number of active controllers in the cluster
- Offline Partitions Count - Number of offline partitions
- Leader Election Rate And Time Ms - Leader election rate and latency
- Unclean Leader Elections Per Sec - Number of “unclean” elections per second
- Total Time ms - Total time to serve the specified request
- Purgatory Size - Number of requests waiting in producer purgatory
- Bytes In Per Sec - Aggregate incoming byte rate
- Network Request Rate - The average number of bytes sent per partition per request
- Network Error Rate - The Error Rate of the network
- Total Broker Partitions - Number of partitions on this broker. This should be mostly even across all brokers
- Young Generation GC Count - The total count of young garbage collection processes executed by the JVM
- Young Generation GC Time - The total amount of time (in milliseconds) the JVM has spent executing young garbage collection processes
- Old Generation GC Count - The total count of old garbage collection processes executed by the JVM
- Old Generation GC Time - The total amount of time (in milliseconds) the JVM has spent executing old garbage collection processes
- Log End Offset - The log end offset is the offset of the last message written to a log
Prerequisites :
- Install of jmxquery module for Python3
pip install jmxquery
- Download and install the latest version of the [Site24x7 Linux agent] (https://www.site24x7.com/app/client#/admin/inventory/add-monitor) in the server where you plan to run the plugin.
Plugin Installation:
- Download respective Kafka plugin, for example, to monitor Kafka consumer,
for example in Linux agent -> /opt/site24x7/monagent/plugins
- Provide appropriate configuration in kafka_consumer_monitoring.cfg file.
Create a directory named “kafka_consumer_monitoring” under plugins folder and place it under the downloaded files. - To test the plugin manually, execute the below command -
python3 kafka_consumer_monitoring.py --kafka_consumer_host=<KAFKA_CONSUMER_HOST_NAME> kafka_consumer_jmx_port=<KAFKA_CONSUMER_PORT_NO> --kafka_consumer_partition=<KAFKA_CONSUMER_PARTITION_NO> --kafka_topic_name=<KAFKA_TOPIC_NAME> --kafka_consumer_client_id=<KAFKA_CONSUMER_CLIENT_ID> --logs_enabled=False --log_type_name=None --log_file_path=None
- On the next data collection of the agent, the Kafka consumer plugin will be added for monitoring.
In case of any clarification, please reply to this thread.
Regards,
Muralikrishnan
Product Manager