Hi Support team,
Currently, I need to monitor Kafka topic message lag in Confluence cloud. Could you please help us create plugin for this propose?
Regards,
Jirawat.po
Hi Jirawat.po,
Thanks for the request.
We do have a plugin for monitoring the metric 'Kafka topic message lag', which is present in our GitHub. The plugin's link is as follows
https://github.com/site24x7/plugins/tree/master/confluent/confluent_kafka
Metric details:
https://github.com/site24x7/plugins/blob/master/confluent/confluent_kafka/metrics.txt
#Lag in number of messages per follower replica
kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=([-.\w]+),topic=([-.\w]+),partition=([0-9]+) <type>.<name>.<clientId>.<topic>.<partition>
Kindly install confluent_kafka using the ReadMe steps present in the plugins' GitHub link.
In case of any clarification, let us know. We are glad to support you.
Regards,
Muralikrishnan
Product Manager
Hi Muralikrishnan,
Currently, I need to access to Confluent Cloud for access to Kafka in cloud service. I have to use connection with
kafka_producer.py. Plug in that provide from you support. Is it possible to provide me the extra feature base on python plugin.
Or If you can provide method for connect to Broker with plugin "github.com/site24x7/plugins/tree/master/confluent/confluent_kafka" is ok. Here is the current method of connection which I am running with your Python-Kafka-Plugin.
KafkaConsumer('test',bootstrap_servers=['pkc-ldvr1.asia-xxxx.xxxx.xxxx:9092'],ssl_cafile='/etc/ssl/certs/ca-bundle.crt',ssl_check_hostname='false',security_protocol="SASL_SSL",sasl_mechanism='PLAIN', sasl_plain_username='xxxxx',sasl_plain_password='xxxx')
Regards,
Jirawat.po
Hi,
The kafka_producer.py script uses the python module "kafka-python" in which the metric "Kafka topic message lag" is not available.
The confluent-based kafka script uses jmx query to fetch the performance metrics of your kafka resources.
So please try "github.com/site24x7/plugins/tree/master/confluent/confluent_kafka" to get the "message lag " metric.
Regards,
Sriram