Go to All Forums

Request plugin for get the Kafka topic message lag

Hi Support team,

Currently, I need to monitor Kafka topic message lag in Confluence cloud. Could you please help us create plugin for this propose?

 

Regards,

Jirawat.po

Like (2) Edit Delete Reply
Replies (3)

Re: Request plugin for get the Kafka topic message lag

Hi Jirawat.po,

Thanks for the request. 

We do have a plugin for monitoring the metric 'Kafka topic message lag', which is present in our GitHub. The plugin's link is as follows

https://github.com/site24x7/plugins/tree/master/confluent/confluent_kafka

Metric details:

https://github.com/site24x7/plugins/blob/master/confluent/confluent_kafka/metrics.txt

#Lag in number of messages per follower replica
kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=([-.\w]+),topic=([-.\w]+),partition=([0-9]+) <type>.<name>.<clientId>.<topic>.<partition>

Kindly install confluent_kafka using the ReadMe steps present in the plugins' GitHub link. 

In case of any clarification, let us know. We are glad to support you. 

Regards,
Muralikrishnan

Product Manager

Like (0) Reply

Re: Re: Request plugin for get the Kafka topic message lag

Hi Muralikrishnan,

Currently, I need to access to Confluent Cloud for access to Kafka in cloud service. I have to use connection with 

kafka_producer.py. Plug in that provide from you support. Is it possible to provide me the extra feature base on python plugin.

 

Or If you can provide method for connect to Broker with plugin "github.com/site24x7/plugins/tree/master/confluent/confluent_kafka" is ok. Here is the current method of connection which I am running with your Python-Kafka-Plugin.

KafkaConsumer('test',bootstrap_servers=['pkc-ldvr1.asia-xxxx.xxxx.xxxx:9092'],ssl_cafile='/etc/ssl/certs/ca-bundle.crt',ssl_check_hostname='false',security_protocol="SASL_SSL",sasl_mechanism='PLAIN', sasl_plain_username='xxxxx',sasl_plain_password='xxxx')

 

 

Regards,

Jirawat.po

Like (0) Edit Delete Reply

Re: Re: Re: Request plugin for get the Kafka topic message lag

Hi,

The kafka_producer.py script uses the python module "kafka-python" in which the metric "Kafka topic message lag" is not available.

The confluent-based kafka script uses jmx query to fetch the performance metrics of your kafka resources.

So please try "github.com/site24x7/plugins/tree/master/confluent/confluent_kafka" to get the "message lag " metric.

 

Regards,

Sriram

Like (0) Reply

Was this post helpful?