Kafka Monitoring with Site24x7 Plugins

Use Site24x7 plugins to monitor the performance metrics of your Apache Kafka server

Install and configure the Kafka plugin to monitor the fault-tolerant, high capacity messaging middleware Kafka. Take informed troubleshooting decisions by keeping track of critical metrics like connection count, incoming and outgoing bytes rate and lot more.

This document details how to configure the Apache Kafka plugin and the monitoring metrics for providing in-depth visibility into the performance, availability, and usage stats of Kafka servers.

Apache Kafka performance monitoring metrics:

Use our wide array of metrics and get notified of hazardous errors that require your attention. Keep track of unexpected trends through our metric graphs and troubleshoot as quickly as possible. Various out-of-the-box metrics we support are:

Connection count

The connection_count gives the current number of active connections with the kafka cluster

Incoming/outgoing byte rate

The incoming_byte_rate and outgoing_byte_rate gives the average number of incoming bytes received per second and outgoing bytes sent per second respectively

Average request latency

A measure of time between a request sent by the producer till a response received from the broker is taken as the avg_request_latency

Number of network operations

The average number of network operations (read or write) on all connections per second is given as the network_io_rate

Request rate

The rate at which producers send data to brokers is calculated by the metric request_rate

Response rate

The response_rate shows the rate of response received from the brokers

Time spent by the input/output thread

io_time_ns_avg gives the average length of time the input/output thread spends on waiting for a socket (in ns)

How it works?

  • Log-in to your Site24x7 account. Sign up here if you don't have one.
  • Download and install the latest version of Site24x7 Linux agent
  • Install the Kafka plugin
  • The agent will execute the Kafka plugin and push the data to the Site24x7 server

Prerequisites:

  • Download the file from https://pypi.python.org/pypi/kafka-python sudo wget https://pypi.python.org/packages/81/94/f9151888b40339f2624759df8241080020acafed49d3a4369ceedd72c096/kafka-python-1.3.1.tar.gz#md5=2494d82f096d21af7a72b3c5bd26bdfa
  • Extract the file sudo tar -xvzf kafka-python-1.3.1.tar.gz
  • Go to the directory where the file is extracted cd kafka-python-1.3.1/
  • Install the module
  • sudo python setup.py build
    sudo python setup.py install

Kafka plugin installation:

  • Create a directory with the name "kafka_producer", under Site24x7 Linux agent plugin directory - /opt/site24x7/monagent/plugins/
    sudo mkdir kafka_producer
  • Download the "kafka_producer.py" from our GitHub repository and place it under the "kafka_producer" directory
  • The default python path given in the plugin script is #!/usr/bin/python. If you wish to provide an alternate python path, replace the existing one preceded by the shebang character "#!".
    Eg : #!/usr/local/bin/python3
  • Change the values of BROKER_NAME and PORT to match your configuration
  • The server agent will report stats on the performance of Kafka under the Plugins tab in the Site24x7 web client. In case the plugin is not listed in the Site24x7 web client, restart the agent.
    sudo /etc/init.d/site24x7monagent restart