Kafka monitoring with Site24x7 plugins.

Use Site24x7 plugins to monitor the performance metrics of your Apache Kafka server

Install and configure the Kafka plugin to monitor the fault-tolerant, high capacity messaging middleware Kafka. Take informed troubleshooting decisions by keeping track of critical metrics like connection count, incoming and outgoing bytes rate and lot more.

This document details how to configure the Apache Kafka plugin and the monitoring metrics for providing in-depth visibility into the performance, availability, and usage stats of Kafka servers.

Apache Kafka performance monitoring metrics:

Use our wide array of metrics and get notified of hazardous errors that require your attention. Keep track of unexpected trends through our metric graphs and troubleshoot as quickly as possible. Various out-of-the-box metrics we support are:

Connection count

The connection_count gives the current number of active connections with the kafka cluster

Incoming/outgoing byte rate

The incoming_byte_rate and outgoing_byte_rate gives the average number of incoming bytes received per second and outgoing bytes sent per second respectively

Average request latency

A measure of time between a request sent by the producer till a response received from the broker is taken as the avg_request_latency

Number of network operations

The average number of network operations (read or write) on all connections per second is given as the network_io_rate

Request rate

The rate at which producers send data to brokers is calculated by the metric request_rate

Response rate

The response_rate shows the rate of response received from the brokers

Time spent by the input/output thread

io_time_ns_avg gives the average length of time the input/output thread spends on waiting for a socket (in ns)

How it works?

  • Log-in to your Site24x7 account. Sign up here if you don't have one.
  • Download and install the latest version of Site24x7 Linux agent
  • Install the Kafka plugin
  • The agent will execute the Kafka plugin and push the data to the Site24x7 server

Prerequisites:

How to install kafka-python module

Kafka plugin installation:

  • Create a directory with the name "kafka_producer", under Site24x7 Linux agent plugin directory - /opt/site24x7/monagent/plugins/
  • sudo mkdir kafka_producer
  • Download the "kafka_producer.py" from our GitHub repository and place it under the "kafka_producer" directory
  • Replace the shebang character "#!" with the appropriate path for Python 3 in your system
  • Eg : #!/usr/local/bin/python3
  • Change the values of BROKER_NAME and PORT to match your configuration
  • The server agent will report stats on the performance of Kafka under the Plugins tab in the Site24x7 web client. In case the plugin is not listed in the Site24x7 web client, restart the agent.
  • sudo /etc/init.d/site24x7monagent restart

Monitoring additional metrics:

  • To monitor additional metrics, edit the "kafka_producer.py" file and add the new metrics that need monitoring
  • Increment the plugin version value in the file "kafka_producer.py" to view the newly added metrics (For e.g., change the default plugin version from PLUGIN_VERSION = "1" to "PLUGIN_VERSION = "2")

Related plugins:

  • ActiveMQ plugin - Monitor performance metrics of your Apache ActiveMQ instances
  • MongoDB plugin - Monitor performance metrics of your MongoDB databases
  • Memcached plugin - Analyze performance of your Memcached server
  • Nagios plugin - Execute thousands of Nagios plugins in Site24x7 without the need of running a Nagios server
  • Out-of-the-box plugins - Monitor your entire app stack with our extensive list of integrations
  • Create custom plugins - Create custom Linux and Windows plugins and monitor custom attributes
Trusted

World's leading companies

Client

Don't have a Site24x7 account? Sign up now!