Kafka Monitoring

Monitor the performance of your entire Apache Kafka infrastructure with Site24x7's ready-to-install Kafka monitoring plugin integrations. Our Kafka monitoring tool tracks critical consumer, broker, and producer metrics to help you stay on top of your Kafka stack and prevent potential data loss.

Add your first server monitoring plugin for free and subsequent plugins for just $1 each per month.

Start 30-day free trial Try now, sign up in 30 seconds
Kafka Monitoring

What is Apache Kafka?

Apache Kafka is an open-source, distributed stream-processing platform used to collect, store and process real-time data streams. It was originally developed by LinkedIn and was open sourced in 2011. Kafka later became a part of the Apache Software Foundation. Written in Java and Scala, its core capabilities include high throughput, scalability, and low latency. Apache Kafka is often used to build stream data pipelines and real-time, event-driven applications.

What is Apache Kafka monitoring?

Apache Kafka monitoring involves collecting, analyzing, and visualizing metrics from the Kafka cluster, including producer, broker, and consumer data. It ensures Kafka's performance, stability, and resource utilization while facilitating fault detection and capacity planning for efficient data streaming pipelines and issue resolution.

Kafka performance monitoring with Site24x7

Track the health of your Kafka infrastructure and identify bottlenecks affecting its performance. With our Kafka monitoring tool, keep track of unexpected trends in Kafka producer, broker, and consumer instances through our metric graphs, so that you can troubleshoot quickly.

Get insights on critical Kafka performance metrics

Leverage key Kafka availability, usage, and performance metrics, including producer's request rate, network request rate by broker, and consumer's fetch rate, and stay ahead of issues affecting them.

Stay updated with instant alerts

Get instant notifications every time values cross pre-defined threshold limits and swiftly take preventive or recovery measures.

Remediate incidents with event-driven IT automation

Set up automated workflows to take corrective measures in Kafka when pre-defined threshold breaches occur.

Create custom dashboards

Assemble data from different parts of your infrastructure with drag-and-drop widgets and build personalized dashboards to get a holistic view of Kafka stack's performance.

Get started with Site24x7's Kafka monitoring tool

  • Supported Platforms:
  • Python Version:
    2.7 & above
  • Linux Agent Version:
    15.0.0 & above

Performance Metrics

The following metrics are captured by the Kafka Producer monitoring plugin:

Name Description
Compression Rate Avg The average compression rate of batches sent
Response Rate The average number of responses received per second across all topics
Request Rate The average number of requests sent per second
Request Latency Avg The average request latency
Outgoing Byte Rate The average number of outgoing bytes per second
IO Wait Time NS Avg The average time the I/O thread spent waiting for a socket
Batch Size Avg The average number of bytes sent per partition per request
Client ID The Client ID name

The following metrics are captured by the Kafka Broker monitoring plugin:

Name Description
Under Replicated Partitions The number of partitions that do not have enough replicas to meet the required replication factor
ISR Shrinks Per Sec The rate at which the pool of in-sync replicas (ISRs) shrink
ISR Expands Per Sec The rate at which the pool of in-sync replicas (ISRs) expand
Active Controller Count The number of active controllers in a cluster
Offline Partitions Count The number of offline partitions
Leader Election Rate And Time Ms The leader election rate and latency. A leader election occurs when the current leader of a partition becomes unavailable, and the election of the new leader should proceed with minimal latency.
Unclean Leader Elections Per Sec The number of unclean or unsuccessful leader elections per second
Total Time Ms The total time to serve a specified request
Purgatory Size The number of requests waiting in the producer purgatory
Bytes In Per Sec The aggregate incoming byte rate
Network Request Rate The number of network requests per second for a broker
Network Error Rate The rate of errors in the network
Total Broker Partitions The number of partitions on the particular broker, which should be mostly even across all brokers
Young Generation GC Count The total count of young garbage collection processes executed by the Java Virtual Machine (JVM)
Young Generation GC Time The total amount of time (in milliseconds) the JVM has spent executing young garbage collection processes
Old Generation GC Count The total count of old garbage collection processes executed by the JVM
Old Generation GC Time The total amount of time (in milliseconds) the JVM has spent executing old garbage collection processes
Log End Offset The offset of the last message written to a log

The following metrics are captured by the Kafka Consumer monitoring plugin:

Name Description
Records Lag Max (All Partitions) The maximum number of messages for which the consumer lags behind the producer across all partitions
Bytes Consumed Rate (All Topics) The average number of bytes consumed per second across all topics
Bytes Consumed Rate (Topic Specific) The average number of bytes consumed per second for a specific topic
Records Consumed Rate (All Topics) The average number of records consumed per second across all topics
Records Consumed Rate (Topic Specific) The average number of records consumed per second for a specific topic
Fetch Rate The number of fetch requests per second from the consumer
Records Lag The number of messages the consumer is behind the producer on the particular partition
Records Lag Max The maximum number of messages the consumer is behind the producer for a specific partition
Records Per Request Avg The average number of records in each request
Fetch Throttle Time Avg The average throttle time in milliseconds
Fetch Throttle Time Max The maximum throttle time in milliseconds
Topic Name The name of the topic specified to collect the metrics
Partition No. The partition number specified to collect the metrics
Client ID The client ID of the consumer to collect the metrics

Setup

Prerequisites

Ensure you meet the following requirements before installing the plugin:

  • A Site24x7 account. If you don't have one, sign up here for free.
  • Download and install the Site24x7 server monitoring agent (Linux | Windows) on the server where you intend to run the plugin.
  • Enable the Kafka JMX ports.
    • To enable the Kafka Broker JMX port, find the following code block in the kafka-server-start.sh script:
      if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
      export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
      fi

      Below the aforementioned code block, paste the following lines:

      export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -
      Dcom.sun.management.jmxremote.authenticate=false -
      Dcom.sun.management.jmxremote.ssl=false -
      Dcom.sun.management.jmxremote.port=9999"
      export JMX_PORT=9999

      Restart the Kafka Broker service after making the above changes.

    • To enable the Kafka Producer JMX port, find the following code block in the kafka-console-producer.sh script:
      if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
      export KAFKA_HEAP_OPTS="-Xmx512M"
      fi

      Below the aforementioned code block, paste the following lines:

      export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -
      Dcom.sun.management.jmxremote.authenticate=false -
      Dcom.sun.management.jmxremote.ssl=false -
      Dcom.sun.management.jmxremote.port=9998"
      export JMX_PORT=9998

      Restart the Kafka Producer service after making the above changes.

    • To enable the Kafka Consumer JMX port, find the following code block in the kafka-console-consumer.sh script:
      if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
      export KAFKA_HEAP_OPTS="-Xmx512M"
      fi

      Below the aforementioned code block, paste the following lines:

      export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -
      Dcom.sun.management.jmxremote.authenticate=false -
      Dcom.sun.management.jmxremote.ssl=false -
      Dcom.sun.management.jmxremote.port=9997"
      export JMX_PORT=9997

      Restart the Kafka Consumer service after making the above changes.

  • Install the jmxquery module for Python3.
    pip install jmxquery

Installation

  • Create a folder named kafka_producer_monitoring.
  • Download the kafka_producer_monitoring.py and the kafka_producer_monitoring.cfg files from our Github repository and place them in the kafka_producer_monitoring folder.
    wget https://raw.githubusercontent.com/site24x7/plugins/master/Kafka_Complete_Monitoring/Kafka_Producer_Monitoring/kafka_producer_monitoring.py
    wget https://raw.githubusercontent.com/site24x7/plugins/master/Kafka_Complete_Monitoring/Kafka_Producer_Monitoring/kafka_producer_monitoring.cfg
  • Add the following configuration in kafka_producer_monitoring.cfg file:
    [kafka_producer_1]
    kafka_producer_host=<KAFKA_PRODUCER_HOST_NAME>
    kafka_producer_jmx_port=<KAFKA_PRODUCER_PORT_NO>
    kafka_producer_client_id=<KAFKA_PRODUCER_CLIENT_ID>
    logs_enabled=False
    log_type_name=None
    log_file_path=None
  • Follow the steps in this article to learn how to run the python script on a Windows server. You don't need to do this for Linux.
  • Move the kafka_producer_monitoring folder with the individual files within to the Site24x7 server monitoring plugin directory.

    For Linux: /opt/site24x7/monagent/plugins/

    For Windows: C:\Program Files (x86)\Site24x7\WinAgent\monitoring\plugins\

Tip: To verify if the plugin is working, execute the command below with appropriate arguments to check for a valid json output:

python3 kafka_producer_monitoring.py --kafka_producer_host= kafka_producer_jmx_port= --kafka_producer_client_id= --logs_enabled=False --log_type_name=None --log_file_path=None

The agent will automatically execute the plugin within five minutes and send performance data to the Site24x7 data center.

To view performance charts and set thresholds for the various performance metrics:

  • Log in to Site24x7.
  • Navigate to Plugins > click the required monitor.
  • Create a folder named kafka_broker_monitoring.
  • Download the kafka_broker_monitoring.py and the kafka_broker_monitoring.cfg files from our Github repository and place them in the kafka_broker_monitoring folder.
    wget https://raw.githubusercontent.com/site24x7/plugins/master/Kafka_Complete_Monitoring/Kafka_Broker_Monitoring/kafka_broker_monitoring.py
    wget https://raw.githubusercontent.com/site24x7/plugins/master/Kafka_Complete_Monitoring/Kafka_Broker_Monitoring/kafka_broker_monitoring.cfg
  • Add the following configuration in kafka_broker_monitoring.cfg file:
    [kafka_broker_1]
    kafka_host=<KAFKA_BROKER_HOST_NAME>
    kafka_jmx_port=<KAFKA_BROKER_PORT_NO>
    kafka_consumer_partition=<KAFKA_CONSUMER_PARTITION_NO>
    kafka_topic_name=<KAFKA_TOPIC_NAME>
    logs_enabled=False
    log_type_name=None
    log_file_path=None
  • Follow the steps in this article to learn how to run the python script on a Windows server. You don't need to do this for Linux.
  • Move the kafka_broker_monitoring folder with the individual files within to the Site24x7 server monitoring plugin directory.

    For Linux: /opt/site24x7/monagent/plugins/

    For Windows: C:\Program Files (x86)\Site24x7\WinAgent\monitoring\plugins\

Tip: To verify if the plugin is working, execute the command below with appropriate arguments to check for a valid json output:

python3 kafka_broker_monitoring.py --kafka_host= --kafka_jmx_port= --kafka_consumer_partition= --kafka_topic_name= --logs_enabled=False --log_type_name=None --log_file_path=None

The agent will automatically execute the plugin within five minutes and send performance data to the Site24x7 data center.

To view performance charts and set thresholds for the various performance metrics:

  • Log in to Site24x7.
  • Navigate to Plugins > click the required monitor.
  • Create a folder named kafka_consumer_monitoring.
  • Download the kafka_consumer_monitoring.py and the kafka_consumer_monitoring.cfg files from our Github repository and place them in the kafka_consumer_monitoring folder.
    wget https://raw.githubusercontent.com/site24x7/plugins/master/Kafka_Complete_Monitoring/Kafka_Consumer_Monitoring/kafka_consumer_monitoring.py
    wget https://raw.githubusercontent.com/site24x7/plugins/master/Kafka_Complete_Monitoring/Kafka_Consumer_Monitoring/kafka_consumer_monitoring.cfg
  • Add the following configuration in kafka_consumer_monitoring.cfg file:
    [kafka_consumer_1]
    kafka_consumer_host=<KAFKA_CONSUMER_HOST_NAME>
    kafka_consumer_jmx_port=<KAFKA_CONSUMER_PORT_NO>
    kafka_consumer_partition=<KAFKA_CONSUMER_PARTITION_NO>
    kafka_topic_name=<KAFKA_TOPIC_NAME>
    kafka_consumer_client_id=<KAFKA_CONSUMER_CLIENT_ID>
    logs_enabled=False
    log_type_name=None
    log_file_path=None
  • Follow the steps in this article to learn how to run the python script on a Windows server. You don't need to do this for Linux.
  • Move the kafka_consumer_monitoring folder with the individual files within to the Site24x7 server monitoring plugin directory.

    For Linux: /opt/site24x7/monagent/plugins/

    For Windows: C:\Program Files (x86)\Site24x7\WinAgent\monitoring\plugins\

Tip: To verify if the plugin is working, execute the command below with appropriate arguments to check for a valid json output:

python3 kafka_consumer_monitoring.py --kafka_consumer_host= kafka_consumer_jmx_port= --kafka_consumer_partition= --kafka_topic_name= --kafka_consumer_client_id= --logs_enabled=False --log_type_name=None --log_file_path=None

The agent will automatically execute the plugin within five minutes and send performance data to the Site24x7 data center.

To view performance charts and set thresholds for the various performance metrics:

  • Log in to Site24x7.
  • Navigate to Plugins > click the required monitor.

Plugin Contribution

Feel free to contribute to our existing plugin and come up with suggestions or feedback on our Community.

  • Supported Platforms:
  • Python Version: 2.7 & above
  • Linux Agent Version:15.0.0 & above

Over 13,000 actively paying customers trust Site24x7

Brand logos of our various customers

Check out our 100+ plugin integrations, or build your own

Choose from our list of more than 100 ready-to-use plugins and monitor your full stack of applications across web servers, databases, load balancers, and more.

See all plugin integrations