Help Amazon Web Services Kinesis Data Firehose

Kinesis Data Firehose Integration

A fully managed service to help you load streaming data continuously to data stores and other destinations in near real time. With Site24x7's AWS integration you can monitor metrics on throughput, delivery, data transformation and API activity to make sure records are reaching their destination.

Setup and configuration

  • If you haven't done it already, enable access to your AWS resource by creating Site24x7 as an IAM user or by creating a cross-account IAM role between your account and Site24x7's AWS account. Learn more.
  • Next, In the Integrate AWS Account page, please make sure the Kinesis Firehose checkbox is selected in the Services to be discovered field. Learn more.

Policies and Permissions

Please make sure the following read level API actions are present in the IAM policy assigned to Site24x7 entity. learn more.

  • firehose:Describe* : Describes the specified delivery stream and gets the status.
  • firehose:List*: Returns a list of delivery streams along with tags.

Polling frequency

Performance metrics are collected as per the poll frequency set (1 minute to a day), by querying the CloudWatch API. Learn more.

To view metrics

  • Log in to the Site24x7 console, click on AWS > Monitored AWS account and select Kinesis Firehose from the drop down
  • From the list of monitored delivery streams, choose the stream for which you want to access metrics for.
  • The various graphs applicable for your delivery stream are displayed in the stream details tab.

Supported metrics

The following metrics are collected on a per delivery stream level:

Based on backup

AttributeDescriptionData typeStatistics
BackupToS3.Bytes Measures the number of bytes delivered to Amazon S3 for backup. Bytes Sum
BackupToS3.DataFreshness (Sum) Measures the age of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup. Seconds Sum
BackupToS3.DataFreshness (Maximum) Measures the age of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup. Seconds Maximum
BackupToS3.Records Measures the number of records delivered to Amazon S3 for backup. Count Sum
BackupToS3.Success Measures the sum of successful Amazon S3 put commands for backup over sum of all Amazon S3 backup put commands. Count  Sum 

Based on destination

AttributeDescriptionData typeStatistics
DeliveryToElasticsearch.Bytes Measures the number of bytes indexed to Amazon ES. Bytes Sum
DeliveryToElasticsearch.Records Measures the number of records indexed to Amazon ES. Count Sum
DeliveryToElasticsearch.Success Measures the sum of the successfully indexed records over the sum of records that were attempted. Count  Sum 
DeliveryToRedshift.Bytes Measures the number of bytes copied to Amazon Redshift. Bytes Sum
DeliveryToRedshift.Records Measures the number of records copied to Amazon Redshift. Count Sum
DeliveryToRedshift.Success Measures the sum of successful Amazon Redshift COPY commands over the sum of all Amazon Redshift COPY commands. Count  Sum 
DeliveryToS3.Bytes Measures the number of bytes delivered to Amazon S3 over the specified time period. Bytes Sum
DeliveryToS3.DataFreshness (Sum) Measures age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the S3 bucket. Seconds Sum
DeliveryToS3.DataFreshness (Maximum) Measures age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the S3 bucket. Seconds Maximum
DeliveryToS3.Records Measures the number of records delivered to Amazon S3. Count Sum 
DeliveryToS3.Success Measures the sum of successful Amazon S3 put commands over the sum of all Amazon S3 put commands. Count  Sum 
DeliveryToSplunk.Bytes Measures the number of bytes delivered to Splunk. Bytes Sum 
DeliveryToSplunk.DataFreshness (Sum) Measures the age (from getting into Kinesis Firehose to now) of the oldest record in Kinesis Firehose. Any record older than this age has been delivered to Splunk. Seconds Sum
DeliveryToSplunk.DataFreshness (Maximum) Measures the age (from getting into Kinesis Firehose to now) of the oldest record in Kinesis Firehose. Any record older than this age has been delivered to Splunk. Seconds Maximum
DeliveryToSplunk.Records Measures the number of records delivered to Splunk Count  Sum 
DeliveryToSplunk.Success Measures the sum of the successfully indexed records over the sum of records that were attempted. Count  Sum 
IncomingBytes Measures the number of bytes ingested into the Kinesis Data Firehose stream. Bytes Sum 
IncomingRecords Measures the number of records ingested into the Kinesis Data Firehose stream. Count Sum 

API-level metrics

AttributeDescriptionData typeStatistics
DescribeDeliveryStream.Latency Measures the time taken per DescribeDeliveryStream operation. Milliseconds Average 
DescribeDeliveryStream.Requests Measures the total number of DescribeDeliveryStream requests. Count Sum 
ListDeliveryStreams.Latency Measures the time taken per ListDeliveryStream operation. Milliseconds Sum 
ListDeliveryStreams.Requests Measures the total number of ListFirehose requests. Count Sum 
PutRecord.Bytes Measures the number of bytes put to the Kinesis Data Firehose delivery stream. Bytes Sum 
PutRecord.Latency Measures the time taken per PutRecord operation. Milliseconds Sum 
PutRecord.Requests Measures the total number of PutRecord requests. Count Sum 
PutRecordBatch.Bytes Measures the number of bytes put to the Kinesis Data Firehose delivery stream using PutRecordBatch. Bytes Sum 
PutRecordBatch.Latency Measures the time taken per PutRecordBatch operation. Milliseconds  Average 
PutRecordBatch.Records Measures the total number of records from PutRecordBatch operations. Count Sum 
PutRecordBatch.Requests Measures the total number of PutRecordBatch requests. Count Sum 
UpdateDeliveryStream.Latency Measures the time taken per UpdateDeliveryStream operation. Milliseconds Average 
UpdateDeliveryStream.Requests Measures the total number of UpdateDeliveryStream requests Count Sum 

Data transformation metrics

Data points for the following performance metrics will only be available if you've enabled data transformation.

AttributeDescriptionData typeStatistics
ExecuteProcessing.Duration Measures the time it takes for each Lambda function invocation performed by Kinesis Data Firehose. Seconds Sum 
ExecuteProcessing.Success Measures the sum of the successful Lambda function invocations over the sum of the total Lambda function invocations. Count  Sum 
SucceedProcessing.Records Measures the number of successfully processed records over the specified time period. Count Sum 
SucceedProcessing.Bytes Measures the number of successfully processed bytes over the specified time period. Bytes Sum 

Configuration details

The following general configuration data is collected on a per delivery stream basis:

AttributeDescription
Stream name Displays the names for the delivery stream. 
Stream ARN  Shows the Amazon Resource Name for the Delivery stream. 
Stream status Displays the current status for the delivery stream. 
Stream Type Displays the logical category or type of index. 
Version ID Shows the version number for the delivery stream
Delivery Start Time Displays the time stamp when the record was first received by Firehose.
Source Stream ARN Shows the Amazon Resource Name for the source - direct or Kinesis stream.
Source IAM Role Displays the IAM Role attached to the source stream.
Destination Type Displays the destination - ElasticSearch cluster, S3 bucket, RedShift or Splunk
Was this document helpful?
Thanks for taking the time to share your feedback. We’ll use your feedback to improve our online help resources.

Help Amazon Web Services Kinesis Data Firehose