Amazon S3 Monitoring

Amazon S3 provides a reliable cloud storage platform to store and retrieve objects at any scale.

With Site24x7's S3 integration you can collect service level storage and request metrics on an individual bucket level. You can also, run periodic checks on critical S3 object objects from global locations to monitor functionality and response time.

Setup and configuration

  • If you haven't already, enable site24x7 access to your AWS account via key based or IAM role base authentication
  • In the Integrate AWS Account page, select the S3 checkbox under the Services to be discovered field.

Policies and Permissions

Site24x7 requires the following permissions to discover and monitor your S3 buckets. Learn more.

  • "s3:Get*" : Returns a list of S3 buckets and a subset of objects.
  • "s3:List*" : Returns configuration information for the S3 buckets.
  • "s3:Head*" : Same as the Get operation, except there is no response body.

Polling frequency

Storage metrics are collected on a day to day basis. Request and data transfer metrics are collected as per the poll frequency set (1 minute to a day). Learn more.

Supported performance metrics

Storage metrics

AttributeDescriptionStatisticsData type
Bucket size Measures the amount of data stored in the bucket for all storage classes, except Amazon Glacier. Average Bytes
Number of objects Measures the number of objects stored in the bucket for all storage classes, except Amazon Glacier. Average Count

Request metrics

Enable request metrics for your S3 buckets, using the Amazon S3 management console to monitor operations on your objects and buckets.

AttributeDescriptionStatisticsData type
All Requests Measures the total number of HTTP requests made to all the objects in a S3 bucket, regardless of type. If you are using metrics configuration for your bucket, then this metric only counts HTTP requests for objects meeting the filter criteria. Sum Count
GET requests Measures the total number of HTTP GET requests made for objects in a Amazon S3 bucket. Sum Count
PUT requests Measures the total number of HTTP PUT requests made for objects in a Amazon S3 bucket. Sum Count
Delete requests Measures the total number HTTP Delete requests made for objects in an Amazon S3 bucket. Multi object delete operation requests are also taken into account. The number of objects that were deleted are not known, only the number of delete request made for objects is measured. Sum Count
Head Requests Measures the total number of HTTP Head requests made to an Amazon S3 bucket. Sum  Count
List requests Measures the total number of HTTP List requests made to an Amazon S3 bucket. Sum Count
Post requests Measures the total number of HTTP POST requests made to an Amazon S3 bucket. Sum Count
Bytes downloaded Measures the total number of bytes downloaded for the requests sent to an Amazon S3 bucket. Where the response included a body. Sum  Bytes
Bytes uploaded Measures the total number of bytes uploaded for the requests sent to an Amazon S3 bucket. where the request included a body. Sum Bytes
4xx errors Measures the total number of generated client side errors for requests made to an Amazon S3 bucket. Sum Count
5xx errors Measures the total number of generated server side errors for requests made to an Amazon S3 bucket. Sum Count
First byte latency The time between when a S3 bucket receives the complete request body to when a response starts to be returned. Average Seconds
Total request latency The elapsed per-request time from the first byte received to the last byte sent to an Amazon S3 bucket. Average Seconds

Replication metrics

AttributeDescriptionStatisticsData type
Replication Latency The maximum number of seconds by which the replication destination Region is behind the source Region for a given replication rule. Maximum Seconds
Bytes Pending Replication The total number of bytes of objects pending replication for a given replication rule. Maximum MB
Operations Pending Replication The number of operations pending replication for a given replication rule. Maximum Count

Configuration details

The following inventory data is collected for your monitored S3 bucket.

AttributeDescription
Bucket Name Name of the S3 bucket
Creation time Date the bucket was created
Location of S3 bucket The AWS region where the bucket is provisioned
Bucket policy Bucket policy and user policy assigned to manage access to your S3 bucket
Lifecycle rules Rules configured for object lifecycle management
Storage class The type of storage class in use (Standard, IA(Infrequent Access), Glacier or RRS(Reduced Redundancy)
Owner Owner of the bucket

Check S3 objects

You can check for the presence of S3 objects at the bucket level and set thresholds for the same. Be notified if the defined object is missing from the S3 bucket.

While entering the value for the object, please make sure to enter the object name along with the prefix.
For example, if the object endpoint is https://s3-region.amazonaws.com/bucket-name/prefix-name/object.txt, provide the value as: prefix-name/object.txt

Monitor the S3 object

Amazon S3 object monitoring

Use Site24x7's prebuilt API monitoring functionality to run periodic checks on S3 object URLs to ensure it is functionality. At the configured interval, Site24x7 sends out HTTP requests to the object URL from each of the configured location and validates the result.

Pre requisites

  • The Amazon S3 monitoring integration must be enabled.
  • The state of the S3 bucket monitor should be active.

Add S3 object monitor

This section describes how to discover, add and configure checks for you object API endpoints:

  • In the Site24x7 web console, select AWS > "Monitored AWS account"
  • From the drop down menu, select Add S3 Endpoint

Discover

This sections describes how you can discover specific object URL endpoints from your S3 bucket:

  • Choose an S3 bucket from the drop down (multiple selection is not possible.)
  • Type the proper Regex to match the object URLs in the selected S3 bucket. Append .* as a suffix or as a prefix to your input string to only opt-in specific object URLs from the selected bucket. For example, you can discover only the objects in a particular path or directory using a Regex pattern like (images/.*) or you can discover objects of a certain type using a Regex pattern like (.png.*).
  • Next click on, Discover Objects

Select bucket and type in regex pattern

Select

All the object URLs matching your Regex pattern would get listed below. Here, you can select the endpoints that you wish to monitor by clicking on the checkbox and choosing Add Selected Objects

Select object URL endpoints

Configure

To add the S3 object endpoint monitor, you need to specify values for the checks configuration.

  • Display Name: Provide an appropriate name for the check
  • S3 Object URL: The selected S3 object endpoint URLs would be listed here. You can remove an endpoint form the list by clicking, deselecting and saving the form.
  • Check frequency: Select any option from 1, 5, 10, 20, 25, 30(minutes) or 1, 2 (hours). For example if you choose 1 minute, each selected geographic location would try to reach your endpoint every 1 minute.
  • Connection timeout: Specify a connection time between 1-45 seconds. Checks that do not generate a response within the period will be termed as failures.
  • Prefer IPv6: If you are monitoring dual stack object endpoints, move the toggle to Yes.
  • Monitoring locations: Create a new location profile or select or edit an exiting profile. The endpoint would receive requests from the applicable locations. Learn more .
  • Monitor Groups: Logically group your monitors for better management. Create a new monitor group or add the end point monitor to an exiting monitor group by clicking on the
  • Dependent on monitor: Alerts to the S3 object monitor will be suppressed based on the DOWN status of your dependent resource.

Configure endpoint checks

Specify the following details for HTTP configuration:

  • HTTP Method: Specify the HTTP method to be used, you can choose between GET or HEAD
  • User Agent: Type in a user agent if you want to emulate a specific browser.

Configure HTTP checks

Specify the following details for response validation:

  • Should contain string(s): Fill in a string whose presence in the response indicates failure
  • Should not contain string(s): Fill in a string whose non presence in the response indicates failure
  • Case sensitive: Switch the toggle to Yes if the response is case sensitive
  • Should match regular expression: Type in a Regex pattern whose presence in response indicates failure.

"Configure

Specify the following details to setup configuration profiles:

  • Threshold and Availability: Configure values for each supported attribute and get alerted when the metric data point crosses the value you have defined. You can either create a new threshold profile or select/edit the default threshold profile
  • Notification Profile: Configure when and who needs to be notifed in case of threshold violation or failure. You can either create a new notification profile or select/edit an existing profile. Refer here to create a customized notification profile.
  • User Alert Group: Select the Alert Group. To add multiple users in a group, refer here
  • Tags: User-defined key-value pairs associated with the object would get discovered and automatically assigned.
  • IT Automation: You can automatically execute operational tasks on AWS resource in response to alert events. You can create an automation profile and associate to the object endpoint monitor and execute it whenever the monitor goes to down/trouble/up/any status change or whenever an attribute value changes relative to a threshold.
  • Third Party Integration: You can integrate the monitor with a third-party alerting service to facilitate improved incident management. If you haven't setup any integrations yet, navigate to ”Admin > Third Party Integration” to create one. Learn more.
  • Click Save.

“configure

Was this document helpful?
Thanks for taking the time to share your feedback. We’ll use your feedback to improve our online help resources.