Collecting S3 logs using the Lambda Function

S3 Buckets acts as scalable containers in which large volumes of data can be stored. Site24x7 uses the Lambda Function to look for new logs added in the S3 Buckets and sends it to Site24x7 for indexing. Learn more about log management with Site24x7.

You can also configure your logs to be collected from S3 buckets using SQS. To avoid the overhead of configuring SQS permissions, you can use Lambda Functions to collect your logs as described below.

Define the Log Type:

A Log Type is a clear definition of the format in which an application writes logs. Different applications (such as IIS, Cassandra, Apache, MySQL) may write logs in different formats. Defining them as Log Types groups logs from different applications to simplify access and assist in efficient searching. Once you define a Log Type for your logs stored in your S3 bucket, list it under a Log Profile and start managing your logs by performing search queries. Create a custom log type.

Create a Log Profile

A Log Profile enables you to associate log types to a particular log source.

To create a Log Profile, navigate to Admin > AppLogs > Log Profile > Add Log Profile, and follow the instructions below:

1. Profile Name: Enter a name for your Log Profile.
2. Log Type: Choose the Log Type of the S3 logs you would like to associate with this profile.
3. Log Source: Choose Amazon Lambda.
4. Timezone: Select a timezone for your logs.
5. Click Save.
6. Configure the Lambda function as described here.

Configure the Lambda Function

• Choose Lambda from the Services drop-down list, and choose Create Function. Select Author from scratch, define a name for the function, and choose Python 3.7 as the Runtime. 

• Permissions: You can choose an existing IAM role or create a new role from the AWS Policy Template. From the Policy Template drop-down select Amazon S3 Object Read-only permission, and enter a role name. You also have the option to create a new user role and extend permission to other services as well.
• Add triggers: Scroll down to choose S3 Bucket. Any log file added to the S3 bucket will be sent to Site24x7 by the Lambda Function. 
• Configure Triggers
  • Bucket:Enter the name of the S3 bucket from which logs will be collected.
  • Event type: Choose All object create events.
  • Click Add.
  In the window that opens, click on the Lambda Function as shown:
• Scroll to the editor, and place the code provided in the link below:

• After entering the code, navigate to the Site24x7 web client, select Admin > Applogs > Log Profile, then select the created Log Profile, and copy the code that appears on the screen as the input for the variable logTypeConfig under the field Environment variables. 
  

Related log types

• CloudFront logs
• CloudTrail logs
• S3 logs