Forward logs to a Google Cloud Storage bucket

Google Cloud Storage is Google's object storage service. With this integration, you can route a continuous copy of AppLogs into a Google Cloud Storage bucket you own, so that data never leaves your GCP environment.

Key capabilities of GCP integration

This integration provides the following capabilities:

• Forwards AppLogs to a Google Cloud Storage bucket.
• Uses the service account of your existing GCP cloud monitor, so no separate credentials are required.
• Forwards logs as gzip-compressed objects in NDJSON, JSON, or CSV format.

Prerequisites

1. Make sure a GCP cloud monitor is already configured in Site24x7. Log forwarding uses the service account associated with this monitor.
   1. If you haven't configured a GCP project, add your GCP project under Admin > Inventory > Add Monitor > Google Cloud Platform.
2. Give the service account permission to create objects in the target Google Cloud Storage bucket, such as giving it the Storage Object Creator role. For example, roles/storage.objectCreator.
3. Make sure the target Google Cloud Storage bucket already exists. You can optionally specify a path prefix to organize the forwarded logs.

Configuring the GCP storage integration

1. Log in your account.
2. Navigate to Admin > Third-Party Integrations.
3. Click the plus icon + next to Log Forwarding to open the supported destination list.
4. Select GCP Storage Bucket and click Integrate Now.

This will open the integration page for the GCP storage destination. Enter the following details:

1. Basic Information: Provide the details of the destination to which the logs will be forwarded.
   • Name: Enter a label to identify this configuration.
   • Enable Forwarding: Toggle to Enabled to activate the configuration once saved.
2. Destinations: Choose where logs should be forwarded and configure destination credentials.
   • Destination Type: By default, GCP Storage Bucket is selected.
   • GCP Monitor: Select the existing GCP cloud monitor whose credentials are used to write to the bucket.
   • Bucket Name: Select or enter the destination Google Cloud Storage bucket. The bucket must already exist.
   • Path Prefix: Optionally, enter a virtual folder prefix prepended to every blob name.
   • File Format: Select the file format: NDJSON (default), JSON, or CSV. Learn more about the file formats
     • Click Test Connection to test the connection and confirm the bucket is reachable and writable.
3. Filtering: Restrict which logs are forwarded by log type and query.
   • Log Types: Optionally, you can select from more than 100+ default log types whose data you want to forward, and enter a Filter Query to forward only matching records. You can select more than one log type.
     • Click the X icon on the respective log type to remove the query.
4. Schedule and Retry: Control how often logs are forwarded and retry behavior on failure.
   • Forwarding Frequency: Select how often batches are written to the bucket. You can select from any one of the following:
     • Every 1 minute
     • Every 5 minutes
     • Every 15 minutes
     • Every 30 minutes
     • Every 60 minutes
   • Batch Size: Set the maximum number of records per object. The default is 1,000. Learn more about batching and delivery options.
   • Retry on Failure: Toggle to Enable to automatically retry failed deliveries.
   • Max Retries: Enter the number of maximum entries the forward can fail.
5. Click Save.
   
   

The forwarding begins with the next matching batch of incoming logs.

Managing the GCP storage bucket log forwarding configuration

1. Navigate to Admin > Third-Party Integrations > Log Forwarding.
2. Locate the configuration on the listing page. Clicking the configured forwarding entry to open the Edit Configuration page.
3. Click the hamburger icon on the right corner of each configured forwarding entry to perform the following actions:
   1. Click Disable forwarding to temporarily stop forwarding.
   2. Click Enable forwarding to resume forwarding.
   3. Click Delete to remove it permanently.
   

Related articles

• GCP VPC Flow Logs
• Log forwarding from Google Cloud Platform