Upload logs from Fluent Bit

Fluent Bit is a lightweight log processor and forwarder for Linux, Windows, and containers. The Site24x7 output plugin lets Fluent Bit send anything it already collects—application logs, container logs, syslog, and other supported sources—directly to the Site24x7 log management.

If Fluent Bit is already deployed in your environment for log collection, you can integrate it with Site24x7 by adding the Site24x7 output plugin to your existing Fluent Bit configuration. This eliminates the need to install an additional log shipping agent.

Log parsing is managed through the Log Type configured in Site24x7, so you don't need to create or maintain parsing rules within Fluent Bit.

Use case

When a team runs Fluent Bit as a DaemonSet across a Kubernetes cluster to collect container logs. Rather than deploying a separate log shipper just for the monitoring tool, they add the Site24x7 output plugin to the same Fluent Bit pipeline, which is already running on every node. The collected logs become available in AppLogs within minutes, eliminating the need for additional infrastructure or application-level changes.

Benefits of Fluent Bit

The Fluent Bit provides the following log ingestion benefits:

• No additional shipper: Integrate with Site24x7 and use your existing Fluent Bit deployment without installing or maintaining another agent.
• Multiple log formats: Forward JSON, plain-text, and multi-line logs, including stack traces.
• Centralized log parsing: Define and manage parsing rules in the Site24x7 Log Type instead of configuring parsers in Fluent Bit.
• Built-in data protection: Apply masking, hashing, and filtering rules configured in the Log Profile before logs are ingested into Site24x7.
• Reliable log delivery: Fluent Bit automatically retries failed uploads to help ensure logs are delivered successfully.

Prerequisites

Before you begin, make sure you have:

1. A Site24x7 account with AppLogs enabled.
2. Fluent Bit 1.9 or later already installed and running with Site24x7 output plugin support. Learn how to build the plugin from source for other platforms.

Collecting logs with Fluent Bit

Log collection with Fluent Bit involves the following steps:

1. Fluent Bit inputs (such as tail) read log lines from your applications and services.
2. The Site24x7 output plugin, downloaded from the fluent-bit-plugin-site24x7 repository, parses each record according to the Log Type configured in Site24x7. No regex or parser configuration is needed in Fluent Bit itself.
3. The plugin batches the events, compresses them, and uploads them to the Site24x7 AppLogs endpoint.

Creating a Log Profile

To create a Log Profile, follow the steps below:

1. Log in to your Site24x7 account.
2. Navigate to Admin > AppLogs > Log Profile > Add Log Profile, and enter the following details.
   1. Profile Name: Enter a name for your Log Profile.
   2. Log Type: Choose an existing Log Type that matches your application's log fields, or create a custom Log Type sample of your log lines. Both JSON logs and plain-text logs are supported.
   3. Log Source: Select Log Collectors, from the drop-down menu.
   4. Log Time Zone: Select the time zone of your logs from the drop-down menu.
3. Click Save.

Open the saved profile, and copy the logTypeConfig value shown on screen. This single string carries everything the plugin needs: the upload endpoint, log type, parsing rules, and date format.

Installing the Site24x7 output plugin

To install the Site24x7 plugin, first you need to download the plugin for your platform from the release page:

• For Linux x86_64: out_site24x7-linux-amd64.so
• For Linux ARM64: out_site24x7-linux-arm64.so
• For Windows x86_64: out_site24x7-windows-amd64.dll

Linux

sudo mkdir -p /usr/lib/fluent-bit/plugins 
sudo cp out_site24x7-linux-amd64.so /usr/lib/fluent-bit/plugins/out_site24x7.so

Windows (PowerShell)

Copy-Item out_site24x7-windows-amd64.dll 'C:\fluent-bit\plugins\out_site24x7.dll'

Containers and Kubernetes (Dockerfile)

Extend the official image with one line

FROM fluent/fluent-bit:3.1
COPY out_site24x7-linux-amd64.so /fluent-bit/lib/out_site24x7.so

Then register the plugin in Fluent Bit's plugins.conf:

[PLUGINS]
    Path /usr/lib/fluent-bit/plugins/out_site24x7.so

Or pass it at startup with:

fluent-bit -e /usr/lib/fluent-bit/plugins/out_site24x7.so ...

Configuring Fluent Bit

Update the fluent-bit.conf file with the input and Site24x7 output configuration.

[INPUT]
    Name    tail
    Path    /var/log/app/*.log
    Tag     app.*
[OUTPUT]
    Name             site24x7
    Match            *
    log_type_config  ##logtypeconfig copied from yourlog profile
    Retry_Limit False

Restart Fluent Bit. Within a few moments, the logs appear in Site24x7 AppLogs, where you can search them using the configured Log Type.

The output section comprises the following parameters:

The Fluent Bit retries the failed uploads caused by transient issues. Configure the retry behavior using the standard Retry_Limit option. Set it to False to retry until the logs are successfully delivered.

How log parsing works

The Site24x7 output plugin uses the parsing rules defined in your Log Type, so no parsing configuration is required in Fluent Bit:

• JSON logs are mapped using the Log Type's field paths, including nested fields.
• Plain-text logs are parsed with the Log Type's pattern. Lines that don't start a new entry, for example stack trace lines, are automatically appended to the previous event's message.
• The plugin uses the log timestamp field configured in the Log Type. If no timestamp field is configured, the log collection time is used.
• Data processing such as field masking, hashing (SHA-256), derived fields, and filters configured in the Log Type are applied by the plugin before upload, so excluded events never leave your server.

How to collect multiple log types with one Fluent Bit

To forward different log files to different Log Types, configure a separate [OUTPUT] section for each Log Profile and use tags for routing:

[INPUT]
    Name  tail
    Path  /var/log/app/api.log
    Tag   api
[INPUT]
    Name  tail
    Path  /var/log/app/payments.log
    Tag   payments[OUTPUT]
    Name             site24x7
    Match            api
    log_type_config  ##logtypeconfig of the API log profile
[OUTPUT]
    Name             site24x7
    Match            payments
    log_type_config  ##logtypeconfig of the payments log profile 

Frequently asked questions (FAQs)

1. How to view plugin activity?
   Run Fluent Bit with-vfor verbose logs. All plugin messages are prefixed with[out_site24x7]. A successful upload logs uploaded N events ... Upload Id: .

2. Why does Fluent Bit show "tried to instance a plugin name that doesn't exist"?
   This usually means the shared library isn't loaded. Check the[PLUGINS]path in plugins.conf, or confirm you passed the-eflag at startup.

3. Why does the plugin fail to load with "cannot open shared object file"?
   This can happen when the plugin binary doesn't match your platform, for example an amd64 build on an ARM64 host. Download the correct artifact for your platform from the Releases page.

4. Why have uploads stopped with a log license limit or max upload limit error?
   This means your account's log license or the Log Type's upload limit was reached. Uploads pause for 10 minutes and resume automatically.

5. Why does the Fluent Bit plugin returns an "Invalid log type", "invalid account", or "API upload not enabled" error?
   This usually means thelog_type_configvalue doesn't match the account or data center it was copied from. Recopy it from the Log Profile, and restart Fluent Bit.

6. How to verify log ingestion in Site24x7?
   Go to AppLogs > AppLogs Search, and search for your Log Type. The logs associated with the selected Log Type will be displayed in the search results. Learn more.

7. What if none of this resolves the issue?
   If you continue to experience issues after the configuration, please contact support@site24x7.com with the Fluent Bit logs.

Related articles

• Manage logs collected via Fluentd and Logstash
• Supported log types
• Predefined alerts for log management