Configure LLM Observability data collection

Site24x7's LLM Observability uses OpenTelemetry to collect telemetry generated by your LLM applications. Instrument your application using an OpenTelemetry-compatible instrumentation library or framework and export the generated telemetry to the Site24x7 OpenTelemetry Protocol (OTLP) endpoint.

Once Site24x7 receives the telemetry, you can analyze LLM performance, token consumption, costs, errors, prompts, traces, and other request-level information under APM > LLM Observability.

Prerequisites

Before configuring LLM Observability, ensure that:

• Your LLM application is instrumented using an OpenTelemetry-compatible instrumentation library or framework.
• You have access to your Site24x7 account.
• Your application can communicate with the Site24x7 OpenTelemetry endpoint.
• The required LLM telemetry is exported using OTLP.

Obtain the Site24x7 OTLP endpoint and API key

To send LLM telemetry to Site24x7:

1. Log in to the Site24x7 web client.
2. Navigate to APM > OpenTelemetry.
3. Copy the OTLP Endpoint and API Key displayed on the Add OpenTelemetry Agent page.

Instrument your LLM application

Instrument your LLM application using an OpenTelemetry-compatible library or framework that can generate the telemetry required for LLM Observability. You can configure your instrumentation to export telemetry either directly to the Site24x7 OTLP endpoint or through an OpenTelemetry Collector, which then forwards the telemetry to Site24x7.

Using Traceloop and OpenLLMetry

If you use Traceloop and OpenLLMetry to instrument your LLM application, first configure the appropriate Traceloop SDK for your application language or framework.

Refer to the corresponding Traceloop documentation:

• Python
• Node.js and TypeScript
• Next.js
• Go
• Ruby

After completing the instrumentation steps for your application, configure Traceloop to export the generated OpenTelemetry data to Site24x7.

Configure Traceloop to export data to Site24x7

Set the following environment variables in the environment where your LLM application runs:

export TRACELOOP_BASE_URL="<SITE24X7_OTLP_ENDPOINT>"
export TRACELOOP_HEADERS="api-key=<SITE24X7_API_KEY>"

• TRACELOOP_BASE_URL specifies the OpenTelemetry endpoint to which Traceloop exports the telemetry.
• TRACELOOP_HEADERS provides the authentication header required by Site24x7.

Initialize Traceloop

Initialize the Traceloop SDK according to the language or framework used by your application. Refer to the respective Traceloop agent documentation listed above for the appropriate initialization procedure.

Start sending LLM telemetry

Start or restart your application after completing the instrumentation and export configuration.

When the instrumented application makes LLM calls, the OpenTelemetry telemetry generated for those interactions is exported to the configured Site24x7 OTLP endpoint.

Depending on the instrumentation and the information available for each request, the collected data can include details such as the model, provider, request duration, input and output token usage, cost, request time, prompts, and traces.

Verify the data in Site24x7

After your application generates LLM requests:

1. Log in to the Site24x7 web client.
2. Navigate to APM > LLM Observability.
3. Select the required time period.
4. Check the Dashboard tab to verify that LLM requests and metrics are being received.
5. Open the Prompt Analysis tab to verify individual LLM interactions.

The Dashboard tab provides aggregated information about LLM performance, utilization, token consumption, costs, and errors, while the Prompt Analysis tab provides request-level visibility into individual LLM interactions. This verification flow already exists in your current document and can largely remain unchanged.

Troubleshooting

If LLM data is not displayed in Site24x7, verify that:

• Your LLM application is properly instrumented.
• The application is exporting telemetry using OTLP.
• The configured OTLP endpoint matches the Site24x7 endpoint.
• The Site24x7 API key is correctly provided in the api-key header.
• Your application or OpenTelemetry Collector can reach the Site24x7 endpoint.
• The application has generated LLM requests after the configuration was applied.