OCI OpenSearch monitoring
Oracle Cloud Infrastructure (OCI) Search with OpenSearch is a managed service that provides OpenSearch clusters, OpenSearch Dashboards, and ingest pipelines for search, log analytics, and observability workloads.
Site24x7 integrates with OCI Search with OpenSearch from a single console to provide visibility into cluster capacity, individual node performance, and ingest pipeline health. The integration provides the following monitors:
- OpenSearch Cluster: Monitors cluster-level performance and capacity, including CPU utilization, JVM memory pressure, storage consumption, indexing and search rates, and indexing and search latency.
- OpenSearch Node: Monitors individual nodes in an OpenSearch cluster, enabling you to isolate resource usage and performance issues affecting a specific node.
- OpenSearch Pipeline: Monitors ingest pipelines, including OpenSearch sink performance, Kafka buffer activity, and source-specific metrics for HTTP, Kafka, OCI Object Storage, and OpenTelemetry metrics, traces, and logs.
Overview
Site24x7's OCI OpenSearch integration provides monitoring at the cluster, node, and pipeline levels. You can track cluster capacity, node resource usage, search and indexing performance, and ingest pipeline activity from the Site24x7 console.
The integration helps identify high JVM memory pressure, increased CPU utilization, storage consumption, search or indexing latency, pipeline errors, dropped records, and source-specific ingest failures.
Use case
An organization uses OCI OpenSearch for centralized log analytics and observability. Application telemetry is sent through OpenSearch ingest pipelines and written to an OpenSearch cluster. Multiple nodes handle search and indexing workloads while different pipeline sources process HTTP requests, Kafka records, OCI Object Storage data, and OpenTelemetry telemetry.
With Site24x7's OCI OpenSearch monitoring, the team can monitor CPU utilization and JVM memory pressure at both cluster and node levels. Indexing and search latency metrics help identify performance degradation, while current cluster storage provides visibility into capacity usage.
Pipeline monitoring helps identify OpenSearch document errors, bulk request failures, dropped records, request failures, parsing errors, and Kafka buffer latency. This allows the team to identify performance and ingest issues before they affect search availability or result in data loss.
Benefits of Site24x7’s OCI OpenSearch integration
Site24x7's OCI OpenSearch integration provides these benefits:
- Cluster capacity monitoring: Track CPU utilization, JVM memory pressure, and current cluster storage against the provisioned capacity.
- Per-node visibility: Monitor resource usage and performance for individual OpenSearch nodes.
- Search and indexing performance: Monitor indexing and search rates along with mean and maximum latency.
- Ingest pipeline monitoring: Track documents written to the OpenSearch sink, bulk request errors, document errors, and source-specific pipeline metrics.
- Source-specific monitoring: Monitor pipelines using HTTP, Kafka, OCI Object Storage, or OpenTelemetry metrics, traces, and logs.
- Error detection: Identify deserialization errors, parsing failures, dropped records, rejected requests, timeouts, and authentication failures.
- Configuration visibility: View cluster, node, and pipeline configuration details from the Site24x7 console.
- Centralized monitoring: Monitor OpenSearch clusters, nodes, and pipelines from a single console.
Setup and configuration
To get started with OCI OpenSearch monitoring, complete these setup steps:
- Site24x7 uses cross-tenancy access to monitor your OCI resources using the Site24x7 tenancy user. Create the required OCI policy to allow Site24x7 to view your resources.
- Log in to your Site24x7 account and navigate to Cloud > OCI > Integrate OCI Monitor.
- On the Integrate OCI Monitor page, select OCI OpenSearch from the Services to be discovered list.
Permissions
Ensure that Site24x7 receives the following permissions to monitor OCI OpenSearch:
- read opensearch clusters
- read opensearch cluster pipelines
- read opensearch-family
- read metrics
Polling frequency
Site24x7 queries OCI service-level APIs according to the configured polling frequency, which can range from once a minute to once a day, to collect metrics from OCI OpenSearch monitors.
Supported metrics
The supported metrics for each OCI OpenSearch monitor are given below.
OCI OpenSearch Cluster
| Metric name | Description | Statistics | Unit |
|---|---|---|---|
| CPU Utilization | Percentage of CPU capacity consumed across the cluster's data nodes. | Average | Percent |
| JVM Memory Pressure | Peak percentage of the Java heap in use by the OpenSearch JVM. | Maximum | Percent |
| Current Cluster Storage | Storage currently consumed by cluster data. | Maximum | Bytes |
| Indexing Rate | Number of documents indexed by the cluster during the polling interval. | Sum | Count |
| Search Rate | Number of search requests served by the cluster during the polling interval. | Sum | Count |
| Indexing Latency (Max) | Longest time taken to index a document during the polling interval. | Maximum | Milliseconds |
| Indexing Latency (Mean) | Average time taken to index a document during the polling interval. | Average | Milliseconds |
| Search Latency (Mean) | Average time taken to complete a search request. | Average | Milliseconds |
| Search Latency (Max) | Longest time taken to complete a search request during the polling interval. | Maximum | Milliseconds |
Note: Indexing Latency and Search Latency are available as both mean and maximum values. The maximum value can be used to identify tail latency that may not be visible through the mean value.
OCI OpenSearch Node
| Metric name | Description | Statistics | Unit |
|---|---|---|---|
| CPU Utilization | Percentage of CPU capacity consumed by the individual node. | Average | Percent |
| JVM Memory Pressure | Peak percentage of the Java heap in use by the node's OpenSearch JVM. | Maximum | Percent |
| Current Cluster Storage | Storage currently consumed by the data held on this node. | Maximum | Bytes |
| Indexing Rate | Number of documents indexed by this node during the polling interval. | Sum | Count |
| Search Rate | Number of search requests served by this node during the polling interval. | Sum | Count |
| Indexing Latency (Max) | Longest time taken by this node to index a document. | Maximum | Milliseconds |
| Indexing Latency (Mean) | Average time taken by this node to index a document. | Average | Milliseconds |
| Search Latency (Mean) | Average time taken by this node to complete a search request. | Average | Milliseconds |
| Search Latency (Max) | Longest time taken by this node to complete a search request. | Maximum | Milliseconds |
OCI OpenSearch Pipeline
| Metric name | Description | Statistics | Unit |
|---|---|---|---|
| OpenSearch Bulk Request Errors | Number of bulk write requests to the OpenSearch sink that failed. | Sum | Count |
| OpenSearch Documents Success | Number of documents successfully written to the OpenSearch sink. | Sum | Count |
| OpenSearch Document Errors | Number of documents rejected by the OpenSearch sink. | Sum | Count |
| HTTP Requests Received | Number of ingest requests received on the HTTP source. | Sum | Count |
| HTTP Success Requests | Number of HTTP source requests processed successfully. | Sum | Count |
| HTTP Bad Requests | Number of HTTP source requests rejected as malformed. | Sum | Count |
| HTTP Request Timeout | Number of HTTP source requests that timed out. | Sum | Count |
| HTTP Requests Too Large | Number of HTTP source requests rejected because the payload exceeded the permitted size. | Sum | Count |
| HTTP Internal Server Error | Number of HTTP source requests that failed with a server side error. | Sum | Count |
| HTTP Requests Rejected | Number of HTTP source requests rejected because the pipeline buffer was full. | Sum | Count |
| Kafka Write Number Of Records Sent | Number of records written by the pipeline to the Kafka sink. | Sum | Count |
| Kafka Write Number Of Record Processing Errors | Number of records that failed while being written to the Kafka sink. | Sum | Count |
| Kafka Read Number Of Records Consumed | Number of records consumed by the pipeline from the Kafka source. | Sum | Count |
| Kafka Buffer Read Latency Average | Average time taken to read a batch of records from the Kafka buffer. | Average | Milliseconds |
| Kafka Read Number Of Deserialization Errors | Number of Kafka records that could not be deserialized. | Sum | Count |
| Kafka Read Number Of Poll Auth Errors | Number of authentication failures encountered while polling the Kafka source. | Sum | Count |
Object Storage Source Pipeline
| Metric name | Description | Statistics | Unit |
|---|---|---|---|
| OpenSearch Bulk Request Errors | Number of bulk write requests to the OpenSearch sink that failed. | Sum | Count |
| OpenSearch Documents Success | Number of documents successfully written to the OpenSearch sink. | Sum | Count |
| OpenSearch Document Errors | Number of documents rejected by the OpenSearch sink. | Sum | Count |
| OCI Object Failed | Number of OCI Object Storage objects the pipeline failed to process. | Sum | Count |
| OCI Object Succeeded | Number of OCI Object Storage objects the pipeline processed successfully. | Sum | Count |
Kafka Source Pipeline
| Metric name | Description | Statistics | Unit |
|---|---|---|---|
| OpenSearch Bulk Request Errors | Number of bulk write requests to the OpenSearch sink that failed. | Sum | Count |
| OpenSearch Documents Success | Number of documents successfully written to the OpenSearch sink. | Sum | Count |
| OpenSearch Document Errors | Number of documents rejected by the OpenSearch sink. | Sum | Count |
| Kafka Topic Number Of Records Consumed | Number of records consumed from the subscribed Kafka topic. | Sum | Count |
| Kafka Topic Number Of Records Failed To Parse | Number of Kafka topic records that could not be parsed by the pipeline. | Sum | Count |
OTel Metrics Source Pipeline
| Metric name | Description | Statistics | Unit |
|---|---|---|---|
| OpenSearch Bulk Request Errors | Number of bulk write requests to the OpenSearch sink that failed. | Sum | Count |
| OpenSearch Documents Success | Number of documents successfully written to the OpenSearch sink. | Sum | Count |
| OpenSearch Document Errors | Number of documents rejected by the OpenSearch sink. | Sum | Count |
| OTEL Metrics Request Timeouts | Number of OpenTelemetry metrics requests that timed out. | Sum | Count |
| OTEL Metrics Requests Received | Number of OpenTelemetry metrics requests received by the pipeline. | Sum | Count |
| OTEL Metrics Success Requests | Number of OpenTelemetry metrics requests processed successfully. | Sum | Count |
| OTEL Metrics Records Created | Number of metric records created by the OpenTelemetry metrics source. | Sum | Count |
| OTEL Metrics Records Dropped | Number of metric records dropped by the OpenTelemetry metrics source. | Sum | Count |
| Kafka Write Number Of Records Sent | Number of records written by the pipeline to the Kafka sink. | Sum | Count |
| Kafka Write Number Of Record Processing Errors | Number of records that failed while being written to the Kafka sink. | Sum | Count |
| Kafka Read Number Of Records Consumed | Number of records consumed by the pipeline from the Kafka source. | Sum | Count |
| Kafka Buffer Read Latency Average | Average time taken to read a batch of records from the Kafka buffer. | Average | Milliseconds |
| Kafka Read Number Of Deserialization Errors | Number of Kafka records that could not be deserialized. | Sum | Count |
| Kafka Read Number Of Poll Auth Errors | Number of authentication failures encountered while polling the Kafka source. | Sum | Count |
OTel Traces Source Pipeline
| Metric name | Description | Statistics | Unit |
|---|---|---|---|
| OpenSearch Bulk Request Errors | Number of bulk write requests to the OpenSearch sink that failed. | Sum | Count |
| OpenSearch Documents Success | Number of documents successfully written to the OpenSearch sink. | Sum | Count |
| OpenSearch Document Errors | Number of documents rejected by the OpenSearch sink. | Sum | Count |
| OTEL Trace Request Timeouts | Number of OpenTelemetry trace requests that timed out. | Sum | Count |
| OTEL Trace Requests Received | Number of OpenTelemetry trace requests received by the pipeline. | Sum | Count |
| OTEL Trace Bad Requests | Number of OpenTelemetry trace requests rejected as malformed. | Sum | Count |
| OTEL Trace Requests Too Large | Number of OpenTelemetry trace requests rejected because the payload exceeded the permitted size. | Sum | Count |
| OTEL Trace Internal Server Error | Number of OpenTelemetry trace requests that failed with a server side error. | Sum | Count |
| OTEL Trace Success Requests | Number of OpenTelemetry trace requests processed successfully. | Sum | Count |
| Kafka Write Number Of Records Sent | Number of records written by the pipeline to the Kafka sink. | Sum | Count |
| Kafka Write Number Of Record Processing Errors | Number of records that failed while being written to the Kafka sink. | Sum | Count |
| Kafka Read Number Of Records Consumed | Number of records consumed by the pipeline from the Kafka source. | Sum | Count |
| Kafka Buffer Read Latency Average | Average time taken to read a batch of records from the Kafka buffer. | Average | Milliseconds |
| Kafka Read Number Of Deserialization Errors | Number of Kafka records that could not be deserialized. | Sum | Count |
| Kafka Read Number Of Poll Auth Errors | Number of authentication failures encountered while polling the Kafka source. | Sum | Count |
OTel Logs Source Pipeline
| Metric name | Description | Statistics | Unit |
|---|---|---|---|
| OpenSearch Bulk Request Errors | Number of bulk write requests to the OpenSearch sink that failed. | Sum | Count |
| OpenSearch Documents Success | Number of documents successfully written to the OpenSearch sink. | Sum | Count |
| OpenSearch Document Errors | Number of documents rejected by the OpenSearch sink. | Sum | Count |
| OTEL Logs Request Timeouts | Number of OpenTelemetry logs requests that timed out. | Sum | Count |
| OTEL Logs Requests Received | Number of OpenTelemetry logs requests received by the pipeline. | Sum | Count |
| OTEL Logs Bad Requests | Number of OpenTelemetry logs requests rejected as malformed. | Sum | Count |
| OTEL Logs Requests Too Large | Number of OpenTelemetry logs requests rejected because the payload exceeded the permitted size. | Sum | Count |
| OTEL Logs Internal Server Error | Number of OpenTelemetry logs requests that failed with a server side error. | Sum | Count |
| OTEL Logs Success Requests | Number of OpenTelemetry logs requests processed successfully. | Sum | Count |
| Kafka Write Number Of Records Sent | Number of records written by the pipeline to the Kafka sink. | Sum | Count |
| Kafka Write Number Of Record Processing Errors | Number of records that failed while being written to the Kafka sink. | Sum | Count |
| Kafka Read Number Of Records Consumed | Number of records consumed by the pipeline from the Kafka source. | Sum | Count |
| Kafka Buffer Read Latency Average | Average time taken to read a batch of records from the Kafka buffer. | Average | Milliseconds |
| Kafka Read Number Of Deserialization Errors | Number of Kafka records that could not be deserialized. | Sum | Count |
| Kafka Read Number Of Poll Auth Errors | Number of authentication failures encountered while polling the Kafka source. | Sum | Count |
Threshold configuration
To configure thresholds for OCI OpenSearch monitors:
- Log in to your Site24x7 account and navigate to Admin > Configuration Profiles > Threshold and Availability.
- Click Add Threshold Profile.
- Select the applicable monitor type from the Monitor Type drop-down menu.
- Provide an appropriate name in the Display Name field.
- The supported metrics are displayed in the Threshold Configuration section. You can set threshold values for the supported metrics.
- Click Save.
Capacity planning
Capacity planning is available for OCI OpenSearch Cluster monitors and helps you understand how cluster resources are being utilized over time. Use cluster-level performance metrics to identify sustained increases in CPU utilization, JVM memory pressure, storage consumption, and indexing and search rates. Reviewing these trends can help you determine when a cluster is approaching its current resource limits and plan capacity changes before performance is affected.
For example, if indexing rate has been growing steadily over several weeks, capacity planning data can show whether current node count and storage are sufficient for projected growth, allowing you to scale the cluster proactively rather than reactively.
Monitor Groups
Monitor Groups can be used to view the overall status of related OCI OpenSearch monitors and streamline day-to-day monitoring. You can add or remove monitors from groups as your OpenSearch environment changes.
For example, you can create separate monitor groups for production and development OpenSearch environments, or group an OpenSearch Cluster with its associated Node and Pipeline monitors for a particular application. This makes it easier to identify issues affecting a specific environment or application without reviewing individual monitors separately.
You can organize your OCI OpenSearch monitors into monitor groups based on application, environment, business unit, region, or pipeline source type. Grouping related cluster, node, and pipeline monitors provides a consolidated view of their availability and performance from a single location.
Licensing
The OCI OpenSearch monitors use the following licensing model:
- OpenSearch Cluster: Each monitor utilizes one basic monitor license.
- OpenSearch Pipeline: Each monitor utilizes one basic monitor license.
- OpenSearch Node: Free monitor.
Viewing OCI OpenSearch data
To monitor your OCI OpenSearch environment, log in to the Site24x7 console and navigate to Cloud > OCI > OpenSearch.
Monitor data
The monitor data for each OCI OpenSearch monitor is given below.
OpenSearch Cluster
Summary
The Summary tab displays the cluster's status, availability, number of downtimes, downtime duration, and a heat map. It provides charts for CPU utilization, JVM memory pressure, current cluster storage, indexing rate, indexing latency, search rate, and search latency. Indexing and search latency are shown as mean and maximum values. The tab also includes the updates list and downtime details.
OpenSearch Nodes
The OpenSearch Nodes tab displays a list of the child Node monitors associated with the cluster. These monitors are identified by the child type OCI_OPENSEARCH_NODE. Selecting a node opens its corresponding monitor page.
Configuration
The Configuration tab provides details about the OpenSearch cluster through four cards: General, Security, Network, and Maintenance. The General card displays the OCID, state, creation and update times, the number and shapes of leader, data, search, and dashboard nodes, total storage, and software version. The Security card displays the master username, security mode, and API and dashboard certificate modes. The Network card provides details such as the VCN, subnet, load balancer type, minimum and maximum bandwidth, API endpoint and private IP, and Dashboards endpoint and private IP. The Maintenance card displays the maintenance state, notification email addresses, and maintenance start and end times.
Backup Details
The Backup Details tab displays a paginated table of cluster backups. The table is loaded live and follows the next_token until the page limit is reached. It includes the backup name, state, type, backup size in GB, creation time, and expiration time.
Pipeline
The Pipeline tab displays a list of the child Pipeline monitors associated with the cluster.
Cross Cluster Connections
The Cross Cluster Connections tab provides an Outbound/Inbound toggle to view the cluster's cross cluster connections. The Outbound view displays the connection type, remote cluster OCID, connection name, mode, ping schedule, and skip unavailable setting. The Inbound view displays the connection type and OCID.
OpenSearch Node
Summary
The Summary tab displays the node's status, availability, number of downtimes, and downtime duration, along with a heat map. It provides the same seven performance charts available in the cluster Summary, but the data is specific to the selected node. This helps identify an individual overloaded or unhealthy node.
Configuration
The Configuration tab displays an Inventory card containing the cluster ID, node name, node type, and region. The node type identifies whether the node is a data, master, search, or dashboard node.
OpenSearch Pipeline
The Summary tab for the OpenSearch Pipeline monitor varies based on the pipeline's source type. It displays the pipeline's status, availability, and heat map. Two charts are available for all pipeline source types: sink errors, which include bulk request and document errors, and successfully written documents. Additional charts are displayed based on the configured source type.
For HTTP pipelines, the Summary tab displays request metrics for received, successful, and bad requests, along with failure metrics for timeouts, oversized requests, server errors, and rejected requests.
For Kafka pipelines, the Summary tab displays topic records consumed and failed to parse, buffer read latency, buffer write metrics for records sent and processing errors, and buffer read metrics for records consumed, deserialization errors, and poll authentication errors.
For Object Storage pipelines, the Summary tab displays the number of objects processed successfully and the number of objects that failed.
For OTel Metrics pipelines, the Summary tab displays incoming request metrics for received, successful, and timed out requests, along with record metrics for records created and dropped.
For OTel Traces and OTel Logs pipelines, the Summary tab displays incoming request metrics for received and successful requests, along with error metrics for timeouts, bad requests, oversized requests, and server errors.
Configuration
The Configuration tab provides pipeline details through three cards: General, Hardware, and Network. The General card displays the OCID, state, pipeline mode, source type, parent cluster ID, and creation and update times. The Hardware card displays the OCPU count, memory in GB, node count, and node shape. The Network card displays the VCN, subnet, NSG, pipeline FQDN, and private IP. This card is available only for HTTP and the three OpenTelemetry source types that receive data through an endpoint. It is hidden for Kafka and Object Storage pipelines.