API Observability

Available in v1.29.0+

The API Observability feature in ScaleOps provides real-time visibility into the performance and reliability of your workload’s HTTP/HTTPS traffic. Powered by eBPF and running as a DaemonSet (scaleops-network-monitor), this feature enables passive monitoring of network traffic without requiring application modifications. The DaemonSet runs unprivileged with explicit Linux capabilities (BPF, PERFMON, SYS_PTRACE, NET_RAW, DAC_READ_SEARCH, CHECKPOINT_RESTORE, NET_ADMIN, SYS_ADMIN). This observability tool helps you track critical API metrics, detect anomalies, and optimize performance by exposing insights into request rates, error rates, and latency at the URL level. The collected data also enhances ScaleOps workload detection, enabling more accurate rightsizing recommendations tailored to each workload’s actual runtime behavior.

Key Features

• Non-intrusive: Uses eBPF for kernel-level monitoring, requiring no code changes.
• Supports HTTP & HTTPS: Captures request metadata (path, method, status code, latency) from both encrypted and unencrypted traffic.
• Per-URL Metrics: Provides insights at the endpoint level.
• Protocol and Method Visibility: Surfaces the HTTP protocol version and request method for each endpoint.
• Per-Status-Code Error Breakdown: Distinguishes between 4xx and 5xx error classes in the error rate charts and table.

Installation

API Observability requires the Network Monitor DaemonSet. All observability features are disabled by default.

If using a Helm command, add the following flag:

--set global.observability.enabled=true

If using a values file, add the following configuration:

global:
  observability:
    enabled: true

This master toggle enables all observability features (network cost monitoring and API observability). To enable only API Observability without network cost monitoring, use the sub-toggle:

global:
  observability:
    apiMonitoring:
      enabled: true

Example upgrade command

helm repo update scaleops
helm show crds scaleops/scaleops | kubectl apply --force -f -
helm get values scaleops -n scaleops-system -oyaml | \
helm upgrade scaleops scaleops/scaleops --set global.observability.enabled=true -n scaleops-system -f -

This will install the Network Monitor DaemonSet (scaleops-network-monitor) on your cluster. For additional configuration options (tolerations, socket paths, etc.), see the Helm Configuration Reference.

Metrics

The APIs tab displays three time-series charts and a detailed table. Metrics can be filtered by path and protocol. Table values represent the average over the selected view period.

Charts

• Requests (req/s): Shows total incoming requests per second, broken down into OK and Error traffic.
• Errors (err/s): Shows error request rate per second, broken down per HTTP status code (4xx and 5xx).
• Latency: Shows P50 and P95 response time percentiles over time.

Table columns

How it works

• DaemonSet Deployment: eBPF probes capture API traffic on each node.
• Automatic Data Aggregation: Maps network traffic to workloads.
• Visualization: Metrics are displayed in the Workload Overview → APIs tab.

Performance & Scalability

The API Observability feature has been tested across various cluster scales to ensure minimal impact. While it operates efficiently using eBPF, enabling it may introduce up to a 5% increase in traffic latency, depending on requests patterns and volume.

Configuration

For additional configuration options (tolerations, socket paths, etc.), see the Helm Configuration Reference.