Prerequisites
- ScaleOps Network Monitor DaemonSet installed and running (see Network Monitor DaemonSet)
API Observability Available in v1.29.0+
The API Observability feature in ScaleOps provides real-time visibility into the performance and reliability of your workload’s HTTP/HTTPS traffic. Powered by eBPF and running as a DaemonSet (scaleops-network-monitor), this feature enables passive monitoring of network traffic without requiring application modifications. The DaemonSet runs unprivileged with explicit Linux capabilities (BPF, PERFMON, SYS_PTRACE, NET_RAW, DAC_READ_SEARCH, CHECKPOINT_RESTORE, NET_ADMIN, SYS_ADMIN).
This observability tool helps you track critical API metrics, detect anomalies, and optimize performance by exposing insights into request rates, error rates, and latency at the URL level. The collected data also enhances ScaleOps workload detection, enabling more accurate rightsizing recommendations tailored to each workload’s actual runtime behavior.
Key Features
- Non-intrusive: Uses eBPF for kernel-level monitoring, requiring no code changes.
- Supports HTTP & HTTPS: Captures request metadata (path, method, status code, latency) from both encrypted and unencrypted traffic.
- Per-URL Metrics: Provides insights at the endpoint level.
- Protocol and Method Visibility: Surfaces the HTTP protocol version and request method for each endpoint.
- Per-Status-Code Error Breakdown: Distinguishes between 4xx and 5xx error classes in the error rate charts and table.
Metrics
The APIs tab displays three time-series charts and a detailed table. Metrics can be filtered by path and protocol. Table values represent the average over the selected view period.
Charts
- Requests (req/s): Shows total incoming requests per second, broken down into OK and Error traffic.
- Errors (err/s): Shows error request rate per second, broken down per HTTP status code (4xx and 5xx).
- Latency: Shows P50 and P95 response time percentiles over time.
Table columns
| Column | Description |
|---|---|
| Path | The request URL path. |
| Protocol | The protocol observed (HTTP or HTTPS). |
| Method | The HTTP request method (e.g., GET, POST). |
| Error Rate | The percentage of requests returning 4xx or 5xx status codes. |
| P50 | Median response time. |
| P95 | 95th-percentile response time, highlighting high-latency outliers. |
| RPS | Requests per second for the endpoint. |
How it works
- DaemonSet Deployment: eBPF probes capture API traffic on each node.
- Automatic Data Aggregation: Maps network traffic to workloads.
- Visualization: Metrics are displayed in the Workload Overview → APIs tab.
Performance & Scalability
The API Observability feature has been tested across various cluster scales to ensure minimal impact. While it operates efficiently using eBPF, enabling it may introduce up to a 5% increase in traffic latency, depending on requests patterns and volume.
Configuration
For additional configuration options (tolerations, socket paths, etc.), see the Helm Configuration Reference.