Performance
Performance shows latency distributions per endpoint, so a slow tail is visible instead of hidden behind an average.
Why distributions, not averages
If 95% of requests take 40 ms and 5% take 3 seconds, the average is about 190 ms and describes no actual request. Meanwhile one user in twenty is watching a spinner.
Averages hide exactly the requests you need to find. Read the percentiles.
| Percentile | What it tells you |
|---|---|
| p50 | The typical request. Good for trend lines, bad for complaints |
| p95 | Where the pain starts. One request in twenty is at least this slow |
| p99 | Your worst regular experience. On a busy endpoint this is thousands of requests a day. It's in the overview panel |
Reading it
- Sort by the tail, not the mean
The endpoint with the worst p99 is where your users are suffering, even if its p50 looks fine.
- Weigh it by traffic
A slow endpoint nobody calls is a curiosity. A slightly-slow endpoint on your hot path is the actual problem.
- Open a trace at that percentile
Aggregates tell you where to look. A trace tells you why.
Latency you cause vs latency you observe
The numbers here are what your service did. A client on a bad connection experiences more, and no server-side percentile will show you that.
If your SDK retries a failed request, the caller waits for every attempt plus backoff, while your server sees several fast failures. A maxAttempts of 5 with no maxBackoffMs can leave a user waiting far longer than any percentile here suggests.
What each row shows
| Column | Meaning |
|---|---|
| Throughput | How many requests |
| Errors | How many failed |
| Average | The mean, for reference |
| p50, p75, p95 | The latency percentiles |
| Total time | Where your service spends most of its time overall |
Sort by Total time to find what's worth optimizing. A fast endpoint called a million times can cost more than a slow one called ten times.
Suspected N+1 queries
Suspected N+1 queries lists a statement issued over and over within one trace, which usually means a loop running a query per item. It needs spans, so add a backend runtime.
Service level objectives
Set a target for a transaction on its row. You can set either or both:
| Target | Meaning |
|---|---|
| p95 latency ceiling | In milliseconds |
| Max error rate | As a percentage |
The SLO column then shows when a target is breached, and how much of the error budget has been used. Above 100%, you've spent more than your target allows, and at the extreme it reads budget spent.
SLOs show here and don't notify anyone. To be told, add an alert.
Watching a deploy
Compare a percentile before and after a release to catch a regression that didn't error, just got slower. See Releases.
From the CLI
octri monitoring performance prints the slowest transactions. octri monitoring slos sets targets and checks them, and --fail stops a deploy job on a breach.