feat(analytics): record API latency by endpoint and client by alanpeixinho · Pull Request #2156 · kernelci/dashboard

alanpeixinho · 2026-10-02T18:24:06Z

What it is

Time each finished /api/ request on dashboard_backend_request_latency_seconds, labeled by endpoint and client.
Stop observing those requests on the Django view-latency histogram. /admin/ and /health/ stay on it. Django status and exception counters stay as they are.
Point Total Calls by Endpoint at dashboard_backend_requests_by_client_total, summed by endpoint and client. Point Average Response Time by Endpoint at the histogram sum divided by count, and add p50 and p95 from its buckets. Endpoint Performance Summary uses the counter for calls and the histogram for total time and average.
Add a 4xx and 5xx by Endpoint panel on that same counter, summed by endpoint and status_class. A 200 does not appear.

The existing Client dropdown filters those panels. New series start at deploy. API views already on the Django histogram stop receiving samples.

Built on #2154. Merge that first.

Closes #2149
Closes #2150

How to test

From backend/:

poetry run pytest -n 0 -m unit kernelCI_app/tests/unitTests/middleware/backendRequestMetricsMiddleware_test.py -q

docker compose -f docker-compose.monitoring.yml up -d
Start the backend with PROMETHEUS_METRICS_ENABLED=true and metrics on port 8001. runserver needs --noreload.
Send a few requests:

curl -s -o /dev/null -D - -H 'User-Agent: kci-dev/0.1.11 (Linux)' http://localhost:8000/api/schema/
curl -s -o /dev/null -D - -H 'User-Agent: curl/8.5.0' http://localhost:8000/api/schema/
curl -s -o /dev/null -D - -A 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36' http://localhost:8000/api/schema/
curl -s -o /dev/null -D - -A 'Mozilla/5.0' http://localhost:8000/admin/
curl -s -o /dev/null -D - http://localhost:8000/api/hardware/

On http://localhost:8001/metrics/ confirm client="kci-dev", client="script", and client="dashboard" on both dashboard_backend_requests_by_client_total and dashboard_backend_request_latency_seconds_count, for endpoint schema. django_http_requests_latency_seconds_by_view_method_count must not have view="schema". It must have admin:index or admin:login. The hardware request is endpoint="hardware", status_class="4xx".
In Grafana at http://localhost:3000, sign in as admin / admin. The Prometheus datasource URL is http://prometheus:9090 and its uid must be prometheus. Import monitoring/dashboard.json. Set the range to the last 15 minutes.
With Client set to All clients, Total Calls shows schema split by dashboard, kci-dev, and script. Average Response Time shows avg, p50, and p95 for those series after Prometheus has scraped twice. Client set to kci-dev leaves only that client. Summary Mode set to All Time shows the totals.
4xx and 5xx by Endpoint shows hardware - 4xx. It does not show schema.

* Time /api/ requests on dashboard_backend_request_latency_seconds by endpoint and client * Keep /admin/ and /health/ on the Django view-latency histogram * Point total calls, average, p50, and p95 at those client series Built on kernelci#2154. Closes kernelci#2149 Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

* Add a panel summed by endpoint and status_class, limited to 4xx and 5xx Part of kernelci#2150 Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

The latency tests replaced the process-wide Prometheus histogram and left it replaced, so the health-skip test saw no /api/ observation once both suites shared a worker. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

Ruff I001 failed pre-commit because prometheus_client sat after the first-party import. Assisted-by: Cursor Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

GET and POST to the same endpoint were recorded as one series. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

felipebergamin · 2026-10-07T21:34:54Z

          },
          "editorMode": "code",
-          "expr": "(sum(django_http_requests_latency_seconds_by_view_method_count{view!=\"<unnamed view>\"}) by (view, method)  and on() label_replace(vector(1), \"mode\", \"$summary_mode\", \"\", \"\")==label_replace(vector(1), \"mode\", \"all\", \"\", \"\") or round(sum(increase(django_http_requests_latency_seconds_by_view_method_count{view!=\"<unnamed view>\"}[$__range:])) by (view, method)) and on() label_replace(vector(1), \"mode\", \"$summary_mode\", \"\", \"\")==label_replace(vector(1), \"mode\", \"range\", \"\", \"\"))",
+          "expr": "(sum by (endpoint, client) (dashboard_backend_requests_by_client_total{client=~\"$client\"}) and on() label_replace(vector(1), \"mode\", \"$summary_mode\", \"\", \"\")==label_replace(vector(1), \"mode\", \"all\", \"\", \"\") or round(sum by (endpoint, client) (increase(dashboard_backend_requests_by_client_total{client=~\"$client\"}[$__range:]))) and on() label_replace(vector(1), \"mode\", \"$summary_mode\", \"\", \"\")==label_replace(vector(1), \"mode\", \"range\", \"\", \"\"))",


This counter has been scraped since #2154. The histogram sum below starts when this deploys, and the table divides that time by these calls. All Time average stays low, because earlier calls are in this count and the histogram has no time for them. A fresh local Prometheus will not show that.

Could this column use dashboard_backend_request_latency_seconds_count, with the same grouping and the same all/range switch? Total Calls can keep this counter.

#2154 is on main, but it is not in production, so dashboard_backend_requests_by_client_total is not being scraped there yet. If we merge this PR, both should reach production together.

felipebergamin · 2026-10-07T21:34:54Z

          },
          "editorMode": "code",
-          "expr": "(sum(django_http_requests_latency_seconds_by_view_method_sum{view!=\"<unnamed view>\"}) by (view, method) and on() label_replace(vector(1), \"mode\", \"$summary_mode\", \"\", \"\")==label_replace(vector(1), \"mode\", \"all\", \"\", \"\") or sum(increase(django_http_requests_latency_seconds_by_view_method_sum{view!=\"<unnamed view>\"}[$__range:])) by (view, method) and on() label_replace(vector(1), \"mode\", \"$summary_mode\", \"\", \"\")==label_replace(vector(1), \"mode\", \"range\", \"\", \"\"))",
+          "expr": "(sum by (endpoint, client) (dashboard_backend_request_latency_seconds_sum{client=~\"$client\"}) and on() label_replace(vector(1), \"mode\", \"$summary_mode\", \"\", \"\")==label_replace(vector(1), \"mode\", \"all\", \"\", \"\") or sum by (endpoint, client) (increase(dashboard_backend_request_latency_seconds_sum{client=~\"$client\"}[$__range:])) and on() label_replace(vector(1), \"mode\", \"$summary_mode\", \"\", \"\")==label_replace(vector(1), \"mode\", \"range\", \"\", \"\"))",


This groups only by endpoint and client, so GET and POST share one average. Total Calls above does the same. The latency chart keeps method. Add method to both sums if the table should keep them apart.

Yes. Total Calls and both summary sums now group by endpoint, method, and client, same as the latency chart. The summary table groups on Method as well, so Grafana does not add GET and POST back into one row.

felipebergamin · 2026-10-07T21:34:54Z

+                endpoint=labels["endpoint"],
+                method=labels["method"],
+                client=labels["client"],
+            ).observe(time.perf_counter() - started)


This sample includes the labeling and the counter update above. The visitor cache write below stays outside it. Save the elapsed time right after get_response returns, and pass that into observe, so the labeling stays out of the histogram.

Changed to make time measurement only on the response generation.

Stop the histogram sample when the response returns, and group the dashboard sums by method so GET and POST stay apart. Assisted-by: Cursor Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

felipebergamin

approving. but please take a look on my comment about the docs/monitoring.md file

felipebergamin · 2026-10-08T15:19:02Z

just a small update on this doc.

it still describes the dashboard from before the method split. The feature list dropped Method, and the Client section says Total Calls is summed by endpoint and client only. The queries also group by method. The same feature list is not mentioning 4xx and 5xx by Endpoint.

alanpeixinho force-pushed the feat/analytics-latency-by-client branch 2 times, most recently from 49ffcb5 to c9ba9d2 Compare October 2, 2026 18:45

felipebergamin reviewed Oct 5, 2026

View reviewed changes

Comment thread backend/kernelCI_app/middleware/backendRequestMetricsMiddleware.py Outdated

Comment thread backend/kernelCI_app/middleware/backendRequestMetricsMiddleware.py Outdated

Comment thread backend/kernelCI_app/middleware/backendRequestMetricsMiddleware.py

felipebergamin mentioned this pull request Oct 5, 2026

feat(analytics): split kci-dev, scripts, and bots from dashboard traffic #2154

Merged

alanpeixinho force-pushed the feat/analytics-latency-by-client branch from 803aa9a to 3e6cda3 Compare October 6, 2026 21:09

alanpeixinho requested a review from felipebergamin October 7, 2026 18:39

alanpeixinho added 5 commits October 7, 2026 15:45

feat(analytics): show 4xx and 5xx by endpoint

4e5a57d

* Add a panel summed by endpoint and status_class, limited to 4xx and 5xx Part of kernelci#2150 Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

fix(analytics): sort request metrics middleware imports

d1c0d97

Ruff I001 failed pre-commit because prometheus_client sat after the first-party import. Assisted-by: Cursor Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

fix(analytics): split API latency by HTTP method

410a7a7

GET and POST to the same endpoint were recorded as one series. Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

alanpeixinho force-pushed the feat/analytics-latency-by-client branch from 0dc78b3 to 410a7a7 Compare October 7, 2026 18:47

felipebergamin reviewed Oct 7, 2026

View reviewed changes

fix(analytics): keep latency on the response and split methods

0a217d1

Stop the histogram sample when the response returns, and group the dashboard sums by method so GET and POST stay apart. Assisted-by: Cursor Signed-off-by: Alan Peixinho <alan.peixinho@profusion.mobi>

felipebergamin approved these changes Oct 8, 2026

View reviewed changes

alanpeixinho added this pull request to the merge queue Oct 8, 2026

Merged via the queue into kernelci:main with commit 70439c9 Oct 8, 2026
7 checks passed

Navigation Menu

Repository navigation

feat(analytics): record API latency by endpoint and client - #2156

feat(analytics): record API latency by endpoint and client#2156
alanpeixinho merged 6 commits into
kernelci:mainfrom
profusion:feat/analytics-latency-by-client

alanpeixinho commented Oct 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

felipebergamin Oct 7, 2026

Uh oh!

alanpeixinho Oct 8, 2026

Uh oh!

felipebergamin Oct 7, 2026

Uh oh!

alanpeixinho Oct 8, 2026

Uh oh!

felipebergamin Oct 7, 2026

Uh oh!

alanpeixinho Oct 8, 2026

Uh oh!

alanpeixinho Oct 8, 2026

Uh oh!

felipebergamin left a comment

Uh oh!

felipebergamin Oct 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

alanpeixinho commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it is

How to test

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felipebergamin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

alanpeixinho commented Oct 2, 2026 •

edited

Loading