Skip to content

[Engine] serve: report timings on /v1/responses - #3291

Open
pschaub wants to merge 1 commit into
Niko1221:mainfrom
pschaub:serve-responses-timings
Open

pschaub wants to merge 1 commit into
Niko1221:mainfrom
pschaub:serve-responses-timings

Conversation

@pschaub

@pschaub pschaub commented Oct 19, 2026

Copy link
Copy Markdown

Problem and result

/v1/chat/completions returns top-level timings (#107), but /v1/responses (#451) does not, although server.py already computes them for every request. Now the final response.completed / response.incomplete event and the non-streaming reply include the same timings object.

Related discussion

Follow-up to #107 (based on #83) for /v1/responses.

Approach

_with_timings in serve/responses.py copies done["timings"] onto the final event; collect() keeps it on the non-streaming reply. Nothing else changes.

Validation

Passed:

  • python -m unittest discover -s serve -t . -p "test_*.py" on this branch (one commit on main e88ee39), Windows 11, Python 3.12: 882 tests OK, 20 skipped. The same command on main e88ee39 without the change: 881 tests OK, 20 skipped.
  • The new UsageAndStatus.test_responses fails without the change.
  • Live check on main e88ee39 (Windows 11, CUDA): the final event of streamed /v1/responses requests carries timings; without the change it does not.

Failed:
none observed

Not run:

  • Linux and macOS.
  • The non-streaming /v1/responses reply was checked by the unit test only; the live check used streaming.
  • No CUDA, HIP, SYCL or Metal build: the change is Python only.

Limits

response.failed carries no timings.


Created with Claude Opus 5.5. The result was verified by a human (me) in llama-swap: prompt and decode speeds now show for /v1/responses requests.

@pschaub
pschaub force-pushed the serve-responses-timings branch from ce43d3c to bc9c42e Compare October 19, 2026 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant