Skip to content

test: correct the REST deviation records - #725

Merged
owenpearson merged 9 commits into
mainfrom
uts/deviations-corrections
Oct 8, 2026
Merged

owenpearson merged 9 commits into
mainfrom
uts/deviations-corrections

Conversation

@owenpearson

@owenpearson owenpearson commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

A triage of the REST entries in test/uts/deviations.md found several that are wrong: an SDK gap that isn't one, spec errors filed as deviations, a spec fault blamed on the wrong spec, adapted tests listed as failing, wrong labels, and statuses that don't hold up against features.md. This corrects them and re-measures the counts in the header.

Test changes

Tests change only where their classification changes.

  • TP3a/d/g are un-gated. The SDK already fills these fields from the ProtocolMessage: RealtimeChannel._on_message calls Message.update_inner_message_fields before decoding. The derived helper skipped that step. It now makes the same call, and TP3g compares against a datetime, as test_tp3_presence_from_json already does. All three pass.
  • RSA16c (expiry renewal) and RSA16d (switch to basic) move from @deviation to @spec_error.

deviations.md corrections

  • Batch envelopes: the spec-error entry had the wrong spec at fault. RSC22b has batchPublish return "an array of BatchResults". ably-js's sandbox tests and the sandbox's own GET /presence?channels= response both use the envelope, so the fault is batch_publish.md's flat results, not batch_presence.md. This was never filed upstream.
  • Moved: RSA4 and RSA12a were listed under Failing Tests, but their tests are adapted, not gated.
  • Labels:
    • The httpRequestTimeout row is TO3l4, not TO3l1 (disconnectedRetryTimeout).
    • The TI row is TI1/TI4.
  • Statuses:
    • RSC15a is an off-by-one: TO3l5 counts fallback hosts, so the default allows 4 attempts, as ably-js, ably-java and ably-go make.
    • REC1b1/c1 is compliant with RSC1b's 40106.
    • RSA6b/d: canonicalisation is permitted, not required by RSA9f.
    • RSC19e: @catch_all would not close it.
    • HP8: its prose and IDL disagree, and it has a non-breaking fix.
  • Claims:
    • validate_message_size is not a TM6 calculation.
    • RSA16b's invented expiry cannot drive renewal for a token-string client.
    • RSA10b and RSA8c1a also need the RSA12a fix.
    • RSA15a's three tests cannot all pass as written.
    • A lowercase authorization header sends two auth headers.
    • urljoin resolves dot-segments in device ids.
    • RSL2's quote_plus reaches presence, annotations, serials and realtime history too.
    • Presence already sends %3A.
    • The logging tests need log events the SDK never emits.
    • The test_ti_errorinfo_from_json adaptation is now recorded.
    • Stale line references for the realtime size check are updated.
  • New spec faults, marked not yet filed:
    • The RSL1i at-limit fixture: 5 + 1024 > 1024 under TM6a.
    • client_id.md: RSA15a timing, and the RSA12a/b labels.
    • token_request_params.md: RSA5c/RSA6c labels.
    • request.md: RSC19b "may vary".
    • fallback.md: REC1b1/c1 40000.
    • batch_publish.md: RSC22_Error1/2.
    • features.md: HP8 prose vs IDL.
  • Also recorded:
    • The test_to3_client_options_custom_hosts adaptation.
    • fallbackHostsUseDefault's REC2b test, and its removal in 2.0.
  • Housekeeping: three Smaller faults rows appeared twice.

Counts

Measured by collecting the suite and mapping each item to its # UTS: id. Against unchanged main, this reproduces the existing header exactly.

  • Test IDs: 907 pass, 210 are gated (215 cases) and 15 cannot run. Spec faults go from 10 to 12 Test IDs, which reduce to 8 root causes. SDK root causes go from 71 to 68, 24 of them REST.
  • helpers/ cases: 130, not 122.
  • Closing counts section: it had drifted from the header, and now matches the run.

Not in this PR

Testing

  • uv run --frozen --extra crypto --extra dev pytest test/uts -q: 1134 passed, 230 skipped.
  • RUN_DEVIATIONS=1 uv run --frozen --extra crypto --extra dev pytest test/uts -q: 215 failed, 1134 passed, 15 skipped. Every gated case fails when enabled.
  • uv run ruff check: clean.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Timed-out connection attempts are now cancelled, and late authentication responses no longer start a connection attempt.
    • Improved handling of asynchronous callbacks across event listeners, timers, and test clocks.
  • Documentation
    • Updated compatibility notes with revised test status, specification gaps, REST behavior findings, and adoption counts.
    • Clarified findings related to authentication, message handling, logging, and REST requests.
  • Tests
    • Updated conformance expectations for authentication renewal, authentication mode changes, and presence-message timestamps.
    • Corrected presence-message test setup to include protocol-message attributes; added coverage for connection-attempt timeouts.

A triage of the REST entries in deviations.md found several that do not
hold against features.md or the code.

TP3a/d/g are not an SDK gap: RealtimeChannel._on_message fills the
presence fields through Message.update_inner_message_fields, and the
derived helper now does the same, so the three tests pass ungated.
RSA16c's expiry renewal and RSA16d's switch to basic auth are spec
errors (RSA4b1, and RSA10a/e/f), so they are marked @spec_error.

The batch envelope entry now names batch_publish.md as the spec at
fault, since RSC22b returns an array of BatchResults. RSA4 and RSA12a
move to the adapted rows they belong in, labels and statuses are
corrected, new spec faults are recorded as not yet filed, duplicate
rows are removed, and the header counts are re-measured.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 002153b8-bbde-4586-a845-09ba8ed05033
📥 Commits

Reviewing files that changed from the base of the PR and between e4dd980 and 61fca8c.

📒 Files selected for processing (14)
  • .claude/skills/uts-to-python/SKILL.md
  • .github/workflows/check.yml
  • .github/workflows/lint.yml
  • .github/workflows/release.yml
  • ably/realtime/channel.py
  • ably/realtime/connectionmanager.py
  • ably/util/eventemitter.py
  • ably/util/helper.py
  • pyproject.toml
  • test/ably/realtime/realtimechannel_publish_test.py
  • test/ably/rest/encoders_test.py
  • test/unit/connectionmanager_test.py
  • test/uts/deviations.md
  • test/uts/helpers/clock.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/uts/deviations.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


Walkthrough

The pull request updates coroutine detection and connection-attempt cleanup. It adjusts related tests, revises UTS deviation records, updates warning calls, and advances pinned GitHub Actions revisions.

Changes

SDK behavior and test maintenance

Layer / File(s) Summary
Coroutine callback handling
ably/util/helper.py, ably/util/eventemitter.py, test/uts/helpers/clock.py, pyproject.toml, test/ably/rest/encoders_test.py
A shared helper recognizes coroutine functions and replaces direct coroutine-function checks in event listeners and timers. The fake clock now awaits callback results when they are awaitable. Pytest warning filters and shared HTTP 201 mocks are also added.
Connection-attempt cancellation
ably/realtime/connectionmanager.py, test/unit/connectionmanager_test.py, test/ably/realtime/realtimechannel_publish_test.py
disconnect_transport cancels an active connection attempt unless it is the current task. New tests cover timeout cancellation and delayed authentication. Publish tests dispose of the transport before notifying DISCONNECTED.
UTS test classification and decoding
test/uts/rest/unit/auth/token_details_test.py, test/uts/rest/unit/types/presence_message_types_test.py
Two token-detail tests change from @deviation to @spec_error. Presence-message tests copy protocol attributes before decoding and assert the timestamp as a datetime.
UTS deviation and behavior records
test/uts/deviations.md, .claude/skills/uts-to-python/SKILL.md
The deviation document updates test totals and records specification faults, REST behavior details, feature gaps, and connection-failure findings. The skill guidance removes the refused-connection task-leak claim.
Workflow revisions and warning calls
.github/workflows/check.yml, .github/workflows/lint.yml, .github/workflows/release.yml, ably/realtime/channel.py, ably/realtime/connectionmanager.py, pyproject.toml
The workflows use updated pinned GitHub Actions revisions. Two warning calls use log.warning, and Ruff enables G010.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 61fca

This change updates async compatibility handling, cleans up connection attempts when a transport is disposed, and refreshes test records and CI action versions. No actionable merge-blocking risk was identified.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 15.91% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 44 functions across 10 files. (6 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the main change: correcting REST deviation records and related test classifications. It is concise and specific.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 15.91% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 44 functions across 10 files. (6 skipped: 6 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit checks the callback’s tune,
Then waits until its task is through.
The tests record each state in sight,
While UTS notes sort facts aright.
New actions hop through workflow lanes,
And warnings speak with proper names.

Comment @coderabbitai help to get the list of available commands.

owenpearson and others added 5 commits September 30, 2026 16:57
When the transition or suspend timer ends a connection attempt, the
transport it was opening is disposed, but connect_base() is left
awaiting a future that only that transport's 'connected' or 'failed'
events settle. Neither fires once the transport is disposed, nor when
the attempt failed with an error ws_connect does not catch, so each such
attempt leaves a pending task for the garbage collector to destroy
("Task was destroyed but it is pending!"). An attempt abandoned while
still authenticating resumes when auth answers, and connects from
DISCONNECTED.

disconnect_transport() cancels the in-flight attempt alongside disposing
its transport, as close_impl() and on_closed() already do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Logger.warn emits a DeprecationWarning. Replace the two remaining calls
in the realtime channel and connection manager, and enable ruff's G010
rule so new ones are caught by lint.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The encoder tests patched Http.post with a bare AsyncMock, whose default
return value is itself an AsyncMock. publish_messages calls the
synchronous Response.to_native() on that value, which produced a
coroutine that was never awaited and emitted a RuntimeWarning for each
of the 11 tests.

The patched post resolves to an empty 201 Response, so the response
parsing in publish_messages runs as it does against a real server.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ng tests

test_fail_on_disconnected_when_queue_messages_false and
test_queue_on_disconnected_when_queue_messages_true forced DISCONNECTED
while the transport was still connected. The immediate reconnect replaced
the transport without closing it, so its websocket tasks were still
pending when the test's event loop closed, and garbage collection later
surfaced them as PytestUnraisableExceptionWarning in whichever test was
running.

Dispose the transport first, as the other simulated-disconnect tests do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
checkout, setup-python, cache, upload-artifact and download-artifact
were pinned to majors that target Node 20, which GitHub warns about and
forces onto Node 24. Pin the latest majors, which run on Node 24
natively.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
owenpearson and others added 3 commits October 1, 2026 13:13
Python 3.14 deprecates asyncio.iscoroutinefunction, and the event
emitter, Timer and is_callable_or_coroutine called it on every listener
registration. is_coroutine_function keeps its semantics on every
supported Python, including callables carrying asyncio's _is_coroutine
marker (AsyncMock from the mock package, asgiref-marked callables),
which inspect.iscoroutinefunction alone rejects. The fake clock awaits
whatever its callback returns.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
pytest-asyncio 0.23 calls asyncio.iscoroutinefunction and the event
loop policy functions, producing ~710k warnings per 3.14 job. The
releases that avoid them need Python 3.9 and pytest 8.2, so filter
those three messages from the pytest_asyncio module only.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Clear the warnings the check workflow prints

@ttypic ttypic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@owenpearson
owenpearson merged commit e195b3c into main Oct 8, 2026
10 checks passed
@owenpearson
owenpearson deleted the uts/deviations-corrections branch October 8, 2026 16:36

This branch was successfully deployed

1 active deployment
staging/pull/725/features — 61fca8cc Deployed Oct 7, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants