refactor(plugin): create one plugin instance per invocation

[ParidelPooya]

Overlapping invocations on Lambda Managed Instances can overwrite shared plugin state and lose telemetry records. This PR moves Core 3.0 and OTel 2.0 to per-invocation plugin factories and makes the shared Telemetry exporter scheduler preserve each invocation's export and flush obligations. Supersedes #734.

Plugin contract and migration

plugins accepts objects with create_plugin(info). The host resolves the factory list once and creates a fresh executor and plugin instances for each invocation. Environment entry points name factory objects, and repeated registrations of the same factory are deduplicated by identity. The removed v1 provider contract, plugin instances and bare callables are rejected at registration.

class MyFactory:
    def create_plugin(self, info):
        return MyPlugin(shared_exporter)


@durable_execution(plugins=[MyFactory()])
def handler(event, context):
    ...

The bundled factories are InvocationOtelPluginFactory, ExecutionOtelPluginFactory, and the factory returned by durable_execution_telemetry. OTel and Telemetry packages require Core >=3.0.0,<4. The renamed Python Telemetry exporter retains version 0.0.1.

Ordinary exceptions from factory calls and validation of their returned objects are contained so healthy plugins and handler code can continue. Thread-control exceptions retain their existing propagation and exception-group handling.

Factory registration preserves upstream view exclusion across explicit and environment configuration, including inherited opt-in groups and supported factory class shapes. Rejected configurations notify opted-in factories once so constructor-owned resources can be released, including when another explicit entry is invalid.

Invocation isolation and lifecycle

Start, the handler and End run in an invocation-owned worker context. Successful Start bindings reach handler code and SDK-managed callbacks; failed Start bindings are rolled back without changing the host context. Completed durable work still replays from checkpoints. End reaches every started plugin, and an End failure does not replace the handler's original failure. Service-requested PENDING remains authoritative after checkpoint-token withdrawal.

OTel state belongs to each plugin instance. The shared log filter resolves a live invocation claim for the emitting thread or task. Logs outside an active invocation retain their original fields, including constructor-time logs with an ambient span. Fallback root export and stable trace identity from upstream are preserved.

Breaking changes: migrate registrations to factory objects and upgrade Core and OTel together. Instrumented handlers inherit host context bindings, which can change ContextVar visibility and the parent of a customer span created at handler entry. Installed-wheel tests cover the coordinated pair and explicit rejection of released OTel 1.x registrations.

Telemetry export and concurrency

One shared worker serializes exporter calls. Pending records coalesce only within the same execution, and a drain retains its own execution's record obligation. Worker recovery, reentrant drain handling and deferred release of displaced records preserve progress across exporter failures and customer finalizers.

Customer content transforms and result overrides run outside the plugin lock. Immutable snapshots and revision checks prevent a stale callback result from overwriting a newer or terminal record. Terminal admission closes before finalization, and obsolete customer objects are released outside both plugin and scheduler locks.

Each drain also captures a finite snapshot of globally scheduled work. A queued or in-flight flush may be shared when it covers that snapshot; otherwise a later flush is queued. Earlier targets stay fixed while new records arrive. A coalesced replacement discharges the original queue slot's obligation, and reschedules after the worker takes that slot do not extend an earlier drain.

Exporter failures remain logged and isolated. Drain coverage describes work offered to exporters and the covering flush attempt; it does not certify delivery to the destination.

Validation

Local validation covers the exporter changes at e10b56dc (tree 8e1ba4a8) and the factory-result containment fix at e447bc50 (tree 7b5b2c71):

• All 202 Telemetry tests pass, including public factory/hook reciprocal reentry, stale builds, duplicate End, exporter reentry, terminal-build failure, queued/in-flight drain snapshots, coalescing and earlier-waiter progress.
• Negative controls reproduce the prior deadlocks and incomplete flushes. Independent review also exercised sequence gaps and a duplicate End waiting while the primary transform fails.
• Telemetry source/tests pass type checks (43 files), lint, formatting, and wheel/sdist builds.
• The factory-result fix passes 321 invocation/plugin/discovery tests and Core type checks (84 files). Eight new public-handler regressions and 11 independent probes cover failing class descriptors, supported proxies, ordinary failures, thread-control identity and mixed exception groups.
• Core, Testing SDK, OTel, examples, conformance and installed-wheel validation passed during upstream integration. The preceding head e447bc50 passed every GitHub check and workflow, including Python 3.11–3.14 builds and cloud tests.

The latest upstream integration at 214c309c (tree 62c3a0ef) includes main 2b408f44 and its chained-invoke trace propagation. The three import conflicts are resolved, the incoming propagation tests use invocation factories and fresh instances on replay, and the compatibility notes match Core 3 / OTel 2. Independent review found no merge-specific issues.

Validation of this integration on Python 3.13:

• All 1,959 Core tests and 736 OTel tests pass. Core coverage is 98.61%.
• Core and OTel type checks pass across 131 files; both packages pass lint, formatting, and wheel/sdist builds.
• Installed wheels pass all 736 OTel tests, 12 lifecycle compatibility tests, and four released-OTel rejection tests.
• The broader Testing SDK, examples, Telemetry, local conformance-handler, and CI-script run has 2,178 passes and two skips. Its only failure is the pre-existing .github/scripts/tests/test_lintcommit.py::test_subject_uppercase: the test expects lowercase-only subjects, while the linter accepts uppercase. The linter and test are byte-identical on main, the prior PR head, and this merge; isolated script tests reproduce the same failure. All tests selected by the CI script-test workflow pass in these runs.
• Cloud tests and the Python 3.11–3.14 CI matrix are left to the current-head GitHub workflows.

No package release or manual AWS deployment was performed. Current-head GitHub checks and independent approval remain release gates.

[yaythomas]

Thank you for the thoroughness here. The Barrier-based isolation test, the measured loss tables, and the BaseException reasoning for worker threads are all correct, and they are exactly the evidence a concurrency fix needs. One blocker in the log filter, one design proposal at the factory type, two items for the breaking-change note, and some nits.

Size. Commit 1 (the Insight scheduler fix) is independently correct and independently urgent, and it does not depend on the contract change.

[github-actions]

Codex AI review

No actionable findings in the PR changes. Residual test risk: tests were inspected but not executed under the read-only review constraints.

Reviewed commit 240ac95400354876b8b2d20c256a9ff6f365bf2e. Workflow run