feat: add FastAPI middleware for per-request emissions tracking

[davidberenstein1957]

Description

Adds per-request energy and emissions attribution for FastAPI/Starlette apps. One tracker runs for the app lifetime; CodeCarbonMiddleware splits each of the tracker's sampling windows across the requests that were in flight during it, weighted by overlap and normalised by the sum of weights (codecarbon/integrations/fastapi/attribution.py). A request's number is only known after the next window closes, so on_request(energy, emissions_kg, status_code) fires then, on the scheduler thread.

Related Issue

N/A

Motivation and Context

Per-request start/stop snapshots cannot do this attribution correctly: with N requests in flight, each one sees the whole machine's delta, so the sum overcounts by roughly N. This middleware maintains the invariant attributed_kwh + unattributed_kwh == settled_kwh after every window, and computes carbon intensity once per window (emissions_kg = energy_kwh x intensity) without touching tracker totals from the scheduler thread.

How Has This Been Tested?

tests/integrations/test_fastapi.py covers attribution invariants and concurrency, the middleware with a fake tracker (window resolution, 500 on raise, close on shutdown, never-started tracker), and the documented lifespan pattern on an OfflineEmissionsTracker.

Screenshots (if appropriate):

N/A

Usage

@asynccontextmanager
async def lifespan(app: FastAPI):
    tracker = EmissionsTracker(allow_multiple_runs=True)
    tracker.start()
    app.state.codecarbon_tracker = tracker
    yield
    tracker.stop()

app = FastAPI(lifespan=lifespan)
app.add_middleware(CodeCarbonMiddleware)  # module level; or tracker=...

Requests are only recorded while the tracker runs. Pending requests are settled when the tracker stops or changes, and on lifespan shutdown.

Changes

• codecarbon/integrations/fastapi/ — attributor + middleware.
• emissions_tracker.py — add/remove_energy_window_observer, notified after each sample, and _carbon_intensity_kg_per_kwh().
• pyproject.toml — codecarbon[fastapi] extra; fastapi/httpx in dev.
• docs/how-to/examples.md — usage section.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

AI Usage Disclosure

• 🟥 AI-vibecoded: You cannot explain the logic. Car analogy : the car drive by itself, you are outside it and just tell it where to go.
• 🟠 AI-generated: Car analogy : the car drive by itself, you are inside and give instructions.
• ⭐ AI-assisted. Car analogy : you drive the car, AI help you find your way.
• ♻️ No AI used. Car analogy : you drive the car.

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the docs/how-to/contributing.md document.
• I have added tests to cover my changes.
• All new and existing tests passed.

[codecov]

Codecov Report

❌ Patch coverage is 93.00000% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.74%. Comparing base (e5e46ab) to head (6819b50).

Files with missing lines	Patch %	Lines
codecarbon/integrations/fastapi/middleware.py	92.20%	6 Missing ⚠️
codecarbon/integrations/fastapi/attribution.py	94.94%	5 Missing ⚠️
codecarbon/emissions_tracker.py	85.71%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1203      +/-   ##
==========================================
+ Coverage   91.70%   91.74%   +0.04%     
==========================================
  Files          49       52       +3     
  Lines        5157     5357     +200     
==========================================
+ Hits         4729     4915     +186     
- Misses        428      442      +14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[benoit-cty]

Hello, thanks for this. There is a problem with your branch : there are many changes that are already merged. Can you do a rebase ?

[davidberenstein1957]

@benoit-cty I have reached out to some people at FastAPI, if they would be interested in a quick review :) For visibility, we could also consider deploying it as a standalone integration, but let's see if people like it.

[SaboniAmine]

Thanks David this a great PR! Left a few questions, and saw that you have already prepared a benchmark script, do you have any numbers / graph already computed to share ?

[davidberenstein1957]

@inimaz This is ready for review. Rebased on v3.2.9 and benchmarked on a real HF embedder workload — middleware adds about 3 ms per request (~24 ms → ~27 ms). Details in the updated PR description and docs/how-to/fastapi.md.

[davidberenstein1957]

@SaboniAmine Thanks again for the thorough review — I've addressed the inline threads in the latest pushes:

• Core enums for emission fields / header presets / HTTP methods (codecarbon/core/emission_fields.py)
• Simpler resolve_header_mapping + package-level Starlette import guard
• Concurrent task naming documented/tested (stable route labels + UUID uniqueness on the lifespan path)
• Measurement model docs + regression test (sync measure on tracker worker before callback)
• compose_lifespans() for stacking with user-owned startup/shutdown
• Benchmarks vs raw FastAPI + Logfire (--with-logfire); ~3 ms CodeCarbon overhead on the HF embedder workload

Local uv run task test-package: 552 passed. Ready for another look when you have a moment.

[davidberenstein1957]

Ponytail cleanup on this PR:

• Removed unused _headers.py / core emission_fields enums (never wired to middleware)
• Added opt-in response_headers= (sync measure → X-CodeCarbon-*; costs client latency)
• Added include_background_tasks (default True; set False to finalize at end-of-body)
• Docs: BackgroundTasks are included by default; WebSockets stay unsupported (no implementation)

See docs/how-to/fastapi.md Limitations + config section.

[davidberenstein1957]

Added a sync response_headers row to the Performance section (and --with-headers on the benchmark script):

Mode	Rough client latency (mocked 20 ms measure)
Deferred default	~+3 ms (embedder table)
response_headers=True, c=1	55 ms (+25 ms vs ~30 ms baseline)
response_headers=True, c=4	~95 ms (tracker worker serializes sync measures)

Reproduce: uv run --extra fastapi python scripts/benchmark_fastapi_middleware.py --quick --with-headers

[davidberenstein1957]

@inimaz @SaboniAmine — ready for another look when you have time.

This commit finishes the HTTP perf work:

• REQUEST lane for mark_http_request_start (no thread-pool hop)
• Cached cloud metadata + emissions template on HTTP finalize
• Docs updated with live HF embedder benchmarks only (--realistic --with-headers)

Deferred middleware stays in the same ballpark as baseline on the live benchmark (~30–32 ms vs ~42 ms baseline mean on Darwin arm64).

[benoit-cty]

🤖 This review comment was written and posted by Claude Opus 5.5 (AI assistant), at the request of @benoit-cty. Findings were checked by reading the code and running tests locally (merged with current master where relevant), but please double-check before acting on them.

Verdict: 🔧 Request changes

Splitting each sampling window's energy across the requests in flight, weighted by overlap, is a nice design, and the attribution unit tests (sum invariant, concurrency) are good. But the PR has drifted: the branch was rewritten into a single commit (e293e0c), and the description and earlier review threads describe code that no longer exists: create_codecarbon_lifespan, compose_lifespans, response_headers, save_to_api, include_background_tasks, the codecarbon[fastapi] extra, docs/how-to/fastapi.md, and the benchmark script. It also conflicts with master in codecarbon/emissions_tracker.py.

Must fix:

1. The documented usage crashes at startup.
   • docs/how-to/examples.md calls add_codecarbon_middleware(app, …) inside lifespan. Starlette builds the middleware stack before lifespan runs, so this raises RuntimeError: Cannot add middleware after an application has started (middleware.py:118). I reproduced this with FastAPI 0.141.1 / Starlette 1.6.0.
   • Fix: document adding the middleware at module level, with the tracker started and stopped in lifespan. The middleware could look up the tracker lazily from app.state. Please add a test for exactly the documented pattern.
2. Memory grows without limit. attribution.py begin() adds to _in_flight, and entries are removed only by on_window() (a tracker sample) or close(). If the tracker was never started or has stopped, and nothing calls close() (nothing does so automatically), every request leaks one _InFlight. Don't record requests when the tracker isn't running, or evict finished requests when no window is pending, and call close() on lifespan shutdown.
3. Heavy work and a race on the scheduler thread.
   • For each finished request, in each window, _resolved calls tracker.http_request_emissions() → _prepare_emissions_data() → _update_emissions() (emissions_tracker.py:1016, middleware.py:91). That runs inside _measure_power_and_energy on the scheduler thread. At high request rates, that is thousands of full EmissionsData builds per window, which delays sampling.
   • It also mutates _total_emissions / _last_energy_covered without a lock, racing with flush() / stop() on the main thread, so a delta can be counted twice.
   • Fix: compute each window's carbon intensity once, and derive request emissions as energy_share × intensity without touching tracker totals.
   • _notify_energy_window_observers also iterates _window_observers while close() can remove from it on another thread. Iterate over a copy.
4. No optional extra. FastAPI and httpx are only in the dev group (pyproject.toml:98-99), and middleware.py:8 imports starlette at the top of the module with no hint. Please add a codecarbon[fastapi] extra, and raise an ImportError that tells users to install it.

Tests:
5. The single end-to-end test relies on time.sleep with a 0.5 s window and a real EmissionsTracker that does an online geo lookup, so it is likely to be flaky. Please use an offline tracker, or mocks. Add tests for close(), a tracker that was never started, the app raising (the status stays 500), and the lifespan pattern from point 1.

Housekeeping:
6. Please rebase, rewrite the description, and re-request review. @SaboniAmine's earlier threads (enums, lifespan, measuring an async stop) were answered against code that has since been removed.

Nits:

• add_codecarbon_middleware defines a subclass on every call just to get the instance onto app.state.
• It reads the private tracker._total_energy.
• The default log_request logs at INFO on every request; DEBUG would be better.
• http_request_emissions() puts a web-framework concept into the core tracker. It would sit better in the integration module.

[davidberenstein1957]

Made the changes in 6819b50: module-level middleware with lifespan tracker, no leak, per-window intensity, fastapi extra, offline tests. Merged master (not rebased, to avoid a force-push) and rewrote the description.
Not done: the attributor still reads the private tracker._total_energy / _start_time; a public accessor felt like more core API than this PR needs.