feat: add SDK telemetry into emissions tracker by davidberenstein1957 · Pull Request #1200 · mlco2/codecarbon

davidberenstein1957 · 2026-05-19T18:01:52Z

Description

Adds product telemetry, sent from EmissionsTracker.stop(), opt-out by default. Telemetry has three levels (disabled, minimal default, extensive) resolved from the telemetry_level= argument / codecarbon monitor --telemetry-level, then telemetry_level in .codecarbon.config, then CODECARBON_TELEMETRY_LEVEL, then the default. It is only sent on a stop() of a run lasting at least 1 s, and only when a telemetry API key is configured; there is no built-in key, so nothing is sent by default today.

Related Issue

Fixes #1106

Motivation and Context

The project wants visibility into how CodeCarbon is used (environment, hardware, and optionally run metrics/usage hints) to guide development, while keeping the feature opt-out by default, never blocking a run, and disclosing exactly what is sent.

How Has This Been Tested?

uv run task test-package
uv run task test-api-unit / test-api-integ for the server side

Screenshots (if appropriate):

N/A

Details

minimal: private POST /telemetry with environment and hardware fields. extensive: adds run metrics and usage hints, plus a public run summary through ApiClient. Every field is listed in docs/how-to/telemetry.md.
Never blocks: the payload is built and sent on a background thread under a 2 s budget. At exit, pending sends get a join capped at 1 s. Logs from that thread are DEBUG.
Asking the user: interactive codecarbon config / codecarbon monitor ask once and save the answer to ~/.codecarbon.config. Non-interactive runs never prompt; they log a notice once per machine saying what is sent and how to opt out.
CLI: codecarbon telemetry (wizard), codecarbon telemetry set <level>, codecarbon telemetry status.
Server: /telemetry route, schema and tests in carbonserver.

Open for maintainers

/telemetry authentication. A write token shipped in the public package is not a secret. An unauthenticated ingestion endpoint with strict schema validation and rate limiting may fit better.
Data retention period, to add to the docs.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

AI Usage Disclosure

🟥 AI-vibecoded: You cannot explain the logic. Car analogy : the car drive by itself, you are outside it and just tell it where to go.
🟠 AI-generated: Car analogy : the car drive by itself, you are inside and give instructions.
⭐ AI-assisted. Car analogy : you drive the car, AI help you find your way.
♻️ No AI used. Car analogy : you drive the car.

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the docs/how-to/contributing.md document.
I have added tests to cover my changes.
All new and existing tests passed.

codecov · 2026-05-19T18:05:54Z

Codecov Report

❌ Patch coverage is 90.65217% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.77%. Comparing base (e5e46ab) to head (cd8ccf6).

Files with missing lines	Patch %	Lines
codecarbon/core/telemetry/collect.py	86.98%	19 Missing ⚠️
codecarbon/core/telemetry/dispatcher.py	84.05%	11 Missing ⚠️
codecarbon/cli/telemetry_cli.py	96.00%	5 Missing ⚠️
codecarbon/core/api_client.py	62.50%	3 Missing ⚠️
codecarbon/core/telemetry/settings.py	92.10%	3 Missing ⚠️
codecarbon/core/telemetry/client.py	95.65%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1200      +/-   ##
==========================================
+ Coverage   91.70%   91.77%   +0.07%     
==========================================
  Files          49       54       +5     
  Lines        5157     5579     +422     
==========================================
+ Hits         4729     5120     +391     
- Misses        428      459      +31

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

davidberenstein1957 · 2026-06-17T15:50:17Z

@inimaz, before merging, we should still add the hardcoded experiment and project for the codecarbon api.

Add CLI helper and interactive-flow tests, cover coordinate rounding in ApiClient, and extend collect environment probes to satisfy Codecov patch and project thresholds on PR #1200. Co-authored-by: Cursor <cursoragent@cursor.com>

inimaz · 2026-07-18T16:31:17Z

 .codecarbon.config*
 scripts/agent-vm.personal.config.sh
+
+# Added by ggshield


Maybe this is not needed as discussed

Dropped, the PR no longer touches .gitignore.

inimaz

Thanks for this PR @davidberenstein1957 . I have left some comments

inimaz · 2026-07-18T16:39:20Z

+        raise typer.BadParameter(str(error)) from error
+
+
+def resolve_config_path(config: Optional[Path], *, create: bool = False) -> Path:


Maybe this is not needed. get_hierarchical_config handles it?

Kept on purpose: set has to pick one file to write to, while get_hierarchical_config only merges what it reads.

inimaz · 2026-08-04T19:18:54Z


-        # logger.info("base tracker init")
        self._external_conf = get_hierarchical_config()
+        self._config_file_conf = get_config_file_settings()


Not needed

Suggested change

self._config_file_conf = get_config_file_settings()

Gone, _config_file_conf was removed.

inimaz · 2026-08-04T19:36:07Z

+    project_token_service: ProjectTokenService = Depends(
+        Provide[ServerContainer.project_token_service]
+    ),
+    x_api_token: str = Header(None),


does this work? Should it not be x_api_token: str = Header(None, alias="x-api-token")?

It works: FastAPI maps x_api_token to the x-api-token header by default, same as the emissions router.

inimaz · 2026-08-04T19:37:06Z

+    ),
+    x_api_token: str = Header(None),
 ) -> UUID:
+    project_token_service.project_token_has_access(


you are not doing anything with the response here, it should be if allowed--> do the action n?

project_token_has_access raises a 403 when access is denied, so the call is the guard. Whether this endpoint should take a token at all is still open (see my PR comment).

inimaz · 2026-08-04T19:41:31Z

+    return importlib.util.find_spec(name) is not None
+
+
+def _round_coordinate(value: Any) -> float | None:


maybe reuse the same function of _round_coordinate ApiClient so that both return the same if None.

Done in cd8ccf6: it uses _round_or_none now, so unknown coordinates are left out instead of sent as 0,0.

Send product telemetry at tracker stop, with the tier resolved from config, environment, or the EmissionsTracker(telemetry_level=...) kwarg. The send runs on a daemon thread so stop() never blocks on the network, and every request of one send shares a single 2s wall-clock budget, so the extensive tier cannot cost more just because it makes two calls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

benoit-cty · 2026-09-23T06:42:28Z

🤖 This review comment was written and posted by Claude Opus 5.5 (AI assistant), at the request of @benoit-cty. Findings were checked by reading the code and running tests locally (merged with current master where relevant), but please double-check before acting on them.

Verdict: 🔧 Request changes

Maintainer direction on the telemetry policy (@benoit-cty): telemetry should be opt-out by default, and the user should be asked the question explicitly. A suggested implementation:

On first interactive use (e.g. codecarbon config / codecarbon monitor in a TTY), ask once which telemetry level the user wants. Explain exactly what each level sends, and store the answer in the config (telemetry_level).
In non-interactive contexts (library use, CI, SLURM, no TTY), never block on a prompt. Apply the default, and show a clear notice once per machine (remember that it was shown) describing what is sent and how to opt out (codecarbon telemetry set disabled / CODECARBON_TELEMETRY_LEVEL).
docs/how-to/telemetry.md should list every field sent, the retention period, and how to opt out.

Must fix:

The default experiment_id changes for every existing user (emissions_tracker.py:609).
- The default is swapped from 5b0fa12a-… to DEFAULT_TELEMETRY_EXPERIMENT_ID (d2d69403-…).
- That value goes into EmissionsData.experiment_id (L1073), so into every CSV row. It is also used for save_to_api runs where the user set no experiment, so those users' emissions would be posted into the telemetry experiment.
- Please revert this line.
stop() blocks about 3 s, contradicting "never block".
- dispatcher.py:57 calls build_payload() synchronously, which calls collect.py:87 get_env_cloud_details(). That is uncached and probes 3 cloud metadata endpoints with a 1 s timeout each, plus NVML and find_spec lookups.
- Your own tests/test_telemetry.py::test_stop_does_not_block_on_hanging_endpoint fails locally (3.01 s against 0.5 s). It passes in CI only because GitHub runners are Azure VMs, where the metadata endpoint answers.
- Fix: build the payload inside the background thread, and/or reuse the cloud and hardware info the tracker has already detected.
The warning is misleading and shown on every run.
- send_at_stop returns early without a telemetry_api_key (dispatcher.py:51), and there's no built-in key, so nothing is sent by default.
- Yet every user without telemetry_level gets a WARNING saying "Minimal telemetry sends on each stop" (dispatcher.py:20-45), and codecarbon telemetry status prints the same.
- The message must match what is actually sent, and should be shown once per machine, as part of the prompt/notice flow above.
Authentication of /telemetry. Shipping a hardcoded project/experiment write token inside the public package (as planned in your comment) makes the new token check (routers/telemetry.py:36-43) ineffective. Anyone can extract it. Consider a dedicated ingestion endpoint that needs no secret, with strict schema validation and rate limiting, rather than a shared write token that looks like a secret. The same applies to extensive creating runs in a shared experiment through ApiClient.
Unknown coordinates are sent as 0,0 (collect.py:244-247, _round_coordinate_or_zero).
- This sends 0.0, 0.0 when the location is unknown, which pollutes the data. Send null instead, reusing _round_or_none as @inimaz suggested.
- Also, conf.get("longitude", emissions.longitude) returns None, and so 0.0, when the key exists with a None value.
Most telemetry is lost. It's sent on a daemon thread (dispatcher.py:61), which is killed at interpreter exit. That covers the common cases: a script ending, @track_emissions, or the CLI exiting right after stop(). Consider an atexit join with a short cap (≤1 s). In extensive mode, ApiClient also logs at ERROR/INFO from that background thread on offline machines. Lower those logs to DEBUG for telemetry.

Housekeeping:
7. Merge conflicts in codecarbon/cli/main.py and codecarbon/core/config.py. The config.py part has to be redone on top of #1389's get_hierarchical_config refactor (read_config_file, source logging), not just rebased.
8. The description is stale. It mentions send_telemetry and "once-per-session deduplication", but the code sends on every stop().

Nits:

_set_from_conf(level, "telemetry_level") puts telemetry_level into _conf, so it ends up in logged and exported config.
The Telemetry import inside __init__ is unnecessary.
_integration() sniffs sys.argv, which is fragile.
About 60 schema fields are duplicated on the client side. The drift test helps, but consider generating one from the other.

# Conflicts: # codecarbon/cli/main.py # codecarbon/core/config.py

- Revert the default experiment_id: telemetry no longer changes the experiment written to every CSV row or used by save_to_api. - stop() never blocks: the payload is built on the send thread, and cloud fields reuse what the tracker detected instead of probing the metadata endpoints again. - Opt-out by default, and ask: interactive `codecarbon config` / `monitor` ask once for a level and save it in ~/.codecarbon.config; non-interactive runs log an accurate notice once per machine (marker in ~/.codecarbon/). `telemetry status` says whether anything is actually sent. - Unknown coordinates are omitted instead of sent as 0,0; a config key holding None no longer masks the detected value. - Pending sends get an atexit join capped at 1 s; logs from the telemetry thread (ApiClient in extensive mode) drop to DEBUG. - telemetry_level no longer leaks into _conf; Telemetry imported at module level. - docs: list every field sent per tier, the prompt/notice flow and opt-out paths. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

davidberenstein1957 · 2026-09-23T07:58:22Z

Made the changes in cd8ccf6: opt-out with a one-time prompt/notice, stop() non-blocking, experiment_id reverted, null coords, atexit join, docs list every field. Merged master, redid config.py on top of get_hierarchical_config, refreshed the description.
Not done: /telemetry auth (a shipped token vs an open rate-limited endpoint is your call), the retention period for the docs, and the nits on argv sniffing and client/server schema generation.

davidberenstein1957 requested a review from a team as a code owner May 19, 2026 18:01

benoit-cty reviewed May 22, 2026

View reviewed changes

Comment thread codecarbon/cli/telemetry_cli.py Outdated

Base automatically changed from feat/telemetry-backend to master June 7, 2026 08:37

davidberenstein1957 linked an issue Jun 9, 2026 that may be closed by this pull request

Add internal and public telemetry #1106

Open

github-advanced-security Bot found potential problems Jun 15, 2026

View reviewed changes

Comment thread codecarbon/core/telemetry/settings.py Fixed

davidberenstein1957 requested a review from inimaz June 17, 2026 15:42

github-advanced-security Bot found potential problems Jun 17, 2026

View reviewed changes

Comment thread codecarbon/core/telemetry/settings.py Fixed

davidberenstein1957 changed the title ~~feat: wire minimal telemetry into emissions tracker~~ feat: add SDK telemetry into emissions tracker Jul 18, 2026

inimaz reviewed Jul 18, 2026

View reviewed changes

Comment thread codecarbon/emissions_tracker.py

inimaz reviewed Jul 18, 2026

View reviewed changes

inimaz reviewed Aug 4, 2026

View reviewed changes

github-actions Bot added the size/XL label Aug 19, 2026

davidberenstein1957 force-pushed the feat/add-telemetry branch from 506030a to e102d91 Compare August 19, 2026 15:21

davidberenstein1957 force-pushed the feat/add-telemetry branch from e102d91 to fd7a63b Compare August 20, 2026 06:21

davidberenstein1957 and others added 2 commits September 23, 2026 16:51

Merge remote-tracking branch 'origin/master' into HEAD

f0fffb3

# Conflicts: # codecarbon/cli/main.py # codecarbon/core/config.py

		raise typer.BadParameter(str(error)) from error


		def resolve_config_path(config: Optional[Path], *, create: bool = False) -> Path:

		return importlib.util.find_spec(name) is not None


		def _round_coordinate(value: Any) -> float \| None:

Uh oh!

Conversation

davidberenstein1957 commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Details

Open for maintainers

Types of changes

AI Usage Disclosure

Checklist:

Uh oh!

codecov Bot commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davidberenstein1957 commented Jun 17, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

inimaz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benoit-cty commented Sep 23, 2026

Verdict: 🔧 Request changes

Uh oh!

davidberenstein1957 commented Sep 23, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

davidberenstein1957 commented May 19, 2026 •

edited

Loading

codecov Bot commented May 19, 2026 •

edited

Loading