Skip to content

fix: raise on offline tracker configuration errors - #1334

Open
davidberenstein1957 wants to merge 3 commits into
masterfrom
fix/suppress-constructor-failures
Open

davidberenstein1957 wants to merge 3 commits into
masterfrom
fix/suppress-constructor-failures

Conversation

@davidberenstein1957

@davidberenstein1957 davidberenstein1957 commented Aug 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Removes @suppress(Exception) from OfflineEmissionsTracker.__init__ in codecarbon/emissions_tracker.py, so configuration errors raised during construction propagate to the caller instead of being swallowed.

Related Issue

Fixes #1311

Motivation and Context

The decorator swallowed configuration errors raised during construction, most visibly the OSError from _set_from_conf when output_dir does not exist. The constructor then returned an object that had never reached _initialize_runtime_state() / _initialize_scheduler_state(), so it had no _start_time, _hardware, _scheduler, or _output_handlers. The subsequent start() and stop() calls are themselves suppressed, so they degraded into AttributeError warnings, and the run finished with no emissions file and no exception. Under the CLI default log_level="error", nothing was printed at all. The online EmissionsTracker already raises for the same input, so this also removes an asymmetry between the two constructors. Suppression on start, flush, and stop is deliberately left in place, since runtime measurement errors must never crash a user job; only construction, where the object invariants were never established, now fails loudly. This is a behavior change at a public boundary and is intended to ride a minor release (3.4.0), not a patch.

How Has This Been Tested?

Added test_offline_tracker_raises_on_invalid_output_dir in tests/test_offline_emissions_tracker.py, asserting both the offline and the online tracker raise OSError for a non-existent output_dir. It fails on master and passes with this change. A second test, test_offline_tracker_raises_on_invalid_region, covers the offline-only region check, which runs before super().__init__ and was suppressed by a different code path than the output_dir OSError. Both new tests fail if @suppress(Exception) is restored on the constructor.

uv run pytest tests/test_offline_emissions_tracker.py tests/test_emissions_tracker.py -q
35 passed

Screenshots (if appropriate):

N/A

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

AI Usage Disclosure

  • 🟥 AI-vibecoded
  • 🟠 AI-generated
  • ⭐ AI-assisted
  • ♻️ No AI used

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the docs/how-to/contributing.md document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

CHANGELOG.md does not exist on master yet, so the release note lives in docs/reference/api.md under "Constructor errors (changed in v3.4.0)": OfflineEmissionsTracker no longer swallows exceptions raised while constructing the tracker; configuration errors (e.g. a non-existent output_dir) now propagate to the caller, matching EmissionsTracker. Suppression on start/flush/stop is unchanged.

@codecov

codecov Bot commented Aug 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.25000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.72%. Comparing base (3ec31a0) to head (1e238ba).
⚠️ Report is 41 commits behind head on master.

Files with missing lines Patch % Lines
codecarbon/emissions_tracker.py 81.25% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1334      +/-   ##
==========================================
+ Coverage   91.43%   91.72%   +0.29%     
==========================================
  Files          49       49              
  Lines        5057     5163     +106     
==========================================
+ Hits         4624     4736     +112     
+ Misses        433      427       -6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@davidberenstein1957
davidberenstein1957 marked this pull request as ready for review August 12, 2026 19:14
@davidberenstein1957
davidberenstein1957 requested a review from a team as a code owner August 12, 2026 19:14
@davidberenstein1957
davidberenstein1957 force-pushed the fix/suppress-constructor-failures branch from 85eedc8 to 3d7c6ec Compare August 19, 2026 09:18
@davidberenstein1957
davidberenstein1957 force-pushed the fix/suppress-constructor-failures branch from 3d7c6ec to df16736 Compare August 19, 2026 14:29
`@suppress(Exception)` on `OfflineEmissionsTracker.__init__` swallowed
configuration errors such as a missing `output_dir`, returning a half-built
object with no `_start_time`, `_hardware` or `_scheduler`. `start()` and
`stop()` then failed silently and no emissions were recorded at all.

Construction now raises, matching `EmissionsTracker`. The suppression on
`start`/`flush`/`stop` is kept, so runtime measurement errors still cannot
crash a user's job. The behaviour change is noted in the docs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@davidberenstein1957
davidberenstein1957 force-pushed the fix/suppress-constructor-failures branch from df16736 to 7b8487b Compare August 20, 2026 06:14
@benoit-cty

Copy link
Copy Markdown
Contributor

🤖 This review comment was written and posted by Claude Opus 5.5 (AI assistant), at the request of @benoit-cty. Findings were checked by reading the code and running tests locally (merged with current master where relevant), but please double-check before acting on them.

Verdict: 💬 Needs a maintainer decision (leaning approve, for a minor release)

The bug is real. A bad output_dir returns a half-built OfflineEmissionsTracker that silently records nothing, and the new tests fail on master. Removing @suppress(Exception) from OfflineEmissionsTracker.__init__ (emissions_tracker.py:1312) matches the online EmissionsTracker, which already raises. start, stop and flush stay suppressed.

Behaviour change to be aware of:

  • CodeCarbon's contract has been "never crash user code". With this PR:
    • @track_emissions(offline=True, output_dir="missing") builds the tracker inside wrapped_fn, so the decorated function never runs and the exception reaches the user's training job. Before, the job ran untracked.
    • with OfflineEmissionsTracker(...) now raises too.
  • Suggestion for keeping both behaviours: keep direct construction strict (this PR), but in track_emissions catch construction errors, log them at ERROR, and run the wrapped function untracked.
  • The docs note says "changed in v3.4.0" (latest tag is v3.3.1), so this must go into a minor release, not a patch. Please also add a changelog entry.

Nit:

  • test_offline_tracker_raises_on_invalid_region expects an AssertionError from an assert (emissions_tracker.py ~L1344). Under python -O, asserts are stripped and nothing is raised. Please raise ValueError explicitly.

Asserts are stripped under python -O, so a non-string region or
country_2letter_iso_code would pass silently.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@davidberenstein1957

Copy link
Copy Markdown
Collaborator Author

Made the changes in e77657f: region / country_2letter_iso_code type checks now raise ValueError instead of assert.

Not done, pending a maintainer decision: whether track_emissions should catch construction errors and run the function untracked, and which release this lands in. No changelog file exists in the repo, so the entry is left for the release notes.

track_emissions now logs construction errors and calls the function
without tracking, matching start_task. Direct construction still raises.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@davidberenstein1957

Copy link
Copy Markdown
Collaborator Author

Made the changes in 1e238ba: @track_emissions now logs construction errors and runs the function untracked, same as start_task. Direct construction still raises. Targeting a minor release (v3.4.0).

@github-actions github-actions Bot added size/L and removed size/S labels Sep 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OfflineEmissionsTracker silently returns a half-built object when construction fails

2 participants