Skip to content

fix: restore signal handlers on lock release - #1329

Open
davidberenstein1957 wants to merge 3 commits into
masterfrom
fix/lock-signal-handler-restore
Open

davidberenstein1957 wants to merge 3 commits into
masterfrom
fix/lock-signal-handler-restore

Conversation

@davidberenstein1957

@davidberenstein1957 davidberenstein1957 commented Aug 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

codecarbon/lock.py now saves the signal handlers replaced by signal.signal() in self._previous_handlers (main thread only, once the lock file is acquired) instead of discarding them. _handle_exit releases the lock and then delegates to the handler it replaced: it calls a previous Python handler, does nothing for SIG_IGN, and for the default disposition raises SystemExit(128 + signum) or KeyboardInterrupt for SIGINT, so finally / __exit__ / @track_emissions still write the final emissions. release() removes the lock file first, then restores the saved handlers, but only when the currently installed handler is still the one it set, so an application that registered its own handler afterward is not clobbered; restoring is skipped off the main thread, where signal.signal() raises, so stop() from a worker thread still writes its CSV and removes the lock file. The internal thread lock is now an RLock, since _handle_exit can call release() on a thread already holding it.

Related Issue

Fixes #1310

Motivation and Context

Lock is constructed whenever allow_multiple_runs=False. Because it overwrote the process signal disposition permanently, restoring the default SIGINT handler never happened, so KeyboardInterrupt stopped being raised at all — every except KeyboardInterrupt: in an embedding application became dead code, including CodeCarbon's own in codecarbon/cli/monitor.py. Applications that had registered a graceful-shutdown SIGTERM handler also lost it silently.

Behavior change: previously any SIGINT/SIGTERM ended in SystemExit(1). Now the process does whatever the application asked for; with no application handler, SIGTERM exits with the conventional code 143 (after cleanup) and SIGINT raises KeyboardInterrupt.

How Has This Been Tested?

tests/test_lock.py (TestLockSignalHandlers) covers: handlers restored on release, forwarding to a previous Python handler, SystemExit(143) on SIGTERM with the default disposition, SIG_IGN kept, release from a worker thread, a None previous handler, no deadlock on re-entrant release, and a failed acquire() leaving handlers alone.

Screenshots (if appropriate):

N/A

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

AI Usage Disclosure

  • 🟥 AI-vibecoded
  • 🟠 AI-generated
  • ⭐ AI-assisted
  • ♻️ No AI used

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the docs/how-to/contributing.md document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@codecov

codecov Bot commented Aug 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.28571% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.73%. Comparing base (e5e46ab) to head (f132178).

Files with missing lines Patch % Lines
codecarbon/lock.py 89.28% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1329      +/-   ##
==========================================
+ Coverage   91.70%   91.73%   +0.03%     
==========================================
  Files          49       49              
  Lines        5157     5178      +21     
==========================================
+ Hits         4729     4750      +21     
  Misses        428      428              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@davidberenstein1957
davidberenstein1957 marked this pull request as ready for review August 12, 2026 19:14
@davidberenstein1957
davidberenstein1957 requested a review from a team as a code owner August 12, 2026 19:14
@davidberenstein1957
davidberenstein1957 force-pushed the fix/lock-signal-handler-restore branch from 94541ee to 984f4b7 Compare August 19, 2026 14:16
`Lock` installed SIGINT/SIGTERM handlers and threw away the previous ones, so
the host application's handlers were destroyed and Ctrl-C stopped raising
KeyboardInterrupt. Save the previous handlers, chain to them from
`_handle_exit`, and restore them in `release()`, unregistering the atexit hook
so a released lock is not pinned.

Handlers are installed in `acquire()` after `open(LOCKFILE, "x")` succeeds,
not in `__init__`. On the "another instance is already running" path
`acquire()` raises, the tracker sets `_another_instance_already_running`, and
`stop()` returns at its early guard without ever reaching `release()` -- so
handlers installed in the constructor stayed hijacked for the life of the
process.

The thread lock is reentrant: `_handle_exit` calls `release()`, which takes
`_thread_lock`, so a signal delivered while the same thread was inside
`acquire()`/`release()` deadlocked on a plain `Lock`. `release()` is also
idempotent now (moved here from #1336, since it edits the same few lines this
branch already rewrites), and the `_atexit_hook` indirection is dropped:
`atexit.unregister()` compares with `==`, not identity, so a bound method
unregisters fine.

Tests cover the default and ignored signal dispositions, and the deadlock test
unregisters its atexit hook so a reverted lock.py fails the suite instead of
wedging the interpreter at exit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@davidberenstein1957
davidberenstein1957 force-pushed the fix/lock-signal-handler-restore branch from 984f4b7 to 004a7e3 Compare August 20, 2026 06:14
@benoit-cty

Copy link
Copy Markdown
Contributor

🤖 This review comment was written and posted by Claude Opus 5.5 (AI assistant), at the request of @benoit-cty. Findings were checked by reading the code and running tests locally (merged with current master where relevant), but please double-check before acting on them.

Verdict: 🔧 Request changes

The bug is real. Lock permanently took over SIGINT/SIGTERM and always raised SystemExit(1). The fix saves the previous handlers, restores them on release, forwards the signal to them, and uses an RLock.

  • Nested trackers are fine: a second lock never acquires, so it installs no handlers.
  • Handlers the user sets after acquire() are kept (the getsignal check).

Must fix:

  1. Calling release() off the main thread loses the final emissions and leaves the lock file behind (codecarbon/lock.py L71-75).
    • signal.signal() only works on the main thread, and release() now calls it with no thread check, so it raises ValueError.
    • stop() calls _lock.release() before saving anything. @suppress swallows the exception, so the rest of stop() never runs.
    • I verified this: with OfflineEmissionsTracker(allow_multiple_runs=False) and stop() called from a worker thread, master writes the CSV and removes the lock. With this PR, stop() returns None, no CSV is written, and the lock file stays until the process exits.
    • Stopping from a callback or worker thread is a common pattern.
    • Fix: only restore handlers when threading.current_thread() is threading.main_thread(). Remove the lock file first, and wrap the restore in try/except. Please add a test that stops from a worker thread.
  2. SIGTERM now kills the process with no final write (lock.py L42-43).
    • When the previous SIGTERM handler was SIG_DFL, the PR restores it and re-sends the signal with os.kill. The process then dies immediately: no finally, no __exit__, no tracker.stop().
    • On master, SystemExit(1) let with / @track_emissions write the final emissions. I verified this: master exits with code 1 and finally runs; the PR exits with 143 and finally does not run.
    • On SLURM/Kubernetes shutdowns this loses the final row that users get today.
    • Suggested fix: forward only to callable previous handlers. When the previous handler is SIG_DFL, raise SystemExit(128 + signum) (or KeyboardInterrupt for SIGINT), so cleanup runs and the exit code stays conventional.

Nits:

  • If the previous handler was installed from C code, signal.getsignal() returns None. release() then calls signal.signal(sig, None), which raises TypeError, and _handle_exit swallows the signal. Treat None as SIG_DFL.
  • The description refers to an _atexit_hook that isn't in the code, and the ordering note about stop idempotency is already covered by fix: make tracker.stop() idempotent #1408. Please update it.

davidberenstein1957 and others added 2 commits September 23, 2026 16:48
- release() removes the lock file first and only restores signal handlers
  on the main thread, so stop() from a worker thread still writes its CSV.
- With a default previous handler, raise SystemExit(128 + signum) (or
  KeyboardInterrupt for SIGINT) instead of re-sending the signal, so
  finally / __exit__ still write the final emissions.
- Treat a None previous handler (installed from C) as SIG_DFL.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@davidberenstein1957

Copy link
Copy Markdown
Collaborator Author

Made the changes in f132178: worker-thread release and SIGTERM cleanup fixed.

  • release() removes the lock file first and only restores handlers on the main thread (new worker-thread test).
  • A default previous handler now raises SystemExit(128 + signum) / KeyboardInterrupt instead of os.kill, so finally and the final CSV row still happen (checked: exit 143, row written).
  • A None previous handler is restored as SIG_DFL. Updated the description too.

@github-actions github-actions Bot added size/L and removed size/M labels Sep 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lock permanently replaces the host application's SIGINT/SIGTERM handlers

2 participants