feat: warn when SLURM ranks double count a node

[davidberenstein1957]

Description

On SLURM, srun --ntasks-per-node=4 starts four ranks on one node. With tracking_mode="machine" (the default), each of those four trackers measures the whole node, so the job's reported total is silently 4x the true value and nothing in the output flags it. This PR adds codecarbon/core/slurm.py, which runs at tracker init and, in machine mode on a non-zero SLURM_LOCALID, logs a warning naming two ways out: start the tracker only on SLURM_LOCALID == 0, or switch to tracking_mode="process". The message includes the step's tasks-per-node (SLURM_STEP_TASKS_PER_NODE, falling back to SLURM_TASKS_PER_NODE). Local rank 0, and the batch step where SLURM_LOCALID is unset, never warn. Split out of #1366, which mixed this in with unrelated job-metadata-column work.

Related Issue

N/A

Motivation and Context

Multi-rank SLURM jobs using the default machine-tracking mode silently overcount emissions by the rank count, and nothing in the current output tells the user this is happening. A warning at init lets users catch and correct the setup before trusting the numbers.

How Has This Been Tested?

tests/test_slurm.py: warns on non-zero local rank (step and job tasks-per-node vars, 4(x2)); silent for local rank 0, the batch step, no SLURM env, and tracking_mode="process"; one test through OfflineEmissionsTracker.__init__.

Screenshots (if appropriate):

N/A

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

AI Usage Disclosure

• 🟥 AI-vibecoded: You cannot explain the logic. Car analogy : the car drive by itself, you are outside it and just tell it where to go.
• 🟠 AI-generated: Car analogy : the car drive by itself, you are inside and give instructions.
• ⭐ AI-assisted. Car analogy : you drive the car, AI help you find your way.
• ♻️ No AI used. Car analogy : you drive the car.

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the docs/how-to/contributing.md document.
• I have added tests to cover my changes.
• All new and existing tests passed.

Not in this PR

Automatic rank election (e.g. suppressing extra trackers) is deferred; it is the same problem as LOCAL_RANK == 0 in distributed training and deserves one generic solution, not a SLURM-specific one. This PR is the warning only.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.71%. Comparing base (3ec31a0) to head (bceb640).
⚠️ Report is 41 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1390      +/-   ##
==========================================
+ Coverage   91.43%   91.71%   +0.28%     
==========================================
  Files          49       50       +1     
  Lines        5057     5169     +112     
==========================================
+ Hits         4624     4741     +117     
+ Misses        433      428       -5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[benoit-cty]

🤖 This review comment was written and posted by Claude Opus 5.5 (AI assistant), at the request of @benoit-cty. Findings were checked by reading the code and running tests locally (merged with current master where relevant), but please double-check before acting on them.

Verdict: 🔧 Request changes

The hazard is real. allow_multiple_runs defaults to True (emissions_tracker.py:163) and the lock file is in per-node /tmp, so nothing stops N trackers per node from each counting the whole machine. The detection needs work, though.

Must fix:

1. Wrong variable, so the common layouts are missed (codecarbon/core/slurm.py:16).
   • SLURM_NTASKS_PER_NODE is only set when --ntasks-per-node is passed explicitly. sbatch -N2 -n8 or --ntasks=8 alone leaves it unset, so the warning never fires.
   • The 4(x2) format that the code and tests/test_slurm.py:17 parse belongs to SLURM_TASKS_PER_NODE / SLURM_STEP_TASKS_PER_NODE, which are always set.
   • Please read SLURM_STEP_TASKS_PER_NODE, falling back to SLURM_TASKS_PER_NODE.
2. The warning fires even when the user follows its advice (slurm.py:14-19).
   • The check is env-only, and every task inherits the env. So a user who starts the tracker only on SLURM_LOCALID == 0, as the message recommends, still gets the warning.
   • The only way to silence it is tracking_mode="process", which measures something different.
   • Suggested fix: warn only when SLURM_LOCALID is set and non-zero. A machine-mode tracker on a non-zero local rank is almost certainly a duplicate. The recommended pattern then never warns, and a naive "tracker on every rank" setup still warns.
   • Also reword the message to "if more than one rank per node starts a tracker…", and consider an opt-out (config key or env var).
3. False positive in the batch step. SLURM_NTASKS_PER_NODE is also exported into the sbatch batch script. A single tracker in the batch step (e.g. wrapping torchrun or an inner srun) warns although nothing is double counted. The step-scoped variables from point 1 avoid this.

Low:
4. Log spam. The warning is emitted from __init__ (emissions_tracker.py:604) once per tracker per rank, so an N-rank job logs it N times. The SLURM_LOCALID != 0 approach from point 2 reduces that to at most the duplicate ranks.

Tests:

• The tests only cover the regex and the mode gate. Please add cases for SLURM_TASKS_PER_NODE / SLURM_STEP_TASKS_PER_NODE, SLURM_LOCALID 0 vs non-zero, and one test that goes through EmissionsTracker.__init__.
• The description says "Based on fix: drop dtype coercion in CSV update path #1370", but the diff contains only the SLURM commit. Please remove that line.

[davidberenstein1957]

Made the changes in bceb640: warn only on non-zero SLURM_LOCALID, read step tasks-per-node, reworded message, added tracker-init test. Dropped the #1370 line from the description.
Skipped the opt-out key: with the LOCALID check, following the advice already silences it.