Notified-COMM: [BUG FIX]: Fixing the failure path check and error handling - #32
Open
joe-explr wants to merge 5 commits into
Open
Notified-COMM: [BUG FIX]: Fixing the failure path check and error handling#32joe-explr wants to merge 5 commits into
joe-explr wants to merge 5 commits into
Conversation
|
Hello! The Git Commit Checker CI bot found a few problems with this PR: d477d4f: Fix MPI_WIN_SET_NUM_NOTIFY hang on an invalid argu...
Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks! |
1 similar comment
|
Hello! The Git Commit Checker CI bot found a few problems with this PR: d477d4f: Fix MPI_WIN_SET_NUM_NOTIFY hang on an invalid argu...
Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks! |
MPI_WIN_SET_NUM_NOTIFY is a blocking, synchronizing collective, but its num_notifications argument is local: MPI-5.1 section 12.6.1 states that the number of notification counters "can be different for different MPI processes". Both the C binding and osc/sm validated that argument and returned early, before the osc module's internal allgather. A single rank passing a bad value therefore returned an error while every other rank stayed blocked in that allgather forever, turning an erroneous argument into a hang. The binding rejected negative counts, and osc/sm additionally rejected counts above an mpi_assert_max_num_notify assertion -- a case the binding never covered, so osc/sm could hang even before this change. Drop the range check from the binding and carry each rank's verdict through the collective instead. osc/sm gathers ULONG_MAX as a sentinel that no legal count can collide with, since valid counts come from an int and never exceed INT_MAX; a single-process window has nobody to agree with and still answers immediately. All ranks then see the same gathered array and fail identically, so the window cannot end up half-reconfigured. osc/ucx already carried its verdict through the allgather and needed no change. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
The previous fix computed "bad" but never sent the ULONG_MAX sentinel through the allgather, leaving the agree: label unused. Only -1 hit the sentinel, by accident of the cast; -5 became a huge request and crashed in the grow path, and a request above mpi_assert_max_num_notify grew instead of failing. Also correct the comment that called the assertion not a limit. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
Test the broadcast descriptor with the shmem framework's validity flag instead of peeking at seg_name. segment_create sets the flag only on success, and the zeroed descriptor rank 0 sends when it cannot create the segment has it clear, so every rank still fails together. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
Removing unecesaary comments Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
joe-explr
force-pushed
the
notified-rma-sm
branch
from
September 11, 2026 19:08
ddb4b23 to
e88846a
Compare
The tests expects the counter to grow past the set max_assert_num_notify value. This commit fixes the comments and the code to cap the growth at the flag value. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
joe-explr
force-pushed
the
notified-rma-sm
branch
from
September 11, 2026 21:10
df83014 to
cece024
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes are made to address the following
When set num notify is passed a bad arg, its erroneous to return early as it will block and hang other ranks. The changes target returning the error after the synchronizing call.
The failure check for segment creation was weak and erroneous.
if ('\0' == new_seg_ds.seg_name[0]) {has a possibility flagging a successful segment creation as a failure. Replaced it with OPAL_SHMEM_DS_IS_VALID(&new_seg_ds)