Skip to content

Fix ESMF read casting elementConn before checking for padding - #1746

Draft
cmdupuis3 wants to merge 1 commit into
mainfrom
cmd/esmf_read_fill_detection
Draft

Fix ESMF read casting elementConn before checking for padding#1746
cmdupuis3 wants to merge 1 commit into
mainfrom
cmd/esmf_read_fill_detection

Conversation

@cmdupuis3

@cmdupuis3 cmdupuis3 commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Closes #1747

Overview

_read_esmf casts elementConn to INT_DTYPE and only then compares against INT_FILL_VALUE, so the padding is recognized in neither of the two ways xarray can hand it over. With CF decoding on (mask_and_scale=True, the default), xarray replaces ESMF's -1 with NaN and promotes the array to float; casting NaN to an integer dtype is platform-dependent, and on arm64 it yields 0, so the padding decodes to -1. With decoding off, the raw -1 survives the cast and decodes to -2. Both are negative indices that silently wrap to the end of the coordinate arrays instead of being treated as fill.

The fix locates the padding before the cast, from numElementConn — the format's own statement of each face's node count — and additionally treats NaN as padding whatever that count claims.

The test builds the ESMF dataset by hand instead of going through _encode_esmf, so it exercises the reader in isolation and covers third-party ESMF files rather than only our own output. It is parametrized over mask_and_scale to pin both decoding paths.

PR Checklist

General

  • An issue is created and linked
  • Added appropriate labels (if your uxarray repo permissions allow it)
  • Filled out Overview and Expected Usage (if applicable) sections

Testing & Benchmarking

  • There is adequate test coverage of changes from this PR (add new tests if needed)

Documentation and Examples

  • Docstrings updated with any function changes, and included in all new functions
  • User (public) functions added to docs/api.rst; internal (private) function names start with an underscore (_)

AI Disclosure

AI Usage: Claude Opus 5

  • I have tested and take responsibility for all AI-generated content in my PR.

`_read_esmf` cast `elementConn` to INT_DTYPE and only then compared against
INT_FILL_VALUE, so the padding was never recognized:

  - decoded (`mask_and_scale=True`), xarray turns ESMF's -1 into NaN and the
    cast is platform-dependent -- arm64 gives 0, so the padding decodes to
    -1, a negative index that silently wraps to the last node
  - undecoded, the raw -1 survives the cast and decodes to -2

Locate the padding before the cast, from `numElementConn`, which states each
face's size directly; also treat NaN as padding whatever that count claims.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@cmdupuis3 cmdupuis3 self-assigned this Sep 10, 2026
@cmdupuis3 cmdupuis3 added bug Something isn't working accuracy Related to mathematical accuracy of operations, such as numerical precision or rounding errors labels Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accuracy Related to mathematical accuracy of operations, such as numerical precision or rounding errors bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix ESMF read casting elementConn before checking for padding

1 participant