Skip to content

Fix Zarr fill-value masking options for DataTree reads - #11663

Open
anshurajbisoyi98-ctrl wants to merge 22 commits into
pydata:mainfrom
anshurajbisoyi98-ctrl:fix-11361-datatree-fill-value-mask
Open

anshurajbisoyi98-ctrl wants to merge 22 commits into
pydata:mainfrom
anshurajbisoyi98-ctrl:fix-11361-datatree-fill-value-mask

Conversation

@anshurajbisoyi98-ctrl

Copy link
Copy Markdown

Description

Fixes an issue where use_zarr_fill_value_as_mask was not accepted when reading Zarr data through the DataTree and group-opening APIs.

The option already controls fill-value masking in the Zarr store, so I forwarded it through open_datatree() → open_groups_as_dict() → ZarrStore.open_store(). This lets DataTree reads reuse the existing masking behaviour without adding a second implementation. The chosen setting now applies consistently to the root and child groups, and the format-dependent defaults are unchanged when no setting is passed or it is set to None.

Cause: The Zarr DataTree and group-opening methods did not accept or forward use_zarr_fill_value_as_mask.

Tests: I added 12 regression cases covering both opening APIs, Zarr formats 2 and 3, True/False/None, and root and child groups. All 12 failed before the fix and passed afterward.

Additional change: A separate commit fixes existing NamedArray masked-array typing errors and adds three unmasked-array test cases.

Docs: Updated the I/O guide and the release notes.

Validation:

  • Zarr/DataTree suites: 1,751 passed, 146 skipped, 20 expected failures.
  • After the typing fix, NamedArray and Zarr DataTree tests: 105 passed, 1 skipped, 3 expected failures.
  • Mypy passed for 203 files, and the standard hooks passed.

Not verified: full-project tests, other platforms, and remote CI.

Checklist

AI Disclosure

  • This PR contains AI-generated content.
    • I have tested any AI-generated content in my PR.

    • I take responsibility for any AI-generated content in my PR.

      Tools: Codex (used for investigation, implementation, tests, verification, documentation, and wording edits to this description).

      Prompt summary: Address add use_zarr_fill_value_as_mask in open_datatree for Zarr Stores #11361 using established repository patterns, add regression tests, verify the changes locally, and fix the remaining masked-array typing errors before submission.

Signed-off-by: Anshu Raj Bisoyi <anshurajbisoyi98@gmail.com>
Signed-off-by: Anshu Raj Bisoyi <anshurajbisoyi98@gmail.com>
@github-actions github-actions Bot added topic-backends topic-zarr Related to zarr storage library io topic-NamedArray Lightweight version of Variable labels Oct 3, 2026
@anshurajbisoyi98-ctrl

Copy link
Copy Markdown
Author

The two failing checks appear to report the same six PyDAP network-test timeouts: test-py314 (flaky) and its aggregated Test Results check. These tests access test.opendap.org, which was also unreachable during investigation. The other 13 test jobs passed, and all 12 Zarr masking regression cases pass locally.
Could a maintainer rerun the failed job to check whether this was a transient network issue?

@anshurajbisoyi98-ctrl

Copy link
Copy Markdown
Author

The two remaining failures are test_batchdap4_downloads[dap2] and test_batchdap4_downloads[dap4] in the flaky/network test job. Both fail while reaching test.opendap.org (HTTP 503 / connection refused); I also confirmed the endpoint is currently returning 503. The other platform test jobs, typing checks, docs, and pre-commit checks passed. These failures appear to be caused by the external OPeNDAP service outage, rather than this PR. Could you please advise whether you would prefer a rerun once the service is available?

@headtr1ck headtr1ck left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Comment thread xarray/tests/backends/test_datatree.py Outdated
anshurajbisoyi98-ctrl and others added 4 commits October 17, 2026 00:27
Removed skip marker from TestPyDAPDatatreeIO due to 403 error.
The merge of main dropped the whats-new entries of pydata#11736 and pydata#11722.
Combine this PR's entry with the one of pydata#11748, which fixed the same
option for open_dataset, and mention all openers in the user guide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@headtr1ck headtr1ck left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took the liberty to fix up the failed what's -new merge.
Looking good!

@headtr1ck headtr1ck added plan to merge Final call for comments and removed io topic-NamedArray Lightweight version of Variable labels Oct 17, 2026
@anshurajbisoyi98-ctrl

Copy link
Copy Markdown
Author

Thanks a lot for fixing the whats-new merge, and sorry, I hadn't noticed that it dropped the #11736 and #11722 entries. Combining the entry with #11748 makes sense.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

plan to merge Final call for comments topic-backends topic-zarr Related to zarr storage library

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add use_zarr_fill_value_as_mask in open_datatree for Zarr Stores

2 participants