cuda-bindings 13.3's cufile is incompatible with CTK 13.4

[mdboom]

Based on @rwgk agent's assessment (below):

cufile.h increased the size of the CUfileStatsLevel1_t struct significantly.

@@ -978,6 +1056,20 @@
     uint64_t batch_completion_lat_sum_us; // Sum of batch completion latencies
     uint64_t last_batch_read_bytes;      // Last batch read bytes
     uint64_t last_batch_write_bytes;     // Last batch write bytes
+
+    // Vector IO (readv/writev) statistics — independent of read/write counters above
+    CUfileOpCounter_t readv_ops;         // cuFileReadv operations
+    CUfileOpCounter_t writev_ops;        // cuFileWritev operations
+    uint64_t readv_bytes;                // Total bytes read via cuFileReadv
+    uint64_t writev_bytes;               // Total bytes written via cuFileWritev
+    uint64_t readv_bw_bytes_per_sec;     // cuFileReadv bandwidth (bytes/sec)
+    uint64_t writev_bw_bytes_per_sec;    // cuFileWritev bandwidth (bytes/sec)
+    uint64_t readv_lat_avg_us;           // Average cuFileReadv latency (microseconds)
+    uint64_t writev_lat_avg_us;          // Average cuFileWritev latency (microseconds)
+    uint64_t readv_ops_per_sec;          // cuFileReadv operations per second
+    uint64_t writev_ops_per_sec;         // cuFileWritev operations per second
+    uint64_t readv_lat_sum_us;           // Sum of cuFileReadv latencies
+    uint64_t writev_lat_sum_us;          // Sum of cuFileWritev latencies
 } CUfileStatsLevel1_t;

 /**

When cuda-bindings 13.3 allocates one of these structs, it uses the older, smaller size. When passing this as a pointer to cuFileGetStatsL1, it writes past the end of the buffer, leading to a segfault.

The test that reproduces the issue

Since CUfileStatsLevel1_t is used as a member of CUfileStatsLevel2_t this issue also exists there.

Unfortunately, I think this is ABI breakage in cufile.

We should discuss a workaround: This is not a blocker for the 13.4.1 release of cuda_bindings -- it should work correctly with /both/ CTK 13.3 and 13.4 (at the expense of over-allocation). But the combination of our released cuda-bindings 13.3 with CTK 13.4 has this bug.

====

@rwgk's agent's research:

The six substantive failures are one issue, not six unrelated failures—and they are not caused by PR #2792.

High — 13.3/13.4 cuFile ABI mismatch

The “CUDA 13.3.0 (wheels)” jobs actually installed cuda-toolkit 13.4.1.0 and nvidia-cufile 1.19.0.109.
That happens because run-tests installs the bindings wheel with [all], whose dependencies allow any 13.x Toolkit in pyproject.toml.
The wheel itself was built from 13.3-generated bindings. Its CUfileStatsLevel1_t ends after last_batch_write_bytes in cycufile.pxd.
CUDA 13.4 adds twelve readv_/writev_ fields to that structure, visible in the 13.4 checkout. That enlarges it by 112 bytes.
StatsLevel1 allocates exactly sizeof(CUfileStatsLevel1_t) in cufile.pyx, while cuFileGetStatsL1 receives only a pointer—no caller-buffer size.
The strongest explanation is therefore:

get_stats_l1() writes the expanded 13.4 result into a smaller 13.3 allocation at test_cufile.py.
The first, unchanged fields remain readable and their assertions pass.
The memory corruption surfaces at the subsequent stats_stop() call on line 1653.
The Python 3.15 native trace explicitly ends in libcufile.so.0 at cuFileStatsStop+0x12c. This mechanism remains a strong inference rather than core-dump proof, but the controls are unusually convincing:

The same-run local-Toolkit 13.3 job uses libcufile.so.1.18.0; all L1/L2/L3 tests pass.
The corresponding native-Linux 13.4 wheel jobs in PR #2789, built with the enlarged structure, install the exact same 13.4.1 packages and all pass. Here is one passing control job.
Failure inventory

Run 34397567270 completed with 102 jobs:

95 passed.
Six native-Linux wheel jobs aborted identically: x86-64 Python 3.10, 3.12, 3.15, 3.15t, and ARM64 Python 3.10 and 3.12.
The seventh failure, Check job status, is only the aggregate reporting those Linux workflow failures.
All builds, Windows tests, sdists, docs, API checks, and pre-commit passed.
The two Python 3.15 failures are experimental via continue-on-error, but the four stable-Python failures are blocking.
WSL passes because cuFile tests are explicitly skipped there.
PR #2792 changes only three cuda-core/release-checklist files, so none of this originates in its diff.

Implications

This is more than misleading CI labeling: the released v13.3.1 bindings have the same cuda-toolkit==13.* metadata, while the documented support policy promises minor-version compatibility. Users pinning 13.3 bindings can now resolve the same unsafe 13.4 cuFile combination.

A plain rerun will resolve the same packages and fail again. For #2792’s intended “land before #2789” ordering, the practical choices are to merge it with the known unrelated failure or first land a narrowly scoped CI/package-constraint fix. Separately, the 13.3-bindings/13.4-cuFile compatibility issue deserves release-team attention; merely pinning CI would hide rather than resolve that user-facing problem.

I made no local changes, reruns, or GitHub mutations.