FIX: Enforce bundled native compatibility in Conda packages

[Jahnvi Thakkar (jahnvi480)]

Work Item / Issue Reference

AB#47315

Summary

This pull request introduces significant improvements to the Conda packaging and validation workflow for mssql-python, clarifies the installation and release process for Conda candidates, and strengthens platform compatibility and auditing. The most important changes are grouped below.

Conda Packaging and Platform Compatibility

• The Conda recipe now requires MSSQL_PYTHON_VERSION to be explicitly set, preventing accidental builds with the wrong code version. The combined Conda package now includes both the code and ODBC payload, eliminating the need for a separate ODBC package as required by pip installations.
• The Linux __glibc dependency is raised to >=2.34 to ensure compatibility with the full native payload, including the bulk-copy core.
• On macOS, the build script now explicitly matches macosx wheels to avoid accidentally using Linux wheels in cross-build scenarios.
• For Windows ARM64, the Conda requirements add a python_abi pin for Python 3.12 to ensure ABI compatibility, and the documentation clarifies that dependencies resolve from the defaults channel without automatic terms acceptance.

Validation and Auditing Enhancements

• The Conda build scripts are updated to never silently drop the bulk-copy core: builds now fail if the required native extension is missing or incompatible, ensuring no silent reduction in functionality.
• The audit logic now applies PE machine-type validation to both win-64 and win-arm64 packages, and always uses the actual target subdir for checks.
• The build environment no longer sets or relies on automatic ToS acceptance for Windows ARM64, and the related environment variable is removed for clarity and compliance.

Testing and CI Improvements

• The CI pipeline now provisions the testuser login with VIEW SERVER PERFORMANCE STATE permission, enabling pool eviction tests to observe physical connections in SQL Server 2022 and 2025 jobs.

Documentation Updates

• The main README.md and a new conda/README.md clarify the temporary nature of the combined Conda candidate, its installation instructions, platform requirements, and that these changes do not constitute a public release or certification.

Validation Logic Hardening

• The Python validation logic now includes an additional check to ensure the required native extension for bulk copy (mssql_py_core) is present and loaded from the correct environment, with improved error messaging and isolation.

These changes collectively improve the reliability, clarity, and platform compliance of the Conda packaging and validation process for mssql-python.

[Copilot]

🟡 Changes recommended

Three moderate packaging and validation gaps remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR hardens Conda packaging and native compatibility for bundled ODBC and bulk-copy components.

Changes:

• Adds ELF, PE, Mach-O, ABI, glibc, and native-core validation.
• Updates Conda recipes for glibc 2.34, Windows ARM64 CPython 3.12, and macOS wheel selection.
• Updates CI permissions, documentation, tests, and locked PyYAML dependencies.

File summaries

File	Reviewed changes
tests/test_035_conda_macho_assert.py	Mach-O architecture and payload validation tests.
tests/test_034_conda_verify_cwd.py	Core probing and Conda orchestration tests.
tests/test_030_pe_machine_assert.py	PE machine and Windows payload validation tests.
tests/test_029_bundled_binary_audit.py	ELF, glibc, ABI, and bundled-binary audit tests.
requirements.txt	Adds PyYAML.
README.md	Adds Conda installation and release guidance.
OneBranchPipelines/scripts/build_conda_packages.py	Builds packages and orchestrates audits and core probes.
eng/scripts/audit_bundled_binaries.py	Audits ELF metadata and glibc compatibility. Moderate (1 vote): enforce the fixed 2.34 floor.
eng/scripts/assert_pe_machine.py	Validates Windows native payload architecture.
eng/scripts/assert_macho_arch.py	Validates macOS native payload architecture.
eng/scripts/_conda_pkg.py	Shared native contract checks. Moderate (2 votes): align accepted stable-ABI names with packaging. Moderate (1 vote): require the core package initializer for cross-target validation.
eng/requirements-test-linux.txt	Adds locked PyYAML dependency.
eng/requirements-build-macos.txt	Adds locked PyYAML dependency.
eng/pipelines/pr-validation-pipeline.yml	Grants and verifies SQL Server DMV permissions.
conda/README.md	Documents Conda candidate guidance.
conda/mssql-python/meta.yaml	Adds glibc and Windows ARM64 ABI constraints.
conda/mssql-python/build.sh	Selects the appropriate macOS wheel.
conda/mssql-python/bld.bat	Enforces Windows bulk-copy core presence.

Review details

Suppressed comments (2)

eng/scripts/_conda_pkg.py:58

• For non-executable cross targets the orchestrator skips _core_probe and relies on this static contract. Because only the extension filename is required, a payload containing mssql_py_core/...so but no mssql_py_core/__init__.py passes the PE/Mach-O audits even though import mssql_py_core cannot provide the bulk-copy package; require the package initializer and cover it in the fixtures, or fail closed on the cross-target path.

    cores = [
        name
        for name in names
        if re.fullmatch(
            rf"{re.escape(root)}mssql_py_core/mssql_py_core(?:\.[^/]+)?\.{suffix}", name
        )

eng/scripts/audit_bundled_binaries.py:297

• The fixed contract is __glibc >=2.34, but this parser only checks that the declared floor is no lower than the symbols it happens to observe. A rebuilt payload whose core requires only GLIBC_2.28 and whose metadata says __glibc >=2.28 would therefore pass the release audit, contradicting the advertised Linux floor. Reject parsed floors below 2.34 here as well as checking symbol compatibility.

    match = re.fullmatch(r"__glibc\s+>=(\d+(?:\.\d+)+)", specs[0])
    if match is None:
        raise ValueError(f"unsupported __glibc dependency: {specs[0]!r}")
    return tuple(map(int, match[1].split(".")))

• Files reviewed: 17/18 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

The changes span native packaging, platform validation, and release workflows requiring final human review.

Review details

• Files reviewed: 18/19 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

The unresolved inherited Conda environment behavior and broad native-packaging changes require human review.

Review details

Suppressed comments (1)

OneBranchPipelines/scripts/build_conda_packages.py:307

• build_env starts from dict(os.environ), so removing an explicit assignment does not guarantee that Conda will stop receiving CONDA_PLUGINS_AUTO_ACCEPT_TOS: a runner or parent process that already exports it will be propagated to every invocation. Since this change is intended to remove automatic terms acceptance, explicitly remove the variable from the copied environment and cover the inherited-value case.

    if cross_target_subdir:
        # conda-build AND the verify `conda create` honor CONDA_SUBDIR -> the packages are
        # stamped for the target subdir and the import check runs the target Python where the
        # host can execute it (natively / Rosetta 2 / QEMU binfmt).
        env["CONDA_SUBDIR"] = cross_target_subdir

• Files reviewed: 17/17 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

The new .conda test needs the same zstandard-backend skip guard as the other archive-based tests.

Review details

Suppressed comments (1)

tests/test_035_conda_macho_assert.py:203

• This new .conda test calls _make_conda, which requires a zstandard backend, but it is not guarded like the other archive-based tests in this file (for example, lines 190 and 222). A direct/source-only run without that optional backend now fails with ModuleNotFoundError instead of skipping; add the same skipif marker.

@pytest.mark.parametrize("state", ["missing", "missing-init", "wrong-arch", "wrong-tag", "abi3"])
def test_required_core_contract(tmp_path, state):

• Files reviewed: 17/17 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

The shared validator must handle malformed non-list depends metadata without aborting the platform audits.

Review details

Suppressed comments (1)

eng/scripts/_conda_pkg.py:35

• validate_native_contract iterates index.get("depends", []) without validating its type. For a readable but malformed package with depends: null (or another non-list), this raises TypeError; all three platform audits call this outside a catch, so audit_package() aborts instead of returning the fail-closed violation used for malformed package metadata. Normalize or validate depends before scanning it.

    pins = [
        d
        for d in index.get("depends", [])
        if isinstance(d, str) and d.split()[:1] == ["python_abi"]

• Files reviewed: 17/17 changed files
• Comments generated: 0 new
• Review effort level: Lite

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 100%

🎯 Overall Coverage 83%

📈 Total Lines Covered: 8222 out of 9874 📁 Project: mssql-python

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

---

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.performance_counter.hpp: 0.7%
mssql_python.pybind.logger_bridge.cpp: 57.9%
mssql_python.pybind.ddbc_bindings.h: 61.5%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.row.py: 77.6%
mssql_python.pybind.ddbc_bindings.cpp: 77.7%
mssql_python.pybind.connection.connection_pool.cpp: 82.4%
mssql_python.logging.py: 85.5%
mssql_python.helpers.py: 89.3%
mssql_python.pooling.py: 90.1%

---

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report

[Copilot]

🔵 Needs a closer look

Cross-platform native packaging and CI changes require final human review.

Review details

• Files reviewed: 17/17 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

Four unresolved moderate findings affect fail-closed packaging and remove dependency or pipeline regression coverage.

Review details

Suppressed comments (4)

Previously missed (4) — in code that hasn't changed since the last review.

tests/test_release_build_dependencies.py:1

• Deleting this entire test module removes the only automated checks for the locked dependency files and refresh workflow (exact pins/hashes, pipeline --require-hashes installs, PR-safe triggers/action SHAs, and the validation matrix). Those files and the workflow remain in the repository, and this PR adds no replacement assertions, so dependency or pipeline regressions can now pass silently. Restore this module or move equivalent checks into a maintained test file.
  conda/mssql-python/bld.bat:46
• This guard is nested only in the non-executable/cross-build branch. Native Windows builds take the else pip-install path and skip the check, so a wheel without mssql_py_core can still produce a successful reduced package when bld.bat is invoked directly; the recipe import test only imports mssql_python. Move the guard after the branch (or apply it to both paths) so every build fails closed, retaining the PE audit for actual machine type.
  conda/mssql-python/build.sh:31
• This macOS cross-extraction path only selects the code wheel; it never verifies that the extracted wheel contains the required mssql_py_core extension. A malformed or legacy wheel can therefore produce a reduced package when the recipe is run without the orchestrator's later audit, unlike the new Windows cross guard. Add a required-core check before staging this package.
  tests/test_034_conda_verify_cwd.py:31
• This deletion removes the three regression tests covering best-effort Conda consolidation, official wheel provenance gating, and the Windows pool demand indentation. The corresponding pipeline/template blocks still exist, and no replacement assertions were added, so future YAML edits can break release safety without CI detecting it. Keep these tests or rehome equivalent assertions while adding the new cases.

• Files reviewed: 17/17 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

Address malformed dependency-field handling and retain or relocate the dependency workflow regression coverage.

Review details

Suppressed comments (2)

eng/scripts/_conda_pkg.py:37

• If a malformed info/index.json contains "depends": null (or another non-list value), this comprehension can raise TypeError before any audit error is returned. The callers promise malformed packages fail through their violation lists, so validate/normalize the dependency field before iterating it.

    pins = [
        d
        for d in index.get("depends", [])
        if isinstance(d, str) and d.split()[:1] == ["python_abi"]
    ]
    abi = re.fullmatch(r"python_abi (3\.\d+)\.\* \*_cp(3\d+)", pins[0]) if len(pins) == 1 else None

tests/test_release_build_dependencies.py:1

• Deleting this file removes the only repository tests that validate the still-active refresh-build-dependencies.yml contract: exact pinned/hash-locked requirements, pipeline --require-hashes installs, the supported matrix, and the workflow's PR-safety/update behavior. With that workflow still present, future lock or pipeline drift can now pass the Python test suite unnoticed; please retain or relocate this regression suite rather than removing it.

• Files reviewed: 17/17 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Unresolved native-core validation and audit issues remain, and important regression-test coverage was removed.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

conda/mssql-python/build.sh:31

• This cross-build path now filters the macOS code wheel, but it still verifies only ddbc_bindings; neither this path nor the native pip install path checks that mssql_py_core was extracted. Because meta.yaml tests only import mssql_python and bulk copy loads the core lazily, a direct conda build can still produce a package with bulk copy silently removed. Add a target-aware core presence/compatibility check to both paths instead of relying only on the later orchestrator audit.

  for w in "$WHEELS_DIR/${pkg_underscore}-${PKG_VERSION}-cp${CONDA_PY}-"*macosx*.whl; do

eng/scripts/_conda_pkg.py:36

• index.get("depends", []) can return None (or another non-iterable) for malformed info/index.json; the list comprehension then raises TypeError before any audit can return its normal violation list. All three callers invoke this function without catching that exception, so malformed packages crash the audit instead of failing closed with a diagnostic. Validate/normalize depends before iterating it.

    pins = [
        d
        for d in index.get("depends", [])
        if isinstance(d, str) and d.split()[:1] == ["python_abi"]
    ]

tests/test_release_build_dependencies.py:1

• This entire regression-test module is removed, but its contracts still cover active files such as .github/workflows/refresh-build-dependencies.yml, the hashed eng/requirements-*.txt locks, and the OneBranch build stages. Deleting it removes coverage for exact pin/hash validation, --require-hashes installation, the supported Python/platform matrix, and the immutable refresh-PR workflow, so later dependency or pipeline drift can pass CI unnoticed. Please retain this module or replace it with equivalent tests.

• Files reviewed: 17/17 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Unresolved findings concern ToS handling and removed dependency and pipeline regression coverage.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

tests/test_release_build_dependencies.py:1

• This removes the only automated suite that checks the release build requirement locks for exact pins, SHA-256 hashes, direct requirements, and their pipeline consumption. The refresh workflow and release-stage installs are still present, so future dependency or pipeline edits can now silently break the locked build path; restore this suite or replace it with equivalent assertions.

tests/test_034_conda_verify_cwd.py:31

• These deletions remove coverage for the still-present Conda pipeline contracts: platform stages depending on ValidateWheelProvenance, best-effort succeededOrFailed() consolidation, and the Official main-wheel provenance gate. Those behaviors remain in OneBranchPipelines/conda-build-pipeline.yml and jobs/consolidate-conda-artifacts-job.yml, so retain or update these regression tests instead of dropping them.

)

• Files reviewed: 17/17 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

Dependency-lock contract tests were removed without replacement coverage.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

tests/test_release_build_dependencies.py:1

• Deleting this entire test module removes the only automated coverage for the committed dependency-lock contracts, including exact pins/hashes, pipeline --require-hashes usage, and the immutable refresh workflow. Those files and workflows still exist and are used by CI, and this PR does not add replacement coverage; please retain or relocate these tests rather than deleting them.

• Files reviewed: 17/17 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

Add the missing Windows ARM64 and macOS wheel-selection coverage, and retain equivalent release-dependency contract tests.

Review details

Suppressed comments (3)

tests/test_030_pe_machine_assert.py:178

• cross_build=True only swaps PYTHON for a non-executable path; on the x64 Windows CI host, sysconfig.get_platform() still selects win_64 and target_platform remains win-64. Consequently this regression never exercises the new bld.bat path with ODBC_ARCH=win_arm64, so the target-dependent core filename check can regress unnoticed. Parameterize the fixture's target subdir/architecture (or explicitly force a win-arm64 target) so both Windows targets are covered.

@pytest.mark.parametrize("cross_build", [False, True])
@pytest.mark.parametrize("state", ["valid", "missing", "missing-init", "wrong-tag", "abi3"])
def test_windows_recipe_requires_core_on_both_install_paths(tmp_path, cross_build, state):

tests/test_035_conda_macho_assert.py:229

• This cross-build fixture creates only the macOS code wheel, so the previous broad cp${CONDA_PY}-* glob would select the same file and the test would pass before the *macosx* restriction. Add a Linux cp312 wheel in the same directory (with a distinguishable or invalid payload) and assert the macOS wheel is the one extracted, otherwise the new wheel-selection guard is untested.

    with zipfile.ZipFile(wheels / f"mssql_python-1.13.0-{code_tag}.whl", "w") as wheel:
        for name, data in payload.items():
            if "/mssql_python_odbc/" not in name:
                wheel.writestr(name.removeprefix("lib/python3.12/site-packages/"), data)

tests/test_release_build_dependencies.py:1

• Deleting this entire test module removes the repository's only contract coverage for the locked release dependencies and refresh workflow (hashes, direct requirements, pipeline --require-hashes usage, target matrix, pinned actions, and immutable update-PR behavior). None of those invariants are replaced by the Conda tests in this PR, so a future lock or workflow change can silently break release builds; please retain or port equivalent coverage rather than deleting the module.

• Files reviewed: 17/17 changed files
• Comments generated: 0 new
• Review effort level: Lite