feat(csr): constraint and export boundaries of the sparse path (PR 2 of #972) by FabianHofmann · Pull Request #974 · PyPSA/linopy

FabianHofmann · 2026-09-23T15:59:58Z

Note

The following content was generated by AI.

PR 2 ("boundaries") of the sparse/CSR plan in #972, stacked on #973 (targets feat/csr-expression-kernel; retarget to master once #973 merges).

Closes #941, closes #963, closes #967, closes #968, closes #970.

What changes

Emit flat/to_polars directly from the CSR store #968 LinearExpression.flat/to_polars on a CSR-backed expression are built from the CSR store (numpy arrays, no dense rectangle, no pyarrow dependency). Rows equal the dense path: absent cells and zero coefficients dropped, same columns and dtypes.
Sparse path for the objective #967 A CSR-backed linear objective stays sparse through add_objective, matrices.c, LP/MPS export, netcdf write, Model.copy, persistent snapshots and direct solver APIs. matrices.c and the persistent snapshot share one reader, Objective.linear_terms(). Quadratic objectives stay dense.
CSRConstraint drops auxiliary coordinates when a sparse expression becomes a constraint #941 CSRConstraint keeps the full Grid incl. aux coords (scalar too) through from_csr, Constraint.freeze(), to_dense()/mutable() and netcdf IO.
CSRConstraint lacks loc/update/from_rule and raises bare AttributeError #963 loc, update, soften, from_rule and the coeffs/vars/sign setters on CSRConstraint raise an explanatory AttributeError naming .mutable() (or Constraint.from_rule + .freeze()), consistent with the existing rhs/lhs/scaling setters, via one helper.
Widen the CSRConstraint fast path in Model.add_constraints #970 An expression rhs is moved to the lhs (lhs - rhs <= 0), so a CSR-backed lhs stays a CSRConstraint; a constant rhs with an explicit join= takes the same route, so sparse matches dense for every join mode. mask= also applies sparsely to a CSRConstraint built beforehand; the mask logic now lives once in CSRConstraint.masked. The dense rebuild in add_constraints (freeze=False, chunked model, penalty=) now emits one densify notice naming its cause.

Decisions

Objective name lives on Objective. CSRLinearExpression stays attrs-free. Objective.name is the fixed "objective", Objective.attrs returns {"name": ...}, the setter writes attrs["name"] only on a dense expression, and the netcdf writer adds it.
penalty/soften + freeze stays unsupported for now. It raises explicitly; soften rewrites the constraint in place, which a frozen constraint does not allow. This is a missing feature, not a technical limit: a native sparse soften (one slack term per active row) is filed as Native sparse soften for frozen constraints (penalty= with freeze=True) #975. The soften error on a frozen constraint now points to freeze=False instead of .mutable(), since soften refuses the detached copy that .mutable() returns.
fix: support adding/removing variables after adding frozen constraint #806 not awaited. It has been stalled since July and conflicts with master. Edits in the CSRConstraint methods it touches are kept small and local.

Also fixed along the way

read_netcdf failed for a whole model when a frozen constraint had a - in a dim, MultiIndex level or aux coord name (the reader splits variable names on -). CSRConstraint coordinate metadata is now stored under positional names with the real names as attributes. Files written by earlier versions still load; files written now need this version to read the frozen constraints.
CSRLinearExpression.from_dense crashed on a 0-d expression, e.g. adding a scalar term to a fully summed sparse expression.

Known gaps (not in scope)

A dense lhs with a sparse rhs (x <= sparse_expr) still densifies.
Some objective edit/diagnostic paths densify with a notice: Objective.data/vars/coeffs, Model.remove_variables, calculate_block_maps, objectiverange.
Per-call sparse= and docs (Make the sparse path observable and controllable (.is_sparse, densify warning, per-call sparse=) #969 part 3, Document the sparse expression path (CSRLinearExpression, sparse=True, sparse_groupby) #971) are PR 3.

Verification

uv run pytest -q --ignore=test/remote -p no:cacheprovider -k "not xpress": 7061 passed, 1073 skipped
uv run --isolated --python 3.11 --extra dev pytest test/test_csr.py -q -p no:cacheprovider: 551 passed, 547 skipped (legacy variants of v1-only tests)
ruff check, ruff format, mypy linopy: clean

…e; Objective owns its name (#967)

…eeded

…ssion rhs and mask on frozen constraints (#941, #963, #970)

…build, join for constant rhs, 0-d merge

…he detached mutable() copy

…ression, named setter in FROZEN_MUTATIONS

codspeed · 2026-09-24T11:20:42Z

Merging this PR will regress 1 benchmark

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
❌ 1 regressed benchmark
✅ 176 untouched benchmarks
⏩ 181 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
❌	`test_to_lp[nodal_balance-severity=0]`	3.3 MB	3.7 MB	-11.58%
⚡	`test_to_lp[sparse_network-n=10]`	1,337.8 KB	718.2 KB	+86.27%
⚡	`test_build[nodal_balance_sparse-severity=0]`	1.5 MB	1.3 MB	+13.94%
⚡	`test_build[nodal_balance_sparse-severity=100]`	1.5 MB	1.3 MB	+13.94%
⚡	`test_build[nodal_balance_sparse-severity=50]`	1.5 MB	1.3 MB	+13.94%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing feat/csr-boundaries (6785da3) with master (38fbd4a)}

181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

github-actions · 2026-09-24T11:21:00Z

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06003 (1.09)   │   15.03 (1.00) 
 (v1)        0.05483 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06522 (1.10)   │   12.04 (1.00) 
 (v1)        0.05932 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02595 (1.10)   │   15.20 (1.0) 
 (v1)        0.02369 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.0399 (1.10)   │   44.93 (1.0) 
 (v1)       0.03626 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02855 (1.08)   │   11.51 (1.0) 
 (v1)        0.02646 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06707 (1.07)   │   24.34 (1.06) 
 (v1)        0.06289 (1.0)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.07519 (1.06)   │   16.12 (1.00) 
 (v1)         0.0707 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.01615 (1.09)   │   752.18 (1.10) 
 (v1)         0.0148 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.01574 (1.08)   │   3.12 (1.33) 
 (v1)        0.01454 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04735 (1.24)   │   126.16 (1.44) 
 (v1)        0.03807 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04657 (1.26)   │   126.16 (1.44) 
 (v1)        0.03706 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04673 (1.25)   │   126.16 (1.44) 
 (v1)        0.03741 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03685 (1.02)   │    715.12 (1.0) 
 (v1)        0.03612 (1.0)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03576 (1.10)   │   4.54 (1.27) 
 (v1)        0.03238 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                time (s)          peak (KiB) 
 name               mean   │            mean 
─────────────────────────────────────────────
 (legacy)   0.252 (1.04)   │   704.12 (1.09) 
 (v1)       0.2422 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.2661 (1.06)   │   18.34 (1.0) 
 (v1)        0.2515 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.2647 (1.06)   │   9.54 (1.0) 
 (v1)        0.2489 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                 time (s)        peak (KiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.0508 (1.10)   │   3.77 (1.12) 
 (v1)       0.04626 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05137 (1.11)   │   216.59 (1.10) 
 (v1)        0.04648 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.02581 (1.10)   │   938.49 (1.0) 
 (v1)        0.02344 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02698 (1.10)   │   9.66 (1.0) 
 (v1)        0.02447 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.0264 (1.10)   │   5.32 (1.0) 
 (v1)       0.02399 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01468 (1.03)   │   1.28 (1.0) 
 (v1)        0.01427 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01464 (1.02)   │   1.28 (1.0) 
 (v1)        0.01431 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01477 (1.03)   │   1.28 (1.0) 
 (v1)        0.01431 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.1295 (1.08)   │   946.85 (1.06) 
 (v1)        0.1202 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1281 (1.08)   │   12.01 (1.00) 
 (v1)        0.1183 (1.0)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0322 (1.09)   │   147.70 (1.06) 
 (v1)       0.02951 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03204 (1.08)   │   2.60 (1.09) 
 (v1)        0.02978 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02645 (1.09)   │   696.75 (1.03) 
 (v1)        0.02436 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)    0.06104 (1.0)   │   137.97 (1.0) 
 (v1)       0.06116 (1.00)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04349 (1.04)   │   69.22 (1.0) 
 (v1)        0.04179 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03073 (1.11)   │   402.33 (1.00) 
 (v1)        0.02772 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03029 (1.11)   │   3.19 (1.19) 
 (v1)        0.02718 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03339 (1.08)   │   29.00 (1.54) 
 (v1)        0.03082 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.04176 (1.12)   │   37.95 (1.43) 
 (v1)        0.03717 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0643 (1.00)   │    410.93 (1.0) 
 (v1)       0.06414 (1.0)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)    0.06702 (1.0)   │     9.94 (1.0) 
 (v1)       0.06926 (1.03)   │   10.22 (1.03)

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}

… polars round trip

…ys, skip unit scaling

CSRLinearExpression.live_terms is the single sparse term filter, shared by _csr_terms and LinearExpression.linear_terms. QuadraticExpression aligns factors with coeffs by dim name. Objective.linear_terms delegates.

This was referenced Sep 24, 2026

Native sparse soften for frozen constraints (penalty= with freeze=True) #975

Closed

One Model-level switch for the sparse path instead of three independent opt-ins #976

Closed

Tracking: sparse/CSR path follow-ups after #961 #972

Open

FabianHofmann force-pushed the feat/csr-boundaries branch from 2aa83b1 to 4e7eb3a Compare September 24, 2026 10:57

Base automatically changed from feat/csr-expression-kernel to master September 24, 2026 11:12

FabianHofmann added 7 commits September 24, 2026 13:16

feat(csr): emit flat/to_polars directly from the CSR store (#968)

416d9c6

feat(csr): keep a CSR-backed objective sparse through export and solv…

080fb8f

…e; Objective owns its name (#967)

fix(csr): build sparse flat/to_polars from numpy arrays, no pyarrow n…

5faae66

…eeded

feat(csr): aux coords on CSRConstraint, explicit frozen errors, expre…

8ffdcf1

…ssion rhs and mask on frozen constraints (#941, #963, #970)

fix(csr): dash-safe netcdf coord names, densify notice on unfrozen re…

037f91c

…build, join for constant rhs, 0-d merge

fix(csr): soften on a frozen constraint points to freeze=False, not t…

aa441bc

…he detached mutable() copy

fix(types): widen frozen-constraint test helper, narrow objective exp…

8f27931

…ression, named setter in FROZEN_MUTATIONS

FabianHofmann force-pushed the feat/csr-boundaries branch from 4e7eb3a to 8f27931 Compare September 24, 2026 11:17

perf(objective): read linear objective terms from numpy arrays, not a…

0cf4a20

… polars round trip

FabianHofmann added 6 commits September 24, 2026 14:31

perf(csr): read objective terms and scale matrix blocks from raw arra…

59146b7

…ys, skip unit scaling

perf(csr): CSR store indices follow the model label dtype (#828)

1d5b580

test(csr): narrow frozen store in label dtype test

3db05c2

test(csr): narrow optional matrix A in label dtype test

24283a0

refactor(expressions): linear_terms lives on the expression classes

cc73834

CSRLinearExpression.live_terms is the single sparse term filter, shared by _csr_terms and LinearExpression.linear_terms. QuadraticExpression aligns factors with coeffs by dim name. Objective.linear_terms delegates.

docs(csr): docstrings state current behavior, not development history

a817a5e

FabianHofmann mentioned this pull request Sep 24, 2026

ci(codspeed): pin polars to one thread in the memory job #980

Merged

test(quadratic): narrow expression type in linear_terms test

6785da3

FabianHofmann merged commit b9a698b into master Sep 24, 2026
21 of 23 checks passed

FabianHofmann deleted the feat/csr-boundaries branch September 24, 2026 14:58

Navigation Menu

feat(csr): constraint and export boundaries of the sparse path (PR 2 of #972) - #974

feat(csr): constraint and export boundaries of the sparse path (PR 2 of #972)#974
FabianHofmann merged 15 commits into
masterfrom
feat/csr-boundaries

FabianHofmann commented Sep 23, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Sep 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

FabianHofmann commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

Decisions

Also fixed along the way

Known gaps (not in scope)

Uh oh!

codspeed Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will regress 1 benchmark

Performance Changes

Footnotes

Uh oh!

github-actions Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build cost — v1 vs legacy

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

FabianHofmann commented Sep 23, 2026 •

edited

Loading

codspeed Bot commented Sep 24, 2026 •

edited

Loading

github-actions Bot commented Sep 24, 2026 •

edited

Loading