One Model-level switch for the sparse path instead of three independent opt-ins

[FabianHofmann]

Note

The following content was generated by AI.

Describe the feature you'd like to see

Discussion issue deferred from #972 (part G) and #969 (part 3). Before PR 4 adds more switches, decide what the configuration surface of the sparse/CSR path should be.

Problem

Important

This issue describes master at b9a698b (after #973 and #974); all code links point there. Since then #979 (closes #975) added native soften for frozen constraints; the softening parts below are updated for it.

A model only stays sparse end to end with three independent opt-ins, and any subset of them silently gives dense results:

1. linopy.options["semantics"] = "v1": the sparse path is v1-only (is_v1() gate at linopy/expressions.py:613-618 for groupby, :2836 for @).
2. linopy.options["sparse_groupby"] = True, or per call groupby(...).sum(sparse=True): turns on sparse groupby (expressions.py:613) and, as a side effect, decides whether @ returns a CSR-backed result (expressions.py:2893).
3. Model(freeze_constraints=True), or per call add_constraints(..., freeze=True): makes the constraint a CSRConstraint (model.py:1310-1311, :1335).

Under legacy semantics sparse_groupby=True is ignored without a message (only an explicit sparse=True raises, expressions.py:614). Without sparse_groupby, groupby().sum() and @ on a dense input return dense expressions. Without freezing, a sparse lhs becomes a dense Constraint. warn_on_densify would report the last case, but it is off by default (config.py:88).

import warnings

import numpy as np
import pandas as pd

import linopy

warnings.simplefilter("ignore", FutureWarning)

groups = pd.Series(np.arange(6) % 2, index=pd.RangeIndex(6, name="i"), name="g")
weights = pd.DataFrame(
    np.eye(6)[:, :2], index=groups.index, columns=pd.Index(["a", "b"], name="k")
)


def build(semantics: str, sparse_groupby: bool, freeze: bool) -> None:
    linopy.options.reset()
    linopy.options(semantics=semantics, sparse_groupby=sparse_groupby)
    m = linopy.Model(freeze_constraints=freeze)
    x = m.add_variables(lower=0, coords=[groups.index], name="x")
    grouped = x.to_linexpr().groupby(groups).sum()
    matmul = x.to_linexpr() @ weights
    con = m.add_constraints(grouped >= 1, name="c")
    print(
        f"{semantics=!s:6} {sparse_groupby=!s:5} {freeze=!s:5} -> "
        f"groupby sparse: {grouped.is_sparse!s:5}  @ sparse: {matmul.is_sparse!s:5}  "
        f"constraint: {type(con).__name__}"
    )


for args in [("legacy", True, True), ("v1", False, True), ("v1", True, False), ("v1", True, True)]:
    build(*args)

Output on master (b9a698b):

semantics=legacy sparse_groupby=True  freeze=True  -> groupby sparse: False  @ sparse: False  constraint: CSRConstraint
semantics=v1     sparse_groupby=False freeze=True  -> groupby sparse: False  @ sparse: False  constraint: CSRConstraint
semantics=v1     sparse_groupby=True  freeze=False -> groupby sparse: True   @ sparse: True   constraint: Constraint
semantics=v1     sparse_groupby=True  freeze=True  -> groupby sparse: True   @ sparse: True   constraint: CSRConstraint

Only the last row is sparse end to end. In the first two rows the CSRConstraint is built from a dense lhs (Constraint.freeze() → CSRConstraint.from_dense), so the build itself was dense and nothing says so.

Current configuration surface (file:line)

Global options (linopy/config.py:84-89):

• semantics (default "legacy", validated at config.py:48), read through is_v1() (linopy/semantics.py:350-352). Sparse gates: expressions.py:613-618 (groupby), expressions.py:2836 (@, dense path otherwise).
• sparse_groupby (default False), read at expressions.py:613 (default of groupby sparse=) and expressions.py:2893 (whether @ on a dense input returns CSR). Also mentioned in the linopy/csr.py:4-5 module docstring. merge has no switch: _try_csr_merge (expressions.py:3668, gate at :3686) stays sparse only if an operand is already CSR-backed.
• warn_on_densify (default False), read only in _densify_notice (csr.py:741-750), called from expressions.py:2419 / :3626-3634, constraints.py:1325 (CSRConstraint.to_dense/mutable) and model.py:1350-1357 (sparse lhs added unfrozen).

Per-call arguments:

• LinearExpressionGroupby.sum(sparse: bool | None = None) (expressions.py:559): None → is_v1() and (sparse_groupby or source is CSR-backed); True under legacy raises; True with an unsupported grouper raises (expressions.py:650-655). This is the only per-call sparse= today; dot says so (expressions.py:1589).
• Model.add_constraints(freeze: bool | None = None) (model.py:1247, default from the model at :1310-1311). freeze is dropped when Model.chunk is set (model.py:1312-1313) (at b9a698b it also raised with penalty=; feat(csr): native soften for frozen constraints #979 removed that).
• Constraints.add(constraint, freeze=False) (constraints.py:2462).

Model-level:

• Model(freeze_constraints: bool = False) (model.py:236, stored at :302), public property with setter (model.py:531-537), round-tripped through netcdf (io.py:1349, model.py:612), also used by add_indicator_constraints (model.py:1507-1508).

Object-level API:

• ConstraintBase.freeze() / .mutable() (constraints.py:279-284), Constraint.freeze() → CSRConstraint.from_dense (constraints.py:2073-2075), CSRConstraint.freeze() returns self, .mutable() / .to_dense() gives a detached dense copy (constraints.py:1319-1330).
• Frozen-constraint errors point to these switches: from_rule → "call .freeze()" (constraints.py:951-956); Constraint.soften docstring (constraints.py:2175-2178).
• LinearExpression.is_sparse (expressions.py:989, :2424) is read-only diagnostics, not configuration.

Proposal

As little exposed configuration as possible. The rule: the user picks the representation of expressions once per model, and never per operation. Mutability of a single constraint stays a per-constraint choice.

• One read-only model key, Model(sparse: bool = False). For that model it turns on sparse groupby().sum(), @/dot and merge results, and frozen constraints by default. No setter: freeze_constraints has one today (model.py:535-537), and flipping it mid-build makes earlier and later objects differ. Model(dtypes=...) is the precedent for a constructor-only setting that expression operations read. A bool, not a mode (backend=...): there is no third state.
• v1 only, checked where it matters. Model(sparse=True) raises under legacy semantics. Because semantics is a global option that can change after creation, the sparse gates (expressions.py:613, :2836, :3686) also raise when the model is sparse and semantics is legacy, instead of falling back to dense without a message. The model does not pin its semantics.
• No per-operation arguments. groupby(...).sum(sparse=...) and options["sparse_groupby"] go. @ no longer reads sparse_groupby (expressions.py:2893).
• Per-constraint override stays. add_constraints(freeze=...) overrides the model default, like mask= overrides auto_mask. Constraint.freeze() / CSRConstraint.mutable() stay. Immutability is a property of one registered object (loc, update, from_rule need it), not of how an intermediate expression is stored.
• Softening. Frozen constraints soften natively since feat(csr): native soften for frozen constraints #979, so penalty= needs no special case in a sparse model. Do not add a softened constraint unfrozen automatically: a dense constraint inside a sparse model is the divergence this issue removes.
• Model.chunk. Model(sparse=True, chunk=...) and setting chunk on a sparse model raise, instead of dropping freeze (model.py:1312-1313).
• warn_on_densify stays global. It configures reporting, not results, and applies across models.
• Not the v1 default. CSRConstraint has fewer methods than Constraint (constraints.py:937-956); a default that removes methods is more than a storage change. Revisit after the update/from_rule gaps close (soften is done in feat(csr): native soften for frozen constraints #979), independently of the v1 switch.
• Persistence. The key is written to netcdf and read back, and Model.copy keeps it. Today Model.copy keeps freeze_constraints but rebuilds a CSRConstraint as a dense Constraint; a copy of a sparse model stays sparse. Old files that store freeze_constraints map onto the key (see Deprecation).
• Deprecation. sparse_groupby, sum(sparse=...) and Model(freeze_constraints=...) map onto the new key with a FutureWarning and are removed with legacy semantics; all are released API since 0.8.0. Netcdf files written by 0.8.x store freeze_constraints (io.py:1349); reading that attribute stays supported indefinitely.
• Definition of done. For the reproduction above, the four configurations collapse to Model(sparse=True), which is sparse end to end (groupby, @, constraint). Each partial setup either raises (legacy + sparse) or is dense by the user's explicit choice (freeze=False), never silently.

Long-term target: a Constraint with an optional CSR backing, like LinearExpression has today. Then "frozen" disappears as a concept and sparse only describes storage. Until then the key bundles storage and immutability, because a plain Constraint cannot hold a CSR backing.

Combining expressions of two models is out of scope: it is accepted today and silently wrong (the second model's labels alias the first's), which is a separate bug.

Open questions

• warn_on_densify default. With the key, a CSR backing can only come from a model that opted in, so warning by default no longer affects existing users. If the default flips to True, a deliberate freeze=False must not warn (model.py:1350-1357 does today); only unrequested densifying should.
• Scope of the key. add_indicator_constraints already follows freeze_constraints (model.py:1507-1508). Do the objective (sparse since Sparse path for the objective #967) and add_expressions follow the key too?
• PyPSA migration. Under the key every constraint in n.model.constraints is a CSRConstraint. Which PyPSA code paths use loc, update or soften on constraints and need freeze=False? (soften works on frozen constraints since feat(csr): native soften for frozen constraints #979.)
• Removal date. The deprecated switches are "removed with legacy semantics", which Remove legacy semantics at linopy 1.0 (Stage 3) #846 tracks for 1.0, but 1.0 has no target release. Add them to the Remove legacy semantics at linopy 1.0 (Stage 3) #846 checklist, or give them their own removal release.
• Constraints.add(freeze=...) (constraints.py:2462). Keep it as the low-level twin of add_constraints(freeze=...), or drop it if nothing outside linopy uses it.

Consequence for the roadmap

• PR 4 part G (Tracking: sparse/CSR path follow-ups after #961 #972, Make the sparse path observable and controllable (.is_sparse, densify warning, per-call sparse=) #969 part 3) changes from adding per-call sparse= on @/dot/merge to wiring the single read-only model key into groupby, @ and merge, turning the silent v1 gates into errors, removing the @ coupling to options["sparse_groupby"] and deprecating the per-call arguments.
• feat(csr): native soften for frozen constraints #979 (PR 3, closes Native sparse soften for frozen constraints (penalty= with freeze=True) #975) is merged, so softening needs no special case in the key.
• The docs (Document the sparse expression path (CSRLinearExpression, sparse=True, sparse_groupby) #971) then describe one switch, the freeze= override, and is_sparse and warn_on_densify as diagnostics.