CrossFittedTransformer(Preprocessor(add_missing_indicator=True)) crashes when a fold's training rows contain no missing value #89

Description

Cross-fitting a Preprocessor that adds missing indicators (add_missing_indicator=True or missing_policy="impute_with_indicator") fails whenever a column with missing values has none in some fold's training rows.

• A column with a single missing value always triggers it.
• With two missing values and 5 folds it happens about 20% of the time.

This affects every numerical method, target-aware or not (PLE, the splines, the feature maps, the scalers), and categorical columns as well. The error message blames adaptive sizing, which is not involved. missing_policy="separate_state" works.

Affected code

• pretab/core/supervised.py:181-190: CrossFittedTransformer.fit_transform refits a clone of the whole wrapped Preprocessor on each fold's training rows and requires the all-data output width.
• pretab/compose/factory.py:121 (numerical) and :198 (categorical): the indicator comes from SimpleImputer(add_indicator=True). Its missing indicator emits a column only for features that have missing values in the rows it is fitted on.

Reproduction

import warnings
import numpy as np, pandas as pd
from pretab import CrossFittedTransformer, Preprocessor

warnings.simplefilter("ignore")
rng = np.random.default_rng(0)
n = 200
a = rng.normal(size=n)
a[7] = np.nan  # a single missing value: one fold's training rows have none
X = pd.DataFrame({"a": a, "b": rng.normal(size=n)})
y = np.nan_to_num(a) + 0.1 * rng.normal(size=n)
for cfg in [dict(add_missing_indicator=True), dict(missing_policy="impute_with_indicator"), dict(missing_policy="separate_state")]:
    pre = Preprocessor(numerical_method="ple", output_dim=4, **cfg)
    print(cfg, "| Preprocessor alone:", pre.fit_transform(X, y).shape)
    try:
        print("   CrossFittedTransformer:", CrossFittedTransformer(pre, n_folds=5, random_state=0).fit_transform(X, y).shape)
    except Exception as e:
        print("   CrossFittedTransformer ->", type(e).__name__ + ":", e)

Output (PreTab 1.0.0 (6bacaba), Python 3.12, numpy 2.5.3, pandas 2.3.3, scikit-learn 1.9.1, scipy 1.18.1):

{'add_missing_indicator': True} | Preprocessor alone: (200, 12)
   CrossFittedTransformer -> IncompatibleParamsError: Cross-fitting requires a fixed output width across folds; expected 12, got 8. Disable adaptive sizing on the wrapped transformer.
{'missing_policy': 'impute_with_indicator'} | Preprocessor alone: (200, 12)
   CrossFittedTransformer -> IncompatibleParamsError: Cross-fitting requires a fixed output width across folds; expected 12, got 8. Disable adaptive sizing on the wrapped transformer.
{'missing_policy': 'separate_state'} | Preprocessor alone: (200, 10)
   CrossFittedTransformer: (200, 10)

Expected behavior

The out-of-fold features have the same columns as transform. The missing indicator does not use the target, so which indicator columns exist can be fixed by the all-data fit.

Actual behavior

IncompatibleParamsError: Cross-fitting requires a fixed output width across folds; expected 12, got 8. Disable adaptive sizing on the wrapped transformer.

Impact

A crash on a combination of two documented features whenever a column has very few missing values: always with one, about 20% of the time with two, and a few percent with three (5 folds). The error message points to the wrong setting.

Root cause

The missing indicator is refitted on each fold's training rows together with the rest of the Preprocessor, so its width depends on which rows the fold contains.

Suggested fix direction

• Take the indicator columns from the all-data fit and refit only the representation per fold, so every fold produces the same columns.
• Make the width-mismatch message describe the general cause (a width that depends on the training rows) rather than adaptive sizing only.

Related issues

• CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79: the same per-fold refit of the whole Preprocessor also relearns column types and category codes.
• add_missing_indicator / impute_with_indicator feeds the indicator through the scaler and basis; lineage crashes #62: the indicator from SimpleImputer(add_indicator=True) is also fed through the scaler and the basis.
• CrossFittedTransformer converts DataFrames to ndarray; a wrapped Preprocessor treats every column as categorical #58: CrossFittedTransformer passes the wrapped Preprocessor an array instead of the DataFrame.