diff --git a/docs/whats_new/index.rst b/docs/whats_new/index.rst index e256a1280..de5de43ae 100644 --- a/docs/whats_new/index.rst +++ b/docs/whats_new/index.rst @@ -8,6 +8,7 @@ Find out what's new in each new version release. .. toctree:: :maxdepth: 2 + v_200 v_190 v_180 v_170 diff --git a/docs/whats_new/v_200.rst b/docs/whats_new/v_200.rst new file mode 100644 index 000000000..14e94ffe2 --- /dev/null +++ b/docs/whats_new/v_200.rst @@ -0,0 +1,264 @@ +Version 2.0.X +============= + +Version 2.0.0 +------------- + +Deployed: TBD + +Contributors +~~~~~~~~~~~~ + +- `Aditya Nikam `_ +- `Ankit Hemant Lade `_ +- `Coro `_ +- `Direk Kakkar `_ +- `favour `_ +- `Jordi Pastor `_ +- `Matias Loiseau `_ +- `Maulik Patel `_ +- `Míriam Méndez `_ +- `Muhammad Aniq Ramzan `_ +- `Ojas Sharma `_ +- `Soledad Galli `_ +- `Sudais Khan `_ +- `Temidayo Akindahunsi `_ +- `Venish Paneliya `_ + +Feature-engine 2.0.0 is our biggest release so far. + +**The big news**: Feature-engine is no longer tied to pandas. Most transformers now work +with pandas and polars dataframes, and return the same type of dataframe they receive. +Under the hood, Feature-engine uses `narwhals `_, +a lightweight library that lets the same code run on different dataframe libraries. +pandas is no longer a required dependency. + +**Fewer errors**: when a transformer can't find variables of the type it needs, you can +now ask it to carry on with an empty list, with the new parameter `return_empty`. Several +transformers that used to stop with an error now handle the situation and let you +proceed. See the section "Transformers that no longer raise errors" below. + +**Better names**: we renamed several transformers to shorter or more consistent names. +The old names still work, but they raise a `FutureWarning` and will be removed in +version 2.1.0. + +**Faster**: many transformers are faster, some of them by orders of magnitude, like +`MathFeatures()` and `DatetimeOrdinal()`. + +This release also brings breaking changes: Feature-engine no longer accepts numpy +arrays, requires Python 3.11 or newer and scikit-learn 1.7 or newer, and two parameters +were removed. Please read the section "Breaking changes" before you upgrade. + +Special thanks to `Ojas Sharma `_ for migrating the +scaling module, and to `Jordi Pastor `_ and +`Míriam Méndez `_ for adding `return_empty` to the +transformers. Thanks also to everyone who renamed transformers, fixed bugs and improved +our tests, and to all who raised issues. + +Check out the full release changelog below. + +Support for polars +~~~~~~~~~~~~~~~~~~ + +The following transformers now work with pandas and polars dataframes: they learn the +same parameters and return the same values with both. (`Soledad Galli `_) + +- **Creation**: `CyclicalFeatures()`, `DecisionTreeFeatures()`, `GeoDistanceFeatures()`, + `MathFeatures()` and `RelativeFeatures()`. +- **Datetime**: `DatetimeFeatures()`, `DatetimeOrdinal()` and `DatetimeSubtraction()`. +- **Discretisation**: `ArbitraryDiscretiser()`, `DecisionTreeDiscretiser()`, + `EqualFrequencyDiscretiser()`, `EqualWidthDiscretiser()` and `GeometricWidthDiscretiser()`. +- **Encoding**: `CountEncoder()`, `DecisionTreeEncoder()`, `MeanEncoder()`, `OneHotEncoder()`, + `OrdinalEncoder()`, `RareLabelEncoder()`, `StringSimilarityEncoder()` and `WoEEncoder()`. +- **Imputation**: `MeanImputer()`, `ArbitraryImputer()`, `CategoricalImputer()`, + `EndTailImputer()`, `RandomSampleImputer()`, `MissingIndicator()` and `DropMissingData()`. +- **Outliers**: `Winsoriser()`, `ArbitraryOutlierCapper()` and `OutlierTrimmer()`. +- **Scaling**: `MeanNormalisationScaler()` (`Ojas Sharma `_). +- **Transformation**: `ArcsinTransformer()`, `ArcSinhTransformer()`, `BoxCoxTransformer()`, + `LogTransformer()`, `PowerTransformer()`, `ReciprocalTransformer()` and `YeoJohnsonTransformer()`. +- **Variable handling**: all the functions of the `variable_handling` module. + +The deprecated names of these transformers work with polars too. + +The feature selection, time series forecasting, preprocessing, text and wrappers modules +still work with pandas only, and need pandas installed. + +With polars, a few features depend on pandas-only concepts and behave differently: + +- `DatetimeFeatures(variables="index")` works with pandas only, because polars dataframes + have no index. With polars, it raises an error. +- In `MathFeatures()`, custom functions receive each row as a tuple with polars, and as a + pandas Series with pandas. Use functions that work with both, like `max(row)` instead of + `row.max()`. Aggregation names that only pandas understands, like `"sem"`, raise an + error with polars. + +Variable handling +~~~~~~~~~~~~~~~~~ + +- All functions of the `variable_handling` module, like `find_numerical_variables()` or + `check_categorical_variables()`, now accept pandas and polars dataframes. (`Soledad Galli `_) +- polars `Categorical` and `Enum` variables are recognised as categorical, and `Date` and + `Datetime` variables as datetime. (`Soledad Galli `_) +- `find_categorical_variables()` and `find_categorical_and_numerical_variables()` have a + new parameter, `exclude_datetime`. It defaults to `True`, which keeps the previous + behaviour: variables whose values can be parsed as dates are not returned as + categorical. Set it to `False` to return them. (`Soledad Galli `_) + +New parameters +~~~~~~~~~~~~~~ + +- `return_empty`: all transformers that find variables automatically have this new + parameter. If `return_empty=True` and the transformer finds no variables of the type it + works with, it raises a warning instead of an error, and returns the data unchanged. This is useful in + pipelines that run on datasets with different variable types. It is only used when + `variables=None`, and it defaults to `False`, which keeps the previous behaviour. + (`Jordi Pastor `_, `Míriam Méndez `_, `Soledad Galli `_) +- `C` in `LogTransformer()`: a constant added to the variables before applying the + logarithm, so you can now transform variables with zero or negative values. + `C="auto"` finds the constant automatically, like `LogCpTransformer()` did. It defaults + to `0`, which keeps the previous behaviour. (`Aditya Nikam `_) +- `n_jobs` in `DecisionTreeDiscretiser()`, `DecisionTreeEncoder()` and + `DecisionTreeFeatures()`: the number of jobs to run in parallel to train the decision + trees. It defaults to `None`, which trains the trees one after the other, as before. + (`Soledad Galli `_) + +Transformers that no longer raise errors +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following situations used to stop the transformers with an error. Now, the +transformers handle them and let you proceed: + +- **No variables of the right type**: with `return_empty=True`, transformers carry on when + the dataframe has no variables of the type they work with (see "New parameters" above). + (`Jordi Pastor `_, `Míriam Méndez `_, `Soledad Galli `_) +- **WoEEncoder()**: the weight of evidence is not defined for categories with no positive + or no negative cases. The encoder used to raise an error, unless you set `fill_value`. + Now, it replaces the zero count by 0.5 and lists the affected variables in the new + attribute `variables_with_zero_counts_`. (`Soledad Galli `_) +- **Winsoriser() and OutlierTrimmer()**: when most values of a variable are the same, the + capping method may find a spread of 0, for example, an IQR of 0 when more than half of + the values are 0. The transformers used to raise an error. Now, the variable has no + outliers: its limits are set to infinity in `right_tail_caps_` and `left_tail_caps_`, + and the variable is left untouched. (`Soledad Galli `_) +- **RandomSampleImputer()**: with `seed="observation"`, negative or very large values in + the variables used for the seed no longer raise an error. (`Soledad Galli `_) +- **DatetimeFeatures()**: extracting `week` or `days_in_month` from variables with + missing values with `missing_values="ignore"` failed. Now, those features are missing + for the rows with missing dates, like the other features. (`Coro `_) +- **DecisionTreeFeatures()**: combining single features failed when the dataframe had + integer column names. (`Soledad Galli `_) +- **ArbitraryOutlierCapper()**: passing an empty `min_capping_dict` together with a + `max_capping_dict`, or the other way round, failed in `fit()`. (`Soledad Galli `_) + +Renamed transformers +~~~~~~~~~~~~~~~~~~~~ + +We renamed the following transformers. The old names still work, but they raise a +`FutureWarning` and will be removed in version 2.1.0. To silence the warning, use the new +names: + +- `AddMissingIndicator()` is now `MissingIndicator()`. (`Muhammad Aniq Ramzan `_) +- `ArbitraryNumberImputer()` is now `ArbitraryImputer()`. (`Muhammad Aniq Ramzan `_) +- `CountFrequencyEncoder()` is now `CountEncoder()`. (`Soledad Galli `_) +- `MeanMedianImputer()` is now `MeanImputer()`. (`Ojas Sharma `_) +- `MeanNormalizationScaler()` is now `MeanNormalisationScaler()`. (`favour `_) +- `SelectByTargetMeanPerformance()` is now `SelectByTargetEncoding()`. (`Soledad Galli `_) +- `SklearnTransformerWrapper()` is now `SklearnWrapper()`. (`Soledad Galli `_) +- `Winsorizer()` is now `Winsoriser()`. (`favour `_) + +Deprecations +~~~~~~~~~~~~ + +- `LogCpTransformer()` is deprecated in favour of `LogTransformer(C="auto")`, and will be + removed in version 2.1.0. (`Aditya Nikam `_) +- The default value of `return_empty` will change from `False` to `True` in version 2.1.0, + in the transformers and in the variable handling functions. To keep the current + behaviour, set `return_empty=False` explicitly. +- The old names of the renamed transformers (see above) will be removed in version 2.1.0. + +Breaking changes +~~~~~~~~~~~~~~~~ + +- **Dataframes only**: transformers no longer accept numpy arrays as `X`. Pass a pandas or + polars dataframe instead. The target `y` can still be a Series, list or numpy array. + (`Soledad Galli `_) +- **Dependencies**: Feature-engine now requires Python 3.11 or newer, and scikit-learn 1.7 + or newer. pandas is no longer a required dependency: install pandas, polars, or both. + narwhals and python-dateutil are new dependencies, and statsmodels is no longer a + dependency. (`Soledad Galli `_, `Matias Loiseau `_) +- **Position of the parameters**: `return_empty` is placed right after `variables` in all + transformers. If you pass the parameters that come after `variables` by position, pass + them by name instead. +- **WoEEncoder()**: the parameter `fill_value` was removed. Categories with no positive or + no negative cases are now handled automatically (see above). (`Soledad Galli `_) +- **RandomSampleImputer()**: with `seed="observation"`, the seed of each observation is now + calculated from the values of the variables in `random_state` with a hash, instead of + their sum or product. Observations with the same values in those variables receive the + same imputation, wherever they are in the dataframe. As a result, the imputed values + differ from previous versions, and the parameter `seeding_method` was removed. + (`Soledad Galli `_) +- **DatetimeOrdinal()**: missing values in the output are now `NaN` in a float column with + pandas, and null in an integer column with polars. Before, they were `pd.NA` in an + object column. (`Soledad Galli `_) +- **TextFeatures()**: `lexical_diversity` now returns the number of unique words divided by + the total number of words, as documented. Before, it returned the inverse. + (`Venish Paneliya `_) + +Bug fixes +~~~~~~~~~ + +- `WindowFeatures()` ignored the parameter `min_periods`. It now uses it to calculate the + window features. (`Venish Paneliya `_) +- Imputers and discretisers now set their learned attributes only after `fit()` finishes + successfully. Before, an error in the middle of `fit()` left them partially fitted, and + `transform()` did not raise a `NotFittedError`. (`Direk Kakkar `_) +- The error message of `get_feature_names_out()` in the encoders now shows the features + it received. (`Soledad Galli `_) + +Performance +~~~~~~~~~~~ + +- `MathFeatures()` calculates sums, means, standard deviations, minimum and maximum values, + and other common aggregations much faster. (`Sudais Khan `_) +- `DatetimeOrdinal()` is between 27 and more than 200 times faster. + (`Soledad Galli `_) +- The discretisers are faster when they return the interval limits + (`return_boundaries=True`), and `EqualFrequencyDiscretiser()` and + `EqualWidthDiscretiser()` learn the intervals faster. (`Soledad Galli `_) +- `LogTransformer()`, `PowerTransformer()`, `ReciprocalTransformer()` and + `YeoJohnsonTransformer()` transform pandas dataframes faster. + (`Soledad Galli `_) +- On large datasets, polars is often faster than pandas. + +Code maintenance +~~~~~~~~~~~~~~~~ + +- Added performance benchmarks with CodSpeed, which run on every pull request. + (`Soledad Galli `_) +- Removed code needed to support scikit-learn 1.6 and older. (`Soledad Galli `_) +- The tests now check that `transform()`, `inverse_transform()` and + `get_feature_names_out()` raise a `NotFittedError` before `fit()`. + (`Temidayo Akindahunsi `_, `Ojas Sharma `_) +- The tests run with pandas and polars dataframes. (`Soledad Galli `_) +- Added an `AGENTS.md` file with the conventions for contributing code, tests and + documentation. (`Soledad Galli `_) + +Documentation +~~~~~~~~~~~~~ + +- Rewrote the homepage, the quickstart and the user guides of the creation, encoding, + imputation, outliers, scaling and transformation modules, and reorganised the user guide + and the API. (`Soledad Galli `_) +- Added examples with polars to the user guides and docstrings. (`Soledad Galli `_) +- Added a note on `return_empty` to the user guide of all transformers. + (`Soledad Galli `_) +- Corrected outputs in the examples of several user guides and docstrings that no longer + matched what the code returns. (`Soledad Galli `_) +- Fixed the user guide and API documentation of `DatetimeOrdinal()`. + (`Ankit Hemant Lade `_) +- Fixed the `precision` example in the `GeometricWidthDiscretiser()` documentation. + (`Maulik Patel `_) +- The return values now render in the API documentation. (`Venish Paneliya `_) +- Replaced the logo with one with a transparent background. + (`Míriam Méndez `_) +- Fixed spelling and grammar across the documentation. (`Soledad Galli `_)