Skip to content

[SYSTEMDS-3166] Refactor Isolation Forest builtin and add edge-case tests - #2498

Closed
PeilinChen01 wants to merge 5 commits into
apache:mainfrom
PeilinChen01:lde-iforest-cleanup
Closed

PeilinChen01 wants to merge 5 commits into
apache:mainfrom
PeilinChen01:lde-iforest-cleanup

Conversation

@PeilinChen01

@PeilinChen01 PeilinChen01 commented Jun 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This pull request contains the current LDE progress for SYSTEMDS-3166 and builds on the original Isolation Forest implementation in #2421.

Main changes:

  • Created 10 test scenarios for two execution modes, Single Node and Hybrid, resulting in 20 automated tests.
  • Added an R script for comparison.
  • Implemented the transfer of a model trained by SystemDS to R.
  • Verified that R and SystemDS produce the same anomaly scores when applying the same isolation forest.

Tests

The following command passes locally:

mvn -Dtest=BuiltinIsolationForestTest test

Covered test cases include:

  • Basic model training and scoring
  • Hybrid execution mode
  • Subsampling-size clamping
  • Single-row apply
  • Single-tree model
  • Constant data
  • Seed reproducibility
  • Anomaly ranking

Experiment results:

Data Size(rows) Feature Number Tree Number Subsampling Size Repetition SystemDS(s) R(s) SystemDS speedup vs R
1000-64000 30 100 256 20 3.85-3.89 0.95-0.97 ~0.25
1000-64000 50 100 512 3 7.56 2.55 ~0.33
1000-64000 30 100 256 20 4.1-4.6 0.95 ~0.22
1000-64000 3 10 20 20 0.075-0.138 0.005-0.006 ~0.06

rbbozkurt and others added 2 commits January 29, 2026 20:10
This patch promotes the existing Isolation Forest algorithm implementation from
the staging phase to builtin status, with improvements. The implementation provides
two main builtins, outlierByIsolationForest for training iForest models and
outlierByIsolationForestApply for scoring samples based on trained models.
Specifically, we optimized the algorithm with vectorized harmonic number
computation for improved scalability. The patch extends test coverage in
`staging/isolationForestTest.dml` with comprehensive tests, and Python API
integration tests. Refer to JIRA for detailed discussions.

Related to apache#1980

Co-authored-by: keremaras1 <60196502+keremaras1@users.noreply.github.com>
Co-authored-by: denizzqq <denizdolanmaz@rocketmail.com>
@PeilinChen01 PeilinChen01 changed the title Refactor Isolation Forest builtin and add edge-case tests [SYSTEMDS-3166] Refactor Isolation Forest builtin and add edge-case tests Jul 24, 2026
@PeilinChen01
PeilinChen01 marked this pull request as ready for review July 24, 2026 07:05
@mboehm7

mboehm7 commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

LGTM - thanks @PeilinChen01 @rbbozkurt and others for your contributions. During the merge, I only moved the java test to builtin/part1 to follow alphabetical ordering, disabled the benchmarking in the unit tests, and deleted the isolation forest from staging.

@mboehm7 mboehm7 closed this in f786949 Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants