Fix flakiness detection querying the security phase timeout by hoco-scy · Pull Request #1852 · WebFuzzing/EvoMaster

hoco-scy · 2026-10-08T09:56:03Z

Problem

FlakinessDetector.hasPhaseTimedOut() queries Phase.SECURITY instead of Phase.FLAKINESS.

ExecutionPhaseController retains timeout flags for completed phases. If the security phase has timed out, querying its flag returns true even after the flakiness phase starts with a fresh timer. Consequently, the timeout guards in static inference and re-execution can stop immediately, skipping flakiness detection.

Fix

Change the queried phase from SECURITY to FLAKINESS.

This is a one-line fix. It preserves the existing phase-wide time budget and does not add configuration options, change timeout policy, or modify result handling.

Validation

Existing EMConfigTest, RestCallResultTest, and FlakinessInferenceUtilTest passed.
Three additional local checks using the real ExecutionPhaseController passed:
1. After a recorded security timeout, a newly started flakiness phase is not timed out. The original SECURITY query still returns true in this same state.
2. After that transition, flakiness still times out when its own budget is exceeded.
3. When security finishes without timing out, flakiness still respects its own budget.

The targeted checks simulated elapsed time by adjusting the controller's start timestamp; they were run locally and are not included in this minimal diff.

man-zhang · 2026-10-10T07:03:43Z

+            if (flakyList != null && flakyList.size <= i){
                break
            }
            val flakyItem = if (flakyList != null) flakyList[i] else null


If i equals flakyList.size, could accessing flakyList[i] here cause an index-out-of-bounds exception?

Kotlin lists are zero-indexed, so flakyList[size] is out of bounds. Changing < to <= makes the loop break when i == flakyList.size, before reaching flakyList[i]. The previous condition missed this boundary case.

man-zhang · 2026-10-10T07:14:19Z

I am not sure about this. If the sizes of merged and original differ, why do we return merged directly (its size may be greater or less than original)?

Updated in 06ebbceb4c. The early return could miss element changes in later observations, so the merge now preserves the size difference while continuing to process shared indices.

For example, with original [1,2] and observations [1] then [9,2], the merged result is now [9] instead of [1].

Added regression tests for both observation orders, nested arrays, and generated assertions. All 68 related tests pass.

man-zhang · 2026-10-13T03:29:39Z

@arcuri82 this PR is ready to review.
It supports configuring the budget percentage for flakiness handling and collecting statistics of the time spent on the re-execution for detecting flakiness. In addition, several bugs have been fixed.

arcuri82 · 2026-10-15T10:32:46Z

                break
            }
-            if (flakyList != null && flakyList.size < i){
+            if (flakyList != null && flakyList.size <= i){


the i is from list, isn't it? shouldn't add at beggining of handleAssertionsOnList some explicit checks on the sizes of the two input data structures, and throw an IllegalArgumentException?

hi, i is valid for list eg [1,2], but the flakyList may be shorter, eg, [3]. then in this case, the code is to ensure to avoid the IndexOutOfBoundsException when getting flakyList[i]

so, could add a if(flakyList!=null && flakyList.size > list.size) throw IllegalArgumentException at the beginning of the function

also, it is more readable when the incrementing index is on left side, ie i >= flakyList.size

arcuri82 · 2026-10-15T10:51:22Z

+     * @return whether the action is follow-up action
+     */
+    fun isFollowUpAction(): Boolean{
+        return path.toString() == "/" && id.contains(HTTP_CALL_LOCATION_FOLLOWUP_KEY)


is path.toString() == "/" necessary? looks like hardcoding the placeholder from HttpSemanticService`

arcuri82 · 2026-10-15T10:52:28Z

 import org.evomaster.core.problem.rest.link.RestLink
 import org.evomaster.core.problem.rest.param.PathParam
 import org.evomaster.core.problem.rest.resource.ActionRToken
+import org.evomaster.core.problem.rest.service.HttpSemanticsService.Companion.HTTP_CALL_LOCATION_FOLLOWUP_KEY


maybe this constant should be moved here in this class? as it might be used in different places in future, and then check forisFollowUpAction is here anyway

arcuri82 · 2026-10-15T10:57:26Z


+    /**
+     * @return whether the action is follow-up action
+     */


in docs, should explain what a follow-up action is. also, if the main problem is the lack of valid RestPath, then the name "follow-up" is confusing here. maybe something like isDirectLocationAction, explaining it is an action that alwasy use full Location from a previous response, and ignore its RestPath

arcuri82 · 2026-10-15T10:59:28Z

+        val seq = seeMainExecutableActions()
+        while (cur >= 0){
+            val found = seq[cur]
+            if (!found.isFollowUpAction())


eh? isn't this just returning the first previous non-followup action? that can be incorrect. as those actions are bound, you can use the links to refer to the source action in RestCallAction fields

arcuri82 · 2026-10-15T11:00:42Z

+    /**
+     * for the follow-up action, get its previous valid rest action
+     */
+    fun getActualPathForFollowUpAction(index: Int): RestPath?{


what does Actual mean here? the name of this method feels confusing. isn't rather the path of the source of Location FUA depends on?

arcuri82 · 2026-10-15T11:31:15Z

-    private fun updateParamInfo(action: RestCallAction, tables: Map<TableId, Table>) {
-        val r = rm.getResourceNodeFromCluster(action.path.toString())
+    private fun updateParamInfo(action: RestCallAction, path: RestPath?, tables: Map<TableId, Table>) {
+        val rPath = if (action.isFollowUpAction()) path else action.path


in the else, should throw exception if path!=null?

arcuri82 · 2026-10-15T11:38:24Z

        }

-        val budget = (config.timeLimitInSeconds() * config.extraPhaseBudgetPercentage).toLong()
+        val extraPercentage = if (target == Phase.FLAKINESS) config.extraBudgetPercentageForDetectFlakiness else config.extraPhaseBudgetPercentage


this is confusing, as now extraPhaseBudgetPercentage is silently ignored for Flakiness. I would prefer something else like unboundBugetForFlakiness:Boolean where, if true, then no bduget checks are done for flakiness phase. false by default

arcuri82 · 2026-10-15T11:45:05Z

    internal fun getSqlInsertionExecutionTimeMs() = sqlInsertionExecutionTimeMs
    internal fun getSqlInsertionExecutionCount() = sqlInsertionExecutionCount
+    internal fun getReExecutionTimesMs(): List<Long> = reExecutionTimesMs.toList()
+    internal fun getReExecutionTimeMs() = reExecutionTimesMs.sum()


those 2 names only differ with a single s in the middle. confusing. maybe rename first as getAllReExecutionTimesMs and second as getSumOfReExecutionTimesMs

arcuri82 · 2026-10-15T11:46:58Z

+    @Cfg("Specify extra budget percentage of the search budget for flakiness detection phase." +
+            "A negative value indicates an unlimited time budget.")
+    @Min(-1.0)
+    var extraBudgetPercentageForDetectFlakiness = -1.0


rather use unboundBugetForFlakiness. also, if this is left as it is, it would mean flakiness phase was always unbound by default?

…tectFlakiness

…nstead of the first non-follow-up action

arcuri82 · 2026-10-18T12:52:48Z

                break
            }
-            if (flakyList != null && flakyList.size < i){
+            if (flakyList != null && flakyList.size <= i){


should rather have i >= flakyList.size with a if(flakyList!=null && flakyList.size > list.size) throw IllegalArgumentException. also add javadocs to explain how flakyList relates to list

arcuri82 · 2026-10-18T12:55:13Z

+            val locationId = seq[cur].usePreviousLocationId
+            cur = (cur - 1 downTo 0).firstOrNull {
+                val candidate = seq[it]
+                candidate.isPotentialActionForCreation() && candidate.creationLocationId() == locationId


why checking for isPotentialActionForCreation ? technically a Location could be unrelated to create

if there is curently a limitation in the algorithm, could leave a TODO comment

cy-shen-ml and others added 5 commits October 8, 2026 16:56

Fix flakiness timeout check using the wrong phase

66c04b2

add unlimited time budget

80c2cfa

skip to detect flakiness for tests which have timeout action

37c9561

collect statistics for each re-execution round for flakiness detection

1a00721

Fix array bounds during flaky response merging and assertion generation

1d10aee

man-zhang requested changes Oct 10, 2026

View reviewed changes

cy-shen-ml and others added 5 commits October 10, 2026 15:44

Merge element differences even when flaky array sizes change

06ebbce

test: reproduce resource lookup failure for dynamic Location follow-up

1948190

remove the test

7e5f9cd

fix getting resource node for location follow up action

791a5ac

add a test to check if flakiness can be worked together with httpOracles

5fceedf

man-zhang requested a review from arcuri82 October 13, 2026 03:29

arcuri82 requested changes Oct 15, 2026

View reviewed changes

man-zhang and others added 8 commits October 15, 2026 21:37

define unboundBugetForFlakiness instead of extraBudgetPercentageForDe…

ef98e58

…tectFlakiness

better name for getter of re-execution time

bbf80a4

better doc and move location follow-up key to rest call action

505fa08

fix a bug to get path of source action of location follow-up action i…

a478e2d

…nstead of the first non-follow-up action

add value validation for non- direction location action case

f98bc8f

Merge branch 'master' into fix/flakiness-phase-timeout

8d0e0e3

fix a bug

f70710a

fix the default setting of unboundBudgetForFlakiness

712b9e5

man-zhang requested a review from arcuri82 October 18, 2026 05:37

arcuri82 requested changes Oct 18, 2026

View reviewed changes

Conversation

hoco-scy commented Oct 8, 2026

Problem

Fix

Validation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

man-zhang commented Oct 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants