Skip to content

Stuck futures and hard timeout #409

Closed

Description

@HenriqueAssumpcao

I was running some experiments in which the evaluator file did not directly implement a hard timeout for a given program, thus a program could run for an arbitrary amount of time. The code in process_parallel.py (L532-778) first finds a given future that is done, and then proceeds to a try-except block to recover the result. We only enforce the timeout with a future.result(timeout=timeout) call, but this is only done to futures that were completed, i.e., the ones with future.done() == True. I think this only handles futures that are done but are stuck in returning their results, but not futures that haven't been completed and are stuck or running for more than the timeout budget. In my experiments, the timeout was 60s, but some futures took over 1500s to complete, which leads to all sorts of weird situations. I know this could be solved by implementing a hard timeout in the evaluator file, but I think it would be nice if this was also enforced by openevolve itself. If this makes sense, I can make a PR later.

Activity

  1. added
    bugSomething isn't working
    enhancementNew feature or request
    help wantedExtra attention is needed
    on Feb 12, 2026
  2. codelion commented on Feb 12, 2026

    @codelion
    Member

    Yes this would be good to handle in OpenEvolve.

  3. codelion commented on Sep 28, 2026

    @codelion
    Member

    Fixed on main by #485. OpenEvolve already applies evaluator.timeout to every evaluation stage, but when a stage timed out, the worker still waited for the orphaned thread at shutdown. That is why futures could run far past the timeout. Evaluations now use a dedicated executor, so a timed-out evaluation returns promptly and the worker moves on. One limitation remains: Python can't kill a running thread, so a runaway program keeps using CPU in the background until it finishes. For hard kills, run the candidate in a subprocess from your evaluator. Closing, but please reopen if you still see stuck futures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions