Skip to content

fix(file-search): back off indexing retries after database timeouts - #8189

Merged
waleedlatif1 merged 1 commit into
stagingfrom
fix/workspace-file-search-statement-timeout
Sep 23, 2026
Merged

waleedlatif1 merged 1 commit into
stagingfrom
fix/workspace-file-search-statement-timeout

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • Indexing a small, ordinary text file could hit the 10 s statement timeout on its first chunk insert while the database was I/O-saturated. A 2–3 row insert (16–20 KiB, normal trigram density) ran 20–55 s before the cancel took effect. One row's direct GIN insert can't be interrupted, so the cancel only lands once that row is done, and smaller batches can't prevent it
  • The task retried after ~1 s and ~2 s, so all three attempts landed in the same slow window and the revision was marked failed for good, with an alert for each run
  • catchError now sorts failures into two kinds. Statement, lock, and transaction timeouts count as missing database capacity and back off about 2, 4, 8, 16, and 30 min (with jitter), six attempts in all, about an hour total. That is well inside the 6 h stale-dispatch window, and each attempt still starts a fresh fenced build. Every other failure keeps the old behavior: three attempts with the runner's default delays
  • No timeout bump and no change to batch sizing or chunking. README updated to match

Type of Change

  • Bug fix

Testing

  • New unit tests for the retry policy, driven by the redacted error the worker actually throws: delays in minutes for statement and lock timeouts, a ceiling on both the delay and the attempt count, and the default path for other failures and user cancels. The task test checks the retry ceiling and the catchError wiring
  • Mutation checks (file copy): removing catchError, going back to maxAttempts: 3, dropping the capacity branch, shrinking the base delay to 1 s, and removing the capacity attempt ceiling each turn at least one test red
  • vitest run lib/workspace-files/search background/workspace-file-search (188 passing), type-check, lint, check:audits (47), docs-manifest:check

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

A statement, lock, or transaction timeout while indexing a workspace file means the
database had no capacity for the build, not that the file is bad. The task retried it
within seconds, so all three attempts landed in the same slow window and the revision
was marked failed for good. Capacity timeouts now retry after about 2, 4, 8, 16 and
30 minutes (six attempts); other failures keep three attempts with the default delays.
@vercel

vercel Bot commented Sep 23, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Sep 23, 2026 5:18am UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 6 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@greptile-apps

greptile-apps Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge; the retry policy is bounded, correctly wired, and covered by focused tests.

Summary

This PR gives workspace-file indexing database-capacity timeouts a longer, jittered retry policy while preserving the existing short retry behavior for other failures.

  • Classifies statement, lock, and transaction timeouts as temporary capacity failures.
  • Schedules up to six attempts with approximately 2, 4, 8, 16, and 30-minute delays.
  • Stops ordinary failures after three attempts despite the task-level six-attempt ceiling.
  • Adds focused retry-policy and task-wiring tests.
  • Documents timeout behavior, dispatch-slot retention, and terminal failure handling.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Index workspace file revision] --> B{Attempt succeeds?}
  B -- Yes --> C[Publish complete fenced build]
  B -- No --> D{PostgreSQL capacity timeout?}
  D -- Yes --> E{Attempt below 6?}
  E -- Yes --> F[Schedule jittered 2–30 minute backoff]
  F --> A
  E -- No --> G[Stop retrying and mark revision failed]
  D -- No --> H{Attempt below 3?}
  H -- Yes --> I[Use runner default retry delay]
  I --> A
  H -- No --> G
Loading

Reviews (1) · Last reviewed commit: "fix(file-search): back off indexing retr..."

@waleedlatif1
waleedlatif1 merged commit a4a903c into staging Sep 23, 2026
34 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/workspace-file-search-statement-timeout branch September 23, 2026 06:20

This branch was previously deployed

1 inactive deployment
Preview — 27ea72bb Deployed Sep 23, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant