Skip to content

fix(files): lock the workspace row before inserting a workspace file - #8972

Merged
waleedlatif1 merged 3 commits into
stagingfrom
fix/file-write-lock-order
Oct 13, 2026
Merged

waleedlatif1 merged 3 commits into
stagingfrom
fix/file-write-lock-order

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • Fixes deadlock detected (40P01) between a file content write and a concurrent new-file insert in the same workspace. It recurs in bursts when a workspace gets concurrent uploads and overwrites, e.g. an agent writing many files.
  • Cycle:
    • updateWorkspaceFileContent locks the workspace row (lockWorkspaceStorageForMutationInTx, FOR NO KEY UPDATE), then UPDATEs the file. The AFTER trigger workspace_files_search_index_pending then upserts the one-row-per-workspace workspace_file_search_dispatch_queue row.
    • uploadWorkspaceFile, registerUploadedWorkspaceFile and fork executeForkFileBlobCopies INSERT the file first, so the trigger takes the queue row. They lock the workspace row only later, in incrementStorageUsageForBillingContextInTx.
    • In prod logs the blocked content write was "inserting index tuple in relation workspace_file_search_dispatch_queue". The transaction blocking it was waiting on select … from workspace … for no key update.
  • Fix: call lockWorkspaceStorageForMutationInTx right before the file insert in those three paths and in createOwnedWorkspaceFile.
    • createOwnedWorkspaceFile uses non-workspace contexts, so the trigger never reaches the queue row today. It gets the same order for consistency.
    • The increment's later re-lock is already held, so it never waits.
  • Documents the canonical order in the TSDoc of lockWorkspaceStorageForMutationInTx: folder advisory lock → workspace row → file rows → file writes (trigger queue row) → payer row.
  • The trigger is unchanged. Making its queue upsert ON CONFLICT DO NOTHING would lose wakeups.

Why it's safe

  • No new lock pair with the folder lock. Upload and register already took the folder advisory lock and then the workspace row (via the increment). Only the workspace row moves ahead of the file/queue writes.
    • Every acquireFolderMutationLock / withFolderTreeLock call is the first lock in its transaction. Mapped-import and lockForkRevision take the folder lock before the workspace row.
    • So no transaction holds the workspace row and then waits on the folder lock.
  • The other workspace-row lockers already follow workspace → files: content write, purgeCreatedWorkspaceFile, cleanup-soft-deletes, payer transfer, and fork KB rollback.
  • The other workspace_files inserts don't charge storage in the same transaction, or don't reach the queue row:
    • chat uploads use mothership context.
    • upload finalizers use execution context and have no tx.
    • chat fork copies don't charge storage in that transaction.
    • KB ownership rows use knowledge-base context, which the trigger skips.
  • The dispatcher never locks workspace rows. It claims queue rows with SKIP LOCKED.
  • Lock waits are unchanged. Upload/register wait under the same 5s lock_timeout the folder lock already sets. The lock is taken after folder-path resolution, so the workspace row is held only across the insert and the increment that already followed it.
  • Extra cost: one cheap re-select of an already-held row per insert. Race and replay paths that end up not charging now also briefly hold the workspace row.
  • No behavior, API or schema changes.

Type of Change

  • Bug fix

Testing

  • New apps/sim/lib/uploads/contexts/workspace/__integration__/storage-lock-order.integration.ts runs against real Postgres, once each for uploadWorkspaceFile and registerUploadedWorkspaceFile:
    • One transaction holds the workspace row the way updateWorkspaceFileContent does.
    • The finalize path starts, and the test polls until it is blocked by that transaction.
    • The holder then bumps content_updated_at on an existing file in the same workspace. Its trigger upserts the queue row the seed upload created.
    • Asserts that both transactions commit.
  • Pre-fix, the test fails (reasoned, not run locally):
    • With the old order, the finalize path has already inserted its row (trigger → queue row lock) before blocking on the workspace row in the increment.
    • The holder's UPDATE then waits on the queue row, which closes the cycle.
    • Postgres aborts one side with 40P01 after deadlock_timeout (1s, well under the 5s lock_timeout), so a settled result is rejected.
    • Post-fix, the finalize path blocks before its insert, the holder's trigger finds the queue row free, and both commit.
  • Ran biome on the changed files. Per repo policy I did not run type-check, check:audits or integration suites locally; CI runs them.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Upload, upload-session registration, owned-file creation, and fork file copy
inserted the workspace_files row first and locked the workspace row only in
the later storage increment. The insert's search-index trigger upserts the
per-workspace workspace_file_search_dispatch_queue row, so these transactions
held the queue row while waiting on the workspace row. updateWorkspaceFileContent
takes the same two locks in the opposite order (workspace row, then the content
UPDATE whose trigger reaches the queue row), and the two deadlocked.

Take lockWorkspaceStorageForMutationInTx before the insert in each path so every
storage-charging file write locks folder -> workspace row -> file rows/queue ->
payer, and document that order on the lock helper.
@vercel

vercel Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Oct 13, 2026 6:35pm UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/lib/billing/storage/tracking.ts Outdated
@greptile-apps

greptile-apps Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[High impact] The PR appears safe to merge; no new actionable issues remain.

Summary

This PR makes file inserts lock the workspace row before the search trigger writes its queue row. That matches the order used by concurrent content writes.

  • Workspace file inserts lock the workspace before writing file rows.

No new actionable issues were found. Both previous unnumbered findings are addressed. Tests were not run during this review.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A["Folder lock, when needed"] --> B["Workspace row lock"]
  B --> C["File insert or update"]
  C --> D["Search trigger writes queue row"]
  D --> E["Storage charge locks payer row"]
Loading

Reviews (2) · Last reviewed commit: "fix(files): skip the lock-order test wit..." · Reviewed by Greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 4 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1
waleedlatif1 merged commit 2abff53 into staging Oct 13, 2026
51 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/file-write-lock-order branch October 13, 2026 19:12

This branch was previously deployed

1 inactive deployment
Preview — ad85d9a0 Deployed Oct 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant