v0.8.48: search improvements, file versions

.claude/rules/global.md

@@ -51,7 +51,10 @@ Use shared helpers from `@sim/utils` instead of writing inline implementations:
 - `structuredClone(value)` — built-in deep clone, no import needed. Never write `JSON.parse(JSON.stringify(obj))`
 - `omit(obj, keys)` from `@sim/utils/object` — remove keys from object
 - `filterUndefined(obj)` from `@sim/utils/object` — strip undefined-valued keys. Never write `Object.fromEntries(Object.entries(obj).filter(([, v]) => v !== undefined))`
+- `isRecordLike(value)` from `@sim/utils/object` — indexable-object guard. Never redeclare `typeof value === 'object' && value !== null && !Array.isArray(value)`
 - `truncate(str, maxLength, suffix?)` from `@sim/utils/string` — safe string truncation with ellipsis
+- `escapeRegExp(value)` from `@sim/utils/string` — escape regex metacharacters. Never inline `replace(/[.*+?^${}()|[\]\\]/g, '\\$&')`
+- `compareStrings(left, right)` from `@sim/utils/string` — code-unit string comparator for hashes, fingerprints, and values compared across processes. Never `localeCompare` there
 - `backoffWithJitter(attempt, retryAfterMs, options?)` from `@sim/utils/retry` — exponential backoff with jitter
 - `parseRetryAfter(header)` from `@sim/utils/retry` — parse HTTP `Retry-After` header to milliseconds
 

.claude/rules/sim-imports.md

@@ -13,8 +13,8 @@ paths:
 
 ```typescript
 // ✓ Good
+import { Chip } from '@sim/emcn'
 import { useWorkflowStore } from '@/stores/workflows/store'
-import { Button } from '@/components/ui/button'
 
 // ✗ Bad
 import { useWorkflowStore } from '../../../stores/workflows/store'

.cursor/rules/global.mdc

@@ -54,7 +54,10 @@ Use shared helpers from `@sim/utils` instead of writing inline implementations:
 - `structuredClone(value)` — built-in deep clone, no import needed. Never write `JSON.parse(JSON.stringify(obj))`
 - `omit(obj, keys)` from `@sim/utils/object` — remove keys from object
 - `filterUndefined(obj)` from `@sim/utils/object` — strip undefined-valued keys. Never write `Object.fromEntries(Object.entries(obj).filter(([, v]) => v !== undefined))`
+- `isRecordLike(value)` from `@sim/utils/object` — indexable-object guard. Never redeclare `typeof value === 'object' && value !== null && !Array.isArray(value)`
 - `truncate(str, maxLength, suffix?)` from `@sim/utils/string` — safe string truncation with ellipsis
+- `escapeRegExp(value)` from `@sim/utils/string` — escape regex metacharacters. Never inline `replace(/[.*+?^${}()|[\]\\]/g, '\\$&')`
+- `compareStrings(left, right)` from `@sim/utils/string` — code-unit string comparator for hashes, fingerprints, and values compared across processes. Never `localeCompare` there
 - `backoffWithJitter(attempt, retryAfterMs, options?)` from `@sim/utils/retry` — exponential backoff with jitter
 - `parseRetryAfter(header)` from `@sim/utils/retry` — parse HTTP `Retry-After` header to milliseconds
 

.cursor/rules/sim-imports.mdc

@@ -13,8 +13,8 @@ globs: ["apps/sim/**/*.ts","apps/sim/**/*.tsx"]
 
 ```typescript
 // ✓ Good
+import { Chip } from '@sim/emcn'
 import { useWorkflowStore } from '@/stores/workflows/store'
-import { Button } from '@/components/ui/button'
 
 // ✗ Bad
 import { useWorkflowStore } from '../../../stores/workflows/store'

.github/workflows/test-build.yml

@@ -149,6 +149,18 @@ jobs:
           KNOWLEDGE_ACL_TEST_DATABASE_URL: postgresql://postgres:postgres@127.0.0.1:5432/sim_auth_scim
         run: bunx vitest run --mode integration lib/workspace-files/search/dispatcher.integration.ts
 
+      - name: Verify workspace file version history on PostgreSQL 17
+        working-directory: apps/sim
+        env:
+          KNOWLEDGE_ACL_TEST_DATABASE_URL: postgresql://postgres:postgres@127.0.0.1:5432/sim_auth_scim
+        run: bunx vitest run --mode integration lib/uploads/contexts/workspace/__integration__/file-versions.integration.ts
+
+      - name: Verify file search trigram estimate against pg_trgm
+        working-directory: apps/sim
+        env:
+          KNOWLEDGE_ACL_TEST_DATABASE_URL: postgresql://postgres:postgres@127.0.0.1:5432/sim_auth_scim
+        run: bunx vitest run --mode integration lib/workspace-files/search/index-plan.integration.ts
+
       - name: Verify SCIM and administration over real HTTP
         working-directory: apps/sim
         env:
@@ -219,6 +231,8 @@ jobs:
           bunx vitest run
           lib/table/rows/secret-provenance.postgres.test.ts
           lib/memory/message-provenance.postgres.test.ts
+          lib/memory/conversation-store.postgres.test.ts
+          lib/memory/summary-store.postgres.test.ts
           executor/handlers/agent/memory-harness.postgres.test.ts
 
       - name: Verify Search vector projection upgrade in PostgreSQL
@@ -229,6 +243,7 @@ jobs:
           bunx vitest run
           script-migrations/0016_backfill_search_vectors.postgres.test.ts
           script-migrations/0018_repair_workspace_file_content_revision.postgres.test.ts
+          script-migrations/0019_tin_keyword_projection.postgres.test.ts
 
       - name: Verify Search progress, pagination, and outbox scheduling in PostgreSQL
         working-directory: apps/sim
@@ -246,6 +261,8 @@ jobs:
           lib/knowledge/__integration__/stored-document-recovery.integration.ts
           lib/knowledge/__integration__/connector-partition-work.integration.ts
           lib/knowledge/__integration__/listing-continuation.integration.ts
+          lib/knowledge/__integration__/member-scope-renewal.integration.ts
+          lib/knowledge/__integration__/slack-empty-threads.integration.ts
           lib/knowledge/__integration__/kb-block-search.integration.ts
           lib/core/outbox/service.integration.ts
           lib/knowledge/__integration__/connector-upload.integration.ts

CLAUDE.md

@@ -16,7 +16,10 @@ You are a professional software engineer. All code must follow best practices: a
   - `getErrorMessage(e, fallback?)` from `@sim/utils/errors` — extract message string from unknown caught value; never write `e instanceof Error ? e.message : 'fallback'`
   - `structuredClone(value)` — built-in deep clone; never `JSON.parse(JSON.stringify(...))`
   - `omit(obj, keys)` / `filterUndefined(obj)` from `@sim/utils/object` — object trimming; never `Object.fromEntries(Object.entries(...).filter(...))`
+  - `isRecordLike(value)` from `@sim/utils/object` — never redeclare `typeof value === 'object' && value !== null && !Array.isArray(value)`
   - `truncate(str, maxLength, suffix?)` from `@sim/utils/string` — never inline slice + ellipsis
+  - `escapeRegExp(value)` from `@sim/utils/string` — never inline `replace(/[.*+?^${}()|[\]\\]/g, '\\$&')`
+  - `compareStrings(left, right)` from `@sim/utils/string` — code-unit ordering for hashes, fingerprints, and cross-process comparisons; never `localeCompare` there
   - `backoffWithJitter(attempt, retryAfterMs, options?)` / `parseRetryAfter(header)` from `@sim/utils/retry` — shared retry pacing; never reimplement exponential backoff inline
 - **Deployment flags in the browser**: client code inside a workspace, organization, or standalone settings surface reads `hosted`, `billingEnabled`, `chatEnabled`, and the enterprise feature set through `useDeploymentShape()` (components) or `getDeploymentShape()` (block conditions, stores, helpers) from `@/lib/core/config/deployment-shape`, never `isHosted`/`isBillingEnabled`/... from `env-flags`. The constants freeze at module init from the root layout's `NEXT_PUBLIC_*` transport, which Next's bare 404 shell and `global-error` never emit; the reader is seeded from the server-resolved workspace host context, organization layout, or standalone settings layout instead. Server code keeps reading `env-flags`
 - **Package Manager**: Use `bun` and `bunx`, not `npm` and `npx`

README.md

@@ -7,7 +7,7 @@
 
 <p align="center">
   <a href="https://deepwiki.com/simstudioai/sim" target="_blank" rel="noopener noreferrer"><img src="https://img.shields.io/badge/Ask-DeepWiki-E6E6E6?labelColor=C3C3C3&color=E6E6E6" alt="Ask DeepWiki"></a>
-  <a href="https://cursor.com/link/prompt?text=Help%20me%20set%20up%20Sim%20locally.%20Follow%20these%20steps%3A%0A%0A1.%20First%2C%20verify%20Docker%20is%20installed%20and%20running%3A%0A%20%20%20docker%20--version%0A%20%20%20docker%20info%0A%0A2.%20Clone%20the%20repository%3A%0A%20%20%20git%20clone%20https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim.git%0A%20%20%20cd%20sim%0A%0A3.%20Generate%20required%20secrets%20%28the%20stack%20will%20not%20start%20without%20them%29%3A%0A%20%20%20cat%20%3E%20.env%20%3C%3C%20EOF%0A%20%20%20BETTER_AUTH_SECRET%3D%24%28openssl%20rand%20-hex%2032%29%0A%20%20%20ENCRYPTION_KEY%3D%24%28openssl%20rand%20-hex%2032%29%0A%20%20%20INTERNAL_API_SECRET%3D%24%28openssl%20rand%20-hex%2032%29%0A%20%20%20CRON_SECRET%3D%24%28openssl%20rand%20-hex%2032%29%0A%20%20%20EOF%0A%0A4.%20Start%20the%20services%20with%20Docker%20Compose%3A%0A%20%20%20docker%20compose%20-f%20docker-compose.prod.yml%20up%20-d%0A%0A4.%20Wait%20for%20all%20containers%20to%20be%20healthy%20(this%20may%20take%201-2%20minutes)%3A%0A%20%20%20docker%20compose%20-f%20docker-compose.prod.yml%20ps%0A%0A5.%20Verify%20the%20app%20is%20accessible%20at%20http%3A%2F%2Flocalhost%3A3000%0A%0AIf%20there%20are%20any%20errors%2C%20help%20me%20troubleshoot%20them.%20Common%20issues%3A%0A-%20Port%203000%2C%203002%2C%20or%205432%20already%20in%20use%0A-%20Docker%20not%20running%0A-%20Insufficient%20memory%20(needs%2012GB%2B%20RAM)%0A%0AFor%20local%20AI%20models%20with%20Ollama%2C%20use%20this%20instead%20of%20step%203%3A%0A%20%20%20docker%20compose%20-f%20docker-compose.ollama.yml%20--profile%20setup%20up%20-d"><img src="https://img.shields.io/badge/Set%20Up%20with-Cursor-E6E6E6?logo=cursor&logoColor=1A1A1A&labelColor=C3C3C3&color=E6E6E6" alt="Set Up with Cursor"></a>
+  <a href="https://cursor.com/link/prompt?text=Help%20me%20set%20up%20Sim%20locally.%20Follow%20these%20steps%3A%0A%0A1.%20First%2C%20verify%20Docker%20is%20installed%20and%20running%3A%0A%20%20%20docker%20--version%0A%20%20%20docker%20info%0A%0A2.%20Clone%20the%20repository%3A%0A%20%20%20git%20clone%20https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim.git%0A%20%20%20cd%20sim%0A%0A3.%20Generate%20required%20secrets%20%28the%20stack%20will%20not%20start%20without%20them%29%3A%0A%20%20%20cat%20%3E%20.env%20%3C%3C%20EOF%0A%20%20%20BETTER_AUTH_SECRET%3D%24%28openssl%20rand%20-hex%2032%29%0A%20%20%20ENCRYPTION_KEY%3D%24%28openssl%20rand%20-hex%2032%29%0A%20%20%20INTERNAL_API_SECRET%3D%24%28openssl%20rand%20-hex%2032%29%0A%20%20%20CRON_SECRET%3D%24%28openssl%20rand%20-hex%2032%29%0A%20%20%20POSTGRES_PASSWORD%3D%24%28openssl%20rand%20-hex%2024%29%0A%20%20%20EOF%0A%0A4.%20Start%20the%20services%20with%20Docker%20Compose%3A%0A%20%20%20docker%20compose%20-f%20docker-compose.prod.yml%20up%20-d%0A%0A4.%20Wait%20for%20all%20containers%20to%20be%20healthy%20(this%20may%20take%201-2%20minutes)%3A%0A%20%20%20docker%20compose%20-f%20docker-compose.prod.yml%20ps%0A%0A5.%20Verify%20the%20app%20is%20accessible%20at%20http%3A%2F%2Flocalhost%3A3000%0A%0AIf%20there%20are%20any%20errors%2C%20help%20me%20troubleshoot%20them.%20Common%20issues%3A%0A-%20Port%203000%20or%203002%20already%20in%20use%0A-%20Docker%20not%20running%0A-%20Insufficient%20memory%20(needs%2012GB%2B%20RAM)%0A%0AFor%20local%20AI%20models%20with%20Ollama%2C%20use%20this%20instead%20of%20step%203%3A%0A%20%20%20docker%20compose%20-f%20docker-compose.ollama.yml%20--profile%20setup%20up%20-d"><img src="https://img.shields.io/badge/Set%20Up%20with-Cursor-E6E6E6?logo=cursor&logoColor=1A1A1A&labelColor=C3C3C3&color=E6E6E6" alt="Set Up with Cursor"></a>
 </p>
 
 <p align="center">

apps/desktop/src/main/ipc.ts

@@ -27,7 +27,7 @@ import {
   type TerminalToolArgs,
 } from '@sim/terminal-protocol'
 import { getErrorMessage } from '@sim/utils/errors'
-import { isRecordLike } from '@sim/utils/object'
+import { isRecordLike, toRecord } from '@sim/utils/object'
 import { PASTE_LIMITS, utf8ByteLength } from '@sim/utils/paste'
 import type { BrowserWindow, IpcMainEvent, IpcMainInvokeEvent, WebContents } from 'electron'
 import { clipboard, ipcMain, shell } from 'electron'
@@ -856,7 +856,7 @@ export function registerIpcHandlers(deps: IpcDeps): void {
         ) {
           return { ok: false, error: `Unknown browser tool: ${String(tool)}` }
         }
-        const toolParams = isRecordLike(params) ? params : {}
+        const toolParams = toRecord(params)
         return executeTool(
           scope,
           tool,
@@ -1532,7 +1532,7 @@ export function registerIpcHandlers(deps: IpcDeps): void {
         ) {
           return { ok: false, error: `Unknown terminal tool: ${String(tool)}` }
         }
-        const call = isRecordLike(params) ? params : {}
+        const call = toRecord(params)
         if (!isTerminalOperation(call.operation)) {
           return { ok: false, error: `Unknown terminal operation: ${String(call.operation)}` }
         }

apps/desktop/src/main/local-filesystem.ts

@@ -19,6 +19,7 @@ import {
 } from '@sim/desktop-bridge/local-filesystem-limits'
 import { generateId } from '@sim/utils/id'
 import { isRecordLike } from '@sim/utils/object'
+import { escapeRegExp } from '@sim/utils/string'
 import { app, dialog, shell } from 'electron'
 import micromatch from 'micromatch'
 import safeRegex from 'safe-regex2'
@@ -1114,7 +1115,7 @@ export class LocalFilesystemService {
       regex =
         rawPattern !== undefined
           ? new RegExp(expression, ignoreCase ? 'i' : '')
-          : new RegExp(expression.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'), ignoreCase ? 'i' : '')
+          : new RegExp(escapeRegExp(expression), ignoreCase ? 'i' : '')
     } catch {
       // An empty result set would tell the model the string appears nowhere in
       // the user's files — a factual claim it will act on, when in truth the

apps/docs/content/docs/academy/agents/memory.mdx

@@ -16,7 +16,9 @@ import { AV_MEMORY_WORKFLOW } from '@/components/workflow-preview/academy-video-
 <div className="mt-8 grid grid-cols-1 items-start gap-10 lg:grid-cols-[1fr_300px]">
 <div>
 
-By default, an agent keeps nothing between runs: every conversation starts completely fresh. The Memory setting changes that: choose Conversation, give it a conversation ID, and everything said under that key is kept and loaded back before the model runs.
+By default, an agent keeps nothing between runs: every conversation starts completely fresh. The Memory setting changes that: choose Conversation, give it a conversation ID, and the agent saves the conversation under that key. Later runs load history according to the memory mode and the model's context limits.
+
+When durable tool history is enabled for your workspace, that history also includes completed tool calls and their results or errors. The agent can remember what it did, such as looking up an order, even when the run failed before giving its final answer. See the [Agent block's memory settings](/workflows/blocks/agent#memory) for tool-history and retry behavior.
 
 Uploaded attachments stay linked to the message that included them. Memory stores file references; each later run reads the accessible files again and prepares them for the selected provider. Attachments follow the selected memory window and the source file's storage retention. A replay can include up to 20 attachment references; use a smaller memory window for longer file-heavy conversations. Files omitted by older versions of memory need to be attached again.
 
@@ -32,7 +34,7 @@ Uploaded attachments stay linked to the message that included them. Memory store
     },
     {
       title: 'Recall happens before the model runs',
-      body: 'On the next run, everything stored under the key is loaded back into the conversation first: so the agent answers like no time has passed.',
+      body: 'On the next run, selected history under the key is loaded into the conversation before the model answers.',
     },
     {
       title: 'Keys are separate threads',
@@ -65,20 +67,22 @@ Here is the agent from the video with Memory set on the block:
 
 ## The same agent, with and without memory
 
-The video runs the same agent twice, side by side: once with no memory and once with the conversation ID. The same follow-up question arrives in both. Without the key, the agent starts from zero and has to ask for everything again; with it, everything stored under the key was loaded back before the model saw the new message, and the answer picks up exactly where yesterday stopped.
+The video runs the same agent twice, side by side: once with no memory and once with the conversation ID. The same follow-up question arrives in both. Without the key, the agent starts from zero and has to ask for everything again; with it, the earlier conversation supplies the context needed to answer the follow-up.
 
 ## When conversations grow
 
 Memory can also be a sliding window, keeping the most recent messages, or the most recent tokens, while the oldest quietly fall away. The stored transcript keeps every turn; the window controls how much of it rides into the model on each run.
 
+Tool exchanges stay together when history is selected. They do not each count as a message in a message window, but their contents still use input tokens. Large results may appear as previews, and history is not automatically summarized. A token window gives more direct control over recalled context than a message count; neither setting caps the total cost of a run that makes further model or tool calls.
+
 ## When to use memory
 
 Enable memory when a follow-up needs earlier conversation context, such as a support ticket or sales conversation. Keep classification and extraction stateless when each input contains everything the task needs.
 
 <FAQ items={[
-  { question: "What memory modes does the Agent block support?", answer: "Conversation keeps every turn under a conversation ID. Sliding window (messages) keeps the most recent messages, and sliding window (tokens) keeps as much recent history as fits a token budget." },
+  { question: "What memory modes does the Agent block support?", answer: "Conversation recalls stored history subject to history-loading and model context limits. Sliding window (messages) selects the most recent messages, and sliding window (tokens) selects recent history against a token budget. Tool-call groups stay together." },
   { question: "What should I use as a conversation ID?", answer: "Any stable key that identifies the thread: a customer ID, a ticket number, a channel ID. Runs that share the key share the conversation; runs with different keys never see each other's history." },
-  { question: "Does agent memory cost extra tokens?", answer: "Recalled history is loaded into the model's context, so it is billed as input tokens like any other message. Sliding-window modes exist to cap that cost on long-running conversations." },
+  { question: "Does agent memory cost extra tokens?", answer: "Recalled messages, tool arguments, and tool results use input tokens. Sliding-window modes reduce how much history is recalled, but a message count is not a token limit and a memory window does not cap the total cost of the run." },
   { question: "Where is the conversation stored?", answer: "In your Sim workspace, as a stored transcript per conversation ID. You can inspect it, and the agent reads from it automatically before each run." },
 ]} />