Repository navigation
Conversation
stepIntoStrings(buffer, capacity, ...params) steps up to `capacity` rows per native call. Numbers are written into a caller-provided ArrayBuffer or SharedArrayBuffer (an 8-byte slot and a tag byte per value); TEXT and BLOB values are created natively and returned as one array per call. A call that does not exhaust the result leaves a cursor open, which later calls continue and closeCursor() abandons. An open cursor locks the statement and counts as an open iterator, exactly like iterate(). On a 92-row, 6-column range fetch this is 253 ns/row with plain row objects built in JS, against 598 ns/row for iterate() and 164 ns/row for the same scan done entirely in SQL.
There was a problem hiding this comment.
🟡 Changes recommended
Statement destruction can leave cursor accounting and database locking permanently active.
1 open issue
What changed in this PR
Adds native batched row extraction via Statement#stepIntoStrings() and cursor cleanup via closeCursor().
Changes:
- Adds batch encoding for numeric, TEXT, BLOB, and NULL values.
- Adds cursor locking, parameter binding, typings, documentation, and tests.
- A critical cursor-accounting cleanup issue remains during statement destruction.
| File | Description |
|---|---|
test/27.statement.step-into-strings.js |
Tests batching, locking, errors, and bindings. |
src/util/binder.cpp |
Supports offset parameter binding. |
src/objects/statement.hpp |
Declares cursor state and methods. |
src/objects/statement.cpp |
Registers the new APIs. |
src/objects/statement-batch.cpp |
Implements batched stepping and cursor management. |
src/better_sqlite3.cpp |
Includes the batch implementation. |
lib/index.d.ts |
Adds TypeScript declarations. |
docs/api.md |
Documents the APIs and buffer format. |
Review effort: Lite
| void Statement::CloseCursor() { | ||
| if (!cursor_open) return; | ||
| cursor_open = false; | ||
| locked = false; | ||
| db->GetState()->iterators -= 1; | ||
| if (alive) { | ||
| sqlite3_reset(handle); | ||
| if (!bound) sqlite3_clear_bindings(handle); | ||
| } |
There was a problem hiding this comment.
Not changed, on purpose: this matches iterate(). An abandoned StatementIterator also never releases stmt->locked or db_state->iterators; statement-iterator.hpp says "we actually DON'T want to revert stmt->locked or db_state->iterators in this destructor, to ensure deterministic database access", so whether writes or close() succeed never depends on when GC runs. Callers must finish the cursor or call closeCursor(), as docs/api.md says, and a test covers the abandoned-cursor behavior. Leaving the thread open in case a maintainer prefers to diverge from iterate() here.
🤖 Addressed by Claude Code
A user-defined function runs JavaScript during sqlite3_step() and could detach the buffer (transfer()) or shrink a resizable one while rows are being written. Hold the BackingStore for the whole call and reject resizable ArrayBuffers. Also reserve the values vector by rows and columns rather than capacity * 4.
# Conflicts: # src/objects/statement.hpp
|
Superseded by #57: StatementIterator#nextBatch() + #restart() measures the same in zero-cache (within ±2%) with about a third of the native code and no new buffer API. |

Adds
Statement#stepIntoStrings(buffer, capacity, ...bindParameters)andStatement#closeCursor(). zero-cache's TableSource will use them to read range-fetch rows in batches instead of oneiterate()step per row.What it does
Each call steps up to
capacityrows.ArrayBufferorSharedArrayBuffer: an 8-byte slot per value (int64 / double / array index) and a tag byte per value.16 + 9 * capacity * columnCountbytes, documented indocs/api.mdwith an example decoder. Because every row fits, there is no overflow or partial-row case.pluck()/expand()/raw()/safeIntegers()don't apply; the caller converts.When a call doesn't exhaust the result (header
doneis0), the statement stays locked and counts as an open iterator, exactly likeiterate().stepIntoStrings(buffer, capacity)with no bind parameters continues it, andcloseCursor()abandons it.Behavior that mirrors
iterate()db.close(): also throws "busy" while a cursor is open, so a cursor can't outlive its database handle.closeCursor()afterclose()is a no-op.Why
Measured on the
rowbenchcomment table from zero-cache profiling: a 92-row, 6-column range fetch (5 TEXT columns, one of them 200 bytes), with a 256 MB page cache and the median of 7 interleaved runs.iterate(), row objects (current zero-cache path)stepIntoStringsK=128, plain row objects built in JScount(length(body))in SQL, one callStrings are created natively on purpose. In the prototype, decoding UTF-8 from the buffer in JS (
Buffer#latin1SliceorTextDecoder) cost 45–97 ns per string and gave back most of the gain. Creating them in the addon costs about 12 ns per string.Testing
test/27.statement.step-into-strings.js(19 tests) covers:raw().all();SharedArrayBuffer;closeCursor();The full suite passes locally on Node 22.23, 24.18 and 26.12 (366 tests).
Not included: a version bump or release.