Skip to content

feat(AIC-3363): Support inline datasets - #95

Open
aknight-ld wants to merge 5 commits into
mainfrom
AIC-3363-inline-datasets
Open

aknight-ld wants to merge 5 commits into
mainfrom
AIC-3363-inline-datasets

Conversation

@aknight-ld

@aknight-ld aknight-ld commented Sep 17, 2026

Copy link
Copy Markdown

Server side support of inline datasets is in flight; this PR covers the corresponding SDK work to pass in dataset rows defined from code.

Sizable test suite added in, testing underway to ensure we are matching the server.


Note

Overview
Adds inline datasets so init_evaluations().run() accepts either a hosted dataset key or caller-supplied rows=[DatasetRow(...)], with exactly one required and no dataset API traffic for inline runs.

Inline rows are normalized through shared _render_row (templates, injected input/expected_output in variables) without mutating caller objects. Pre-flight validation covers row count (10k), unique non-negative row_index, JSON-serializable variables/metadata, and rendered UTF-8 byte limits (1 MiB per field) so bad payloads fail before generation or silent event loss.

Generation events for inline runs omit datasetId/datasetKey (five-field eventId) and include row input, expectedOutput, variables, and metadata; hosted runs keep excluding row bodies. Run creation sends {"source": "api"} without datasetId when inline. Criterion payloads treat dataset identifiers as optional.

Docs in packages/ai and packages/client READMEs plus agents.md describe the new mode and pitfalls. A large test_evaluations_run.py suite covers API shapes, events, rendering, limits, and validation errors.

Reviewed by Cursor Bugbot for commit d9971cd. Bugbot is set up for automated code reviews on this repo. Configure here.

@aknight-ld
aknight-ld marked this pull request as ready for review September 18, 2026 18:58

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 6748eaa. Configure here.

Comment thread packages/client/src/launchdarkly_ai_server/evaluations/module.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant