fix(evaluations): reject boolean scorer scores

[donei003]

Follow-up to #63, now merged. Brings the scorer return contract in line with ai-sdks-monorepo TESTING.md §8.8.4, which was tightened after #63 was written.

Problem

A scorer's contract was "a bool or a finite number in 0–1", with True/False coerced to 1.0/0.0. Two problems with the coercion:

It put two score types on the wire. A threshold then meant different things depending on which type a scorer happened to return, even though LaunchDarkly rules on score-vs-threshold identically for both — so the comparison a caller reasoned about and the one ingest performed could differ.

It hid bugs, and the failure mode was a green run. Every non-empty value is truthy, so a scorer that returned "high", or a stray object, scored 1.0 and passed:

# Before: scored 1.0, row passed, no error anywhere.
Scorer(name="quality", fn=lambda row, output: grade(output))  # grade() returns "high"

A false pass is the one failure mode an evaluation harness must not have — the point of the gate is that a broken check fails loudly.

Change

The contract is now a finite number in 0–1, and nothing else. A scorer answering a yes/no question returns 1.0 or 0.0 itself:

def mentions_policy(row: DatasetRow, output: str | None) -> float:
    return 1.0 if "refund policy" in (output or "").lower() else 0.0

A bool, a non-finite number, or one out of range is an invalid_score result — a per-criterion ERROR event, never a raise, since the row's generation has already been paid for.

numeric_score() already excluded bool (Python's bool is an int subclass, which is exactly the trap), so the fix deletes the coercion branch rather than adding a check — the two error paths collapse into one whose message names the offending value.

ScorerFn drops bool from its signature, so a caller who annotates their scorer sees this at type-check time rather than at ingest.

⚠️ Breaking

A scorer returning a bool now produces an invalid_score result instead of 1.0/0.0. Callers return the number directly. This is deliberate per §8.8.4 ("there is no boolean shortcut and nothing is coerced on the caller's behalf") — but it is a behavior change and worth a release note.

Every scorer in the examples repo is converted to match in launchdarkly-labs/ai-sdk-evaluations-example#4.

Validation

• uv run pytest -q — 1264 passed, 11 skipped
• uv run mypy .../evaluations — clean; ruff check / ruff format --check — clean
• New parametrized coverage: True, False, NaN, Infinity, 3, -1, "high", None all produce invalid_score with the value named and no score field, and the run still completes, flushes, and polls
• Boundary coverage: 0, 1, 0.0, 1.0, 0.25 stay valid, ints included — only bool is excluded
• The suite's own scorers were converted from bool to numeric, which is the change callers make

🤖 Generated with Claude Code