feat: let the engine run host agent classes, and teach SchemaTools about Rails enums - #467
Merged
Merged
Conversation
…out enums
Four gaps a host hits when its agents live in code and the dashboard mirrors
them. Each is something the framework has the information to do itself, and
each was found by reading a real integration (support-hub) against the engine.
Rails enums reached the model as a bare integer. SchemaGenerator derives an
enum constraint from inclusion validators and never consulted defined_enums,
so a `status` column was offered as {type: "integer"} with no labels — and
every host explained the mapping in prose instead. Worse, both failure modes
were silent: `status: "pending"` returned {count: 0}, indistinguishable from
"none match", and the range form the schema advertised on an enum meant
`status: {gt: 1}` returned a confident, arbitrary count. range_predicates!
already rejects an unknown operator for exactly this reason; enum values now
get the same treatment.
The engine could not run an agent that exists in code. AgentExecutionService
builds an anonymous ActiveAgent::Base subclass from the record's `tools` and
`instructions` columns — right for a dashboard-authored agent, which is rows
and has no class, but a mirrored agent's SchemaTools rosters, delegations and
rendered instructions cannot be expressed that way. A host therefore flattens
its agents to sync them, and the dashboard evaluates a lookalike rather than
what production runs. Both runtimes now coexist: ActionAgent.run_host_agent_
classes (default false) runs the real class when one resolves, and anything
else — no class name, a stale name, a name that is not an agent — falls back
to the dynamic runtime rather than failing.
AgentSync mirrors classes into Agent records, setting the agent_class_name
AgentRelease already expects a host to have written, with the code owning what
an agent is and the operator owning how it runs. rendered_instructions makes
the text reachable without `send` into a private renderer.
Full suite: 2040 runs, 0 failures; 253 errors, all pre-existing missing-API-key
errors in docs/integration tests (baseline on the same checkout: 261). rubocop
clean across 555 files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A record agent's prompt comes from a template it shares with sibling agents, filled from assigns the instance computes. rendered_instructions resolves only an agent's own template with assigns the caller supplies, so syncing such an agent stored empty instructions. AgentSync now asks for dashboard_instructions_text when the class defines it — only the class knows how its own prompt is built. Found by refactoring support-hub onto this branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rails 7.2 raises "Undeclared attribute type for enum 'state' in EnumPost" when a subclass declares an enum over a column it inherited; Rails 8 infers it from the schema. Declaring `attribute :state, :integer` satisfies both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…-agents # Conflicts: # CHANGELOG.md
Rails 7.2 does not resolve an inherited column through a subclass the way Rails 8 does: EnumPost < Post raised "Undeclared attribute type for enum", and once that was declared, SchemaTools' own resolve_column! rejected `state` as "not a column on EnumPost". The column belongs to Post, so the enum is declared there and the tools front Post directly — no subclass, and the same behaviour on both Rails lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI builds the dummy database with db:migrate, not by loading schema.rb, so adding `state` to the schema alone left Rails 7's test database without the column — and Post's `enum :state` then raised "Undeclared attribute type for enum" before any test ran. Rails 8 was green only because its schema was already loaded locally. Verified the way CI does it: rebuilt test/dummy from migrations under gemfiles/rails7.gemfile and ran the suite — 2067 runs, 0 failures (253 pre-existing API-key errors), and the same on Rails 8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The schema advertises enum names only, so an integer arriving here means the model ignored it — worth stating that the tolerance is deliberate rather than an oversight, since the neighbouring branches both reject. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SolidAgentRunsTest asserted claude-sonnet-5 at 0.048 for 12k/800 tokens. That figure is solid_agent's static table ($3/$15 per million). But ModelPricing prefers RubyLLM's registry whenever ::RubyLLM is defined, and the registry shipped in ruby_llm 2.0.0 prices the same model at $2/$10 — 0.032. Which table answers depends on whether the RubyLLM provider tests loaded the gem, and they do exactly when OPENAI_API_KEY is set: CI has no key, so it skipped them and stayed green; a local .env.test with a placeholder key loaded them and failed every full run, while the file alone passed. The old first assertion also compared ModelPricing.estimate with itself through the record, so it would have passed on nil. Now the test asserts what this seam actually guarantees, in any load order: usage reaches the generation record; the mock model prices to 0.0 rather than nil; a real model prices above zero and the record agrees with ModelPricing for the same tokens; and the explicit-rate branch, which bypasses every table, still computes 0.048. The dollar figures behind the tables belong to solid_agent's own suite. Verified under gemfiles/rails7.gemfile with ruby_llm both unloaded and loaded, and full suites on Rails 7 and 8: 2071 runs, 0 failures, with the 34 errors unchanged (missing API keys). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Ports the MetaStrip from #470 so the evaluation runs list can keep its cost, movement and status columns aligned on every row — a failed run with nothing to put in a column prints a dash there rather than sliding its neighbours over. The trace and interaction lists take the same strip, as that change has it; the source is identical to the branch so it no-ops once #470 merges. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmHbHdDkfVrg3WX95CnFhd
Every client wants to know what operating an agent will cost, and an evaluation run is the closest thing to a controlled measurement: a suite's replays are simulated user–agent interactions, a sampling run's cohorts are real ones. But the run also asks a judge model to score, recommend and rule, and a run that reported that spend in the same number overstated the first. The runners now meter every judge call under what it was for, through the `kind:` keyword ActiveAgent::Evals::Judge hands a block that accepts it, and persist the total as scores["_judge_usage"] — calls, tokens, estimated cost, model and a by-kind tally. A generation-sampling run records scores["_cohorts"] beside it: per model, how many generations were sampled, how many cleared every criterion, their latency and tokens, and what those interactions cost to serve. EvaluationRun#usage reports both sides, the agent's with a per-interaction rate, and nil for a run that recorded neither. The API numbers runs oldest-first (`number`), reports `run_count` and a `previous_run` summary per evaluation, so the dashboard can say "Run #3" and "+3 passed vs #2" without a request per evaluation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmHbHdDkfVrg3WX95CnFhd
Implements the "Evaluations Redesign v2" design (its REVIEW EVALS RUNS section) in the engine, where the platform's earlier implementation was lost when it swapped its own dashboard for this one. Evaluations are the top level; every run is kept and listed with its movement against the run before it; a sampling evaluation's run opens to a page of its own — a scorecard per model cohort, the judge's verdict, the criteria × models matrix and what the run asks to fix — at /evaluations/:id/runs/:run_id. A scenario suite's runs are the same list, full width, and a row selects the run its model scorecards, fix items and scenario matrix show; the deep link selects it too. What a run cost is shown as two figures everywhere it appears — on the run row, on the run page, on a page tile and in the footer: the agent's spend with its per-interaction rate, the operating figure a budget is set against, apart from the judge's own offline calls. The shared vocabulary (criterion labels and expectations, cohorts, scorecards, movement, spend, the sampling run's fix items) lives in utils/evaluationRuns.mjs, pinned by node tests. The report route, the ?evaluation= links and the e2e hooks (evaluation-card, data-telemetry, score-source-telemetry) are kept. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmHbHdDkfVrg3WX95CnFhd
Adds framework/ruby_llm_dashboard, a guide for an application that already calls models through RubyLLM and wants the telemetry dashboard without adopting ActiveAgent::Base: the three topologies (same-app local_store, separate dashboard app, hosted platform), the install and authentication steps, naming traffic with with_agent / an agent_resolver, content capture, verification, and the failure modes met while validating it (the --traces_only 500, the Sass compressor, 2.x token counts before adapter 0.3.1). Links it from the self-hosted dashboard page and the RubyLLM provider page, and adds it to the Framework sidebar. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D5GomP4QsAneCiVN6D78od
Rebuild Evaluations page around runs with detailed history and scoring
docs: add the Dashboard for RubyLLM Apps guide
…ues they don't define
SchemaGenerator derives an `enum` constraint from inclusion validators and
never consulted `defined_enums`, so a Rails-enum `status` column reached the
model as a bare integer with the range form attached. Two silent failures
followed, measured on a host with 47 tickets:
- `status: "pending"` (not a status) returned `{count: 0}`
- `status: {gt: 1}` returned `{count: 27}`
Neither errors, and `{count: 0}` reads to an agent as "none match".
An enum is now offered as `{type: "string", enum: [names]}` without the range
form, and a value it does not define is rejected naming the valid ones — the
same rule `range_predicates!` already applies to an unknown operator. Each
member of an IN list is checked too, since `where` silently drops an undefined
one there as well. The integer backing is still accepted for Ruby callers.
Cut from v1.6.3 so the patch carries only this fix. Extracted from #467,
whose engine changes stay in review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V9uHHSqFk1eqFjvozPQ4Nn
# Conflicts: # CHANGELOG.md
Conflicts were 1.6.4's own release of the SchemaTools enum fix; main's version is kept, which also checks each member of an IN filter. #467's unreleased additions (rendered_instructions, AgentSync, run_host_agent_classes) go under Unreleased. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KfFzc97pARf86bj8LNR59B
Agent slugs are unique per owner, but AgentSync found records by slug alone, so on a per-user or multi-tenant install a second owner's sync took over the first owner's agent: rewrote its instructions and tools and returned it as the second owner's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KfFzc97pARf86bj8LNR59B
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four gaps a host hits when its agents live in Ruby and the dashboard mirrors them. Each was found by reading a real integration (support-hub) against the engine, and each is something the framework already has the information to do itself.
Based on
release/1.6.3(#457). No overlap with #460, which fixes the host-adapter / eval-catalog path (adapted-replay metering,ModelSpecvalidation,replace_scenarios!retention) — that host supplies ascenario_evaluation_adapter_resolver; this one supplies agent classes. Different seams, no shared files.A Rails enum reached the model as a bare integer, and wrong values failed silently
SchemaToolsbuilds filter properties fromSchemaGenerator, which derives anenumconstraint from inclusion validators. It never consultedmodel.defined_enums. So astatuscolumn backed by a Rails enum was offered as:Two failure modes, both silent — measured against a host with 47 tickets, 2 open:
status: "pending"(not a real status){count: 0}draft, review, livestatus: {gt: 1}(meaningless on an enum){count: 27}{count: 0}is indistinguishable from "no tickets match", so an agent reports "nothing is under review" as fact and nothing errors anywhere.The framework already had the right instinct one level up — an unknown column is rejected with
`description` is not a filterable attribute. Allowed filters: …— andrange_predicates!rejects an unknown operator with this reasoning stated in its own comment:This extends that same rule to enum values. Enums are now offered as
{type: "string", enum: [names]}and excluded from the range form, so the prose mapping every host writes into its instructions ("status: open (0), in_progress (1)…") is no longer needed.The engine could not run an agent that exists in code
AgentExecutionServicebuilds an anonymousClass.new(ActiveAgent::Base)from the record'stoolsandinstructionscolumns, overriding.nameto impersonate the host class.That is exactly right for a dashboard-authored agent — it is rows: a tool selection from
AVAILABLE_TOOLSand instructions typed in the builder, with no Ruby class anywhere. That runtime is untouched here.It is wrong for a mirrored agent. Its
SchemaToolsrosters, delegations and rendered instructions cannot be expressed astools+instructions, so a host flattens its agents to sync them — and the dashboard then evaluates a lookalike rather than what production runs. One host's own docs record the consequence: "evals score the dashboard hub (flat roster), production runs the code hub (delegations)."Both runtimes now coexist, chosen by whether the class resolves:
Default off, because it changes what a run of a mirrored agent executes. Agents naming no class are unaffected either way, and a name that no longer resolves — renamed, deleted, or not an agent at all — falls back to the dynamic runtime rather than taking the dashboard down.
AgentSync, soagent_class_nameis the engine's jobAgentReleasealready readsagent_class_nameand documents that "whatever syncs its ActiveAgent classes into Agent records sets it" — but the engine never shipped that syncer, so each host writes ~90 generic lines.ActionAgent::AgentSync.call(RecordAgent.all, owner:)is that step, with the split that makes it safe on every deploy:rendered_instructionsTwo surfaces need an agent's instructions without running it: the dashboard mirror and tests. Both previously reached a private renderer via
send.TicketAgent.rendered_instructions(topic: "tickets")is now public API — and givesrelease_digesta stronger thing to hash later (rendered text, not template file digests).Testing
schema_tools_test.rb(enum schema, no range form, undefined value rejected, defined values still accepted, range predicate rejected), 3 inrendered_instructions_test.rb, 4 inagent_sync_test.rb, 5 inhost_agent_class_execution_test.rbcovering both runtimes and each fallback.billing/ticket-agent, which failsAgent's slug format. Now flattened tobilling-ticket-agent.bin/lint: 555 files, no offenses.Notes for review
chars/4token estimator is untouched here, but it is worth a separate look: measured against a real trace it runs ~31% low (2.75 chars/token actual on JSON schema + markdown), so the context meter is a relative breakdown rather than a budget.provider_available?still ignores the provider ENV fallback, so a host with onlyOPENROUTER_API_KEYset sees the dashboard report the provider unconfigured while direct calls work. Related: openai is a development dependency but OpenRouterProvider requires it at runtime #416.🤖 Generated with Claude Code