Skip to content

feat: let the engine run host agent classes, and teach SchemaTools about Rails enums - #467

Merged
TonsOfFun merged 18 commits into
mainfrom
feat/engine-runs-code-agents
Sep 24, 2026
Merged

TonsOfFun merged 18 commits into
mainfrom
feat/engine-runs-code-agents

Conversation

@TonsOfFun

Copy link
Copy Markdown
Contributor

Four gaps a host hits when its agents live in Ruby and the dashboard mirrors them. Each was found by reading a real integration (support-hub) against the engine, and each is something the framework already has the information to do itself.

Based on release/1.6.3 (#457). No overlap with #460, which fixes the host-adapter / eval-catalog path (adapted-replay metering, ModelSpec validation, replace_scenarios! retention) — that host supplies a scenario_evaluation_adapter_resolver; this one supplies agent classes. Different seams, no shared files.

A Rails enum reached the model as a bare integer, and wrong values failed silently

SchemaTools builds filter properties from SchemaGenerator, which derives an enum constraint from inclusion validators. It never consulted model.defined_enums. So a status column backed by a Rails enum was offered as:

"status": {
  "anyOf": [
    { "type": "integer", "default": 0 },                       // no labels
    { "type": "object", "properties": { "gt": {…}, "lt": {…} } } // comparisons, on an enum
  ]
}

Two failure modes, both silent — measured against a host with 47 tickets, 2 open:

Call Before After
status: "pending" (not a real status) {count: 0} error naming draft, review, live
status: {gt: 1} (meaningless on an enum) {count: 27} rejected

{count: 0} is indistinguishable from "no tickets match", so an agent reports "nothing is under review" as fact and nothing errors anywhere.

The framework already had the right instinct one level up — an unknown column is rejected with `description` is not a filterable attribute. Allowed filters: … — and range_predicates! rejects an unknown operator with this reasoning stated in its own comment:

Rails silently turns where(col: { "before" => x }) into col = NULL, which matches nothing and reports zero rather than failing — the worst outcome for an agent, which reads it as a truthful empty answer.

This extends that same rule to enum values. Enums are now offered as {type: "string", enum: [names]} and excluded from the range form, so the prose mapping every host writes into its instructions ("status: open (0), in_progress (1)…") is no longer needed.

The engine could not run an agent that exists in code

AgentExecutionService builds an anonymous Class.new(ActiveAgent::Base) from the record's tools and instructions columns, overriding .name to impersonate the host class.

That is exactly right for a dashboard-authored agent — it is rows: a tool selection from AVAILABLE_TOOLS and instructions typed in the builder, with no Ruby class anywhere. That runtime is untouched here.

It is wrong for a mirrored agent. Its SchemaTools rosters, delegations and rendered instructions cannot be expressed as tools + instructions, so a host flattens its agents to sync them — and the dashboard then evaluates a lookalike rather than what production runs. One host's own docs record the consequence: "evals score the dashboard hub (flat roster), production runs the code hub (delegations)."

Both runtimes now coexist, chosen by whether the class resolves:

ActionAgent.run_host_agent_classes = true   # default: false

Default off, because it changes what a run of a mirrored agent executes. Agents naming no class are unaffected either way, and a name that no longer resolves — renamed, deleted, or not an agent at all — falls back to the dynamic runtime rather than taking the dashboard down.

AgentSync, so agent_class_name is the engine's job

AgentRelease already reads agent_class_name and documents that "whatever syncs its ActiveAgent classes into Agent records sets it" — but the engine never shipped that syncer, so each host writes ~90 generic lines. ActionAgent::AgentSync.call(RecordAgent.all, owner:) is that step, with the split that makes it safe on every deploy:

  • The code owns what an agent is — name, description, instructions, tools. Rewritten each sync, so it cannot drift.
  • The operator owns how it runs — provider, model, status. Set on create, never touched again, so a model picked in the dashboard survives the next deploy.

rendered_instructions

Two surfaces need an agent's instructions without running it: the dashboard mirror and tests. Both previously reached a private renderer via send. TicketAgent.rendered_instructions(topic: "tickets") is now public API — and gives release_digest a stronger thing to hash later (rendered text, not template file digests).

Testing

  • 9 new tests: 5 in schema_tools_test.rb (enum schema, no range form, undefined value rejected, defined values still accepted, range predicate rejected), 3 in rendered_instructions_test.rb, 4 in agent_sync_test.rb, 5 in host_agent_class_execution_test.rb covering both runtimes and each fallback.
  • One test caught a real bug pre-merge: namespaced classes produced billing/ticket-agent, which fails Agent's slug format. Now flattened to billing-ticket-agent.
  • Full suite 2040 runs, 0 failures. 253 errors are pre-existing missing-API-key errors in docs/integration tests — baseline on the same checkout before these changes is 261, so this removes 8 and adds none.
  • bin/lint: 555 files, no offenses.

Notes for review

  • The chars/4 token estimator is untouched here, but it is worth a separate look: measured against a real trace it runs ~31% low (2.75 chars/token actual on JSON schema + markdown), so the context meter is a relative breakdown rather than a budget.
  • provider_available? still ignores the provider ENV fallback, so a host with only OPENROUTER_API_KEY set sees the dashboard report the provider unconfigured while direct calls work. Related: openai is a development dependency but OpenRouterProvider requires it at runtime #416.

🤖 Generated with Claude Code

TonsOfFun and others added 7 commits September 17, 2026 20:36
…out enums

Four gaps a host hits when its agents live in code and the dashboard mirrors
them. Each is something the framework has the information to do itself, and
each was found by reading a real integration (support-hub) against the engine.

Rails enums reached the model as a bare integer. SchemaGenerator derives an
enum constraint from inclusion validators and never consulted defined_enums,
so a `status` column was offered as {type: "integer"} with no labels — and
every host explained the mapping in prose instead. Worse, both failure modes
were silent: `status: "pending"` returned {count: 0}, indistinguishable from
"none match", and the range form the schema advertised on an enum meant
`status: {gt: 1}` returned a confident, arbitrary count. range_predicates!
already rejects an unknown operator for exactly this reason; enum values now
get the same treatment.

The engine could not run an agent that exists in code. AgentExecutionService
builds an anonymous ActiveAgent::Base subclass from the record's `tools` and
`instructions` columns — right for a dashboard-authored agent, which is rows
and has no class, but a mirrored agent's SchemaTools rosters, delegations and
rendered instructions cannot be expressed that way. A host therefore flattens
its agents to sync them, and the dashboard evaluates a lookalike rather than
what production runs. Both runtimes now coexist: ActionAgent.run_host_agent_
classes (default false) runs the real class when one resolves, and anything
else — no class name, a stale name, a name that is not an agent — falls back
to the dynamic runtime rather than failing.

AgentSync mirrors classes into Agent records, setting the agent_class_name
AgentRelease already expects a host to have written, with the code owning what
an agent is and the operator owning how it runs. rendered_instructions makes
the text reachable without `send` into a private renderer.

Full suite: 2040 runs, 0 failures; 253 errors, all pre-existing missing-API-key
errors in docs/integration tests (baseline on the same checkout: 261). rubocop
clean across 555 files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A record agent's prompt comes from a template it shares with sibling agents,
filled from assigns the instance computes. rendered_instructions resolves only
an agent's own template with assigns the caller supplies, so syncing such an
agent stored empty instructions. AgentSync now asks for
dashboard_instructions_text when the class defines it — only the class knows
how its own prompt is built.

Found by refactoring support-hub onto this branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rails 7.2 raises "Undeclared attribute type for enum 'state' in EnumPost"
when a subclass declares an enum over a column it inherited; Rails 8 infers it
from the schema. Declaring `attribute :state, :integer` satisfies both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rails 7.2 does not resolve an inherited column through a subclass the way
Rails 8 does: EnumPost < Post raised "Undeclared attribute type for enum",
and once that was declared, SchemaTools' own resolve_column! rejected `state`
as "not a column on EnumPost". The column belongs to Post, so the enum is
declared there and the tools front Post directly — no subclass, and the same
behaviour on both Rails lines.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI builds the dummy database with db:migrate, not by loading schema.rb, so
adding `state` to the schema alone left Rails 7's test database without the
column — and Post's `enum :state` then raised "Undeclared attribute type for
enum" before any test ran. Rails 8 was green only because its schema was
already loaded locally.

Verified the way CI does it: rebuilt test/dummy from migrations under
gemfiles/rails7.gemfile and ran the suite — 2067 runs, 0 failures (253
pre-existing API-key errors), and the same on Rails 8.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The schema advertises enum names only, so an integer arriving here means the
model ignored it — worth stating that the tolerance is deliberate rather than
an oversight, since the neighbouring branches both reject.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@TonsOfFun
TonsOfFun changed the base branch from release/1.6.3 to main September 18, 2026 21:48
TonsOfFun and others added 11 commits September 19, 2026 12:15
SolidAgentRunsTest asserted claude-sonnet-5 at 0.048 for 12k/800 tokens.
That figure is solid_agent's static table ($3/$15 per million). But
ModelPricing prefers RubyLLM's registry whenever ::RubyLLM is defined, and
the registry shipped in ruby_llm 2.0.0 prices the same model at $2/$10 —
0.032. Which table answers depends on whether the RubyLLM provider tests
loaded the gem, and they do exactly when OPENAI_API_KEY is set: CI has no
key, so it skipped them and stayed green; a local .env.test with a
placeholder key loaded them and failed every full run, while the file
alone passed.

The old first assertion also compared ModelPricing.estimate with itself
through the record, so it would have passed on nil.

Now the test asserts what this seam actually guarantees, in any load
order: usage reaches the generation record; the mock model prices to 0.0
rather than nil; a real model prices above zero and the record agrees with
ModelPricing for the same tokens; and the explicit-rate branch, which
bypasses every table, still computes 0.048. The dollar figures behind the
tables belong to solid_agent's own suite.

Verified under gemfiles/rails7.gemfile with ruby_llm both unloaded and
loaded, and full suites on Rails 7 and 8: 2071 runs, 0 failures, with the
34 errors unchanged (missing API keys).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Ports the MetaStrip from #470 so the evaluation runs list can keep its
cost, movement and status columns aligned on every row — a failed run with
nothing to put in a column prints a dash there rather than sliding its
neighbours over. The trace and interaction lists take the same strip, as
that change has it; the source is identical to the branch so it no-ops once
#470 merges.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmHbHdDkfVrg3WX95CnFhd
Every client wants to know what operating an agent will cost, and an
evaluation run is the closest thing to a controlled measurement: a suite's
replays are simulated user–agent interactions, a sampling run's cohorts are
real ones. But the run also asks a judge model to score, recommend and rule,
and a run that reported that spend in the same number overstated the first.

The runners now meter every judge call under what it was for, through the
`kind:` keyword ActiveAgent::Evals::Judge hands a block that accepts it, and
persist the total as scores["_judge_usage"] — calls, tokens, estimated cost,
model and a by-kind tally. A generation-sampling run records
scores["_cohorts"] beside it: per model, how many generations were sampled,
how many cleared every criterion, their latency and tokens, and what those
interactions cost to serve. EvaluationRun#usage reports both sides, the
agent's with a per-interaction rate, and nil for a run that recorded neither.

The API numbers runs oldest-first (`number`), reports `run_count` and a
`previous_run` summary per evaluation, so the dashboard can say "Run #3" and
"+3 passed vs #2" without a request per evaluation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmHbHdDkfVrg3WX95CnFhd
Implements the "Evaluations Redesign v2" design (its REVIEW EVALS RUNS
section) in the engine, where the platform's earlier implementation was
lost when it swapped its own dashboard for this one. Evaluations are the
top level; every run is kept and listed with its movement against the run
before it; a sampling evaluation's run opens to a page of its own — a
scorecard per model cohort, the judge's verdict, the criteria × models
matrix and what the run asks to fix — at /evaluations/:id/runs/:run_id. A
scenario suite's runs are the same list, full width, and a row selects the
run its model scorecards, fix items and scenario matrix show; the deep link
selects it too.

What a run cost is shown as two figures everywhere it appears — on the run
row, on the run page, on a page tile and in the footer: the agent's spend
with its per-interaction rate, the operating figure a budget is set
against, apart from the judge's own offline calls.

The shared vocabulary (criterion labels and expectations, cohorts,
scorecards, movement, spend, the sampling run's fix items) lives in
utils/evaluationRuns.mjs, pinned by node tests. The report route, the
?evaluation= links and the e2e hooks (evaluation-card, data-telemetry,
score-source-telemetry) are kept.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmHbHdDkfVrg3WX95CnFhd
Adds framework/ruby_llm_dashboard, a guide for an application that already
calls models through RubyLLM and wants the telemetry dashboard without
adopting ActiveAgent::Base: the three topologies (same-app local_store,
separate dashboard app, hosted platform), the install and authentication
steps, naming traffic with with_agent / an agent_resolver, content capture,
verification, and the failure modes met while validating it (the
--traces_only 500, the Sass compressor, 2.x token counts before adapter
0.3.1). Links it from the self-hosted dashboard page and the RubyLLM
provider page, and adds it to the Framework sidebar.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D5GomP4QsAneCiVN6D78od
Rebuild Evaluations page around runs with detailed history and scoring
docs: add the Dashboard for RubyLLM Apps guide
…ues they don't define

SchemaGenerator derives an `enum` constraint from inclusion validators and
never consulted `defined_enums`, so a Rails-enum `status` column reached the
model as a bare integer with the range form attached. Two silent failures
followed, measured on a host with 47 tickets:

- `status: "pending"` (not a status) returned `{count: 0}`
- `status: {gt: 1}` returned `{count: 27}`

Neither errors, and `{count: 0}` reads to an agent as "none match".

An enum is now offered as `{type: "string", enum: [names]}` without the range
form, and a value it does not define is rejected naming the valid ones — the
same rule `range_predicates!` already applies to an unknown operator. Each
member of an IN list is checked too, since `where` silently drops an undefined
one there as well. The integer backing is still accepted for Ruby callers.

Cut from v1.6.3 so the patch carries only this fix. Extracted from #467,
whose engine changes stay in review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V9uHHSqFk1eqFjvozPQ4Nn
# Conflicts:
#	CHANGELOG.md
Conflicts were 1.6.4's own release of the SchemaTools enum fix; main's
version is kept, which also checks each member of an IN filter. #467's
unreleased additions (rendered_instructions, AgentSync,
run_host_agent_classes) go under Unreleased.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KfFzc97pARf86bj8LNR59B
Agent slugs are unique per owner, but AgentSync found records by slug
alone, so on a per-user or multi-tenant install a second owner's sync
took over the first owner's agent: rewrote its instructions and tools
and returned it as the second owner's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KfFzc97pARf86bj8LNR59B
@TonsOfFun
TonsOfFun merged commit 25e06bb into main Sep 24, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants