Pair a nominal and a retimed NMG arm in the benchmark

[Yuan-Xinyi]

Stack

• Stack: main ← Replace external TOPPRA with differentiable CPU/CUDA retiming #724 ← Measure dynamic limits and mark nominal timing in the NMG benchmark #713 ← Retime NMG rollouts under joint velocity and acceleration limits #714 ← Pair a nominal and a retimed NMG arm in the benchmark #715
• Base: xinyi/nmg-toppra-retiming
• Depends on: Retime NMG rollouts under joint velocity and acceleration limits #714 (and Measure dynamic limits and mark nominal timing in the NMG benchmark #713, Replace external TOPPRA with differentiable CPU/CUDA retiming #724 below it)
• Followed by: none

Description

#713 measures dynamic limits and #714 lets NMG retime its rollout, but nothing yet runs the same policy both ways over the same cases. This PR adds that arm, which is the thing that actually answers "does retiming help, and what does it cost".

What changed:

• retime option on the NMG adapter. It forwards NeuralPlannerCfg.constraints and reports native_timing=True, since the row now solves a duration instead of assuming one.
• The retiming limits come from the same sources the metrics check. Velocity from robot.get_qvel_limits(), acceleration from the suite protocol, which PlannerContext now carries. A row that retimed against limits of its own choosing would satisfy them trivially, so both must read from one place. Retiming with no protocol acceleration limit raises rather than falling back silently.
• nmg_retimed in the free-space suites, identical to nmg apart from retime. A test asserts that equality, so the pair cannot drift into measuring a configuration difference as well as a timing one.

The two rows share a path and differ only in timing, so the comparison reads straight off report.md: the nominal row publishes its peaks and utilizations with no validity verdict, the retimed row carries both plus a comparable duration.

Both rows ship enabled: false with onnx_model_path: null, matching the existing NMG row — no policy checkpoint is committed, so this changes no default run.

Dependencies: none beyond the layers below.

Refs #684

Type of change

• Enhancement (non-breaking change which improves an existing functionality)

Measured evidence

python -m scripts.benchmark.motion_generation.run_benchmark \
  --suite smoke --algorithms nmg nmg_retimed \
  --nmg-onnx-path <swa3.onnx> --device cuda --num-trials 3

Real NMG checkpoint (swa3.onnx, K=5) on Franka FR3, on this stack with #724's backend. Both rows plan the same 9 free-space cases at W=1/3/5 from the same seeds, differing only by retime.

arm	motion_valid	waypoints	v viol	a viol	j viol	v utilization	a utilization	duration	plan ms (per-case median)
nmg	100%	100%	100%	100%	100%	1.21 – 5.29	52 – 122	N/A	43 – 295
nmg_retimed	100%	100%	0%	0%	0%	0.20 – 0.61	0.998 – 1.000	0.57 – 1.73 s	79 – 424

Per case, acceleration limit utilization (1.0 = exactly at the limit):

case	W=1	W=1	W=1	W=3	W=3	W=3	W=5	W=5	W=5
nmg	72.3	81.1	122.3	69.2	97.7	97.2	52.4	85.1	61.3
nmg_retimed	0.9998	0.9992	0.9993	0.9981	0.9983	0.9977	0.9990	0.9996	0.9994

Two things a summary rate hides:

• The retimed acceleration is 0.998–1.000 on all nine cases, never over and not scattered below. A time-optimal parameterization rides its binding constraint, and Replace external TOPPRA with differentiable CPU/CUDA retiming #724's conservative interval bounds keep it just under the limit between grid points as well as at them.
• The nominal arm's velocity utilization falls with waypoint count (5.2 at W=1 down to 1.2–4.6 at W=5) while acceleration stays 52–122× throughout. Longer sequences let the policy take smaller steps, but step-to-step action reversal, which is what sets acceleration, does not improve.

So the pair measures duration and some planning latency traded for executability, not accuracy. Latency is noisy on this machine (the nominal arm alone ranges 43–295 ms per case); retiming adds roughly 30–130 ms per plan at B=1.

A defect the run found

The first attempt produced a retimed arm bounded only by acceleration: the simulator reports the float32 maximum for every joint velocity limit, so retiming was constraining a quantity it then reported as respected. Fixed in 9379987e — retiming resolves its limits through the same helper the metrics use and refuses to run when neither the asset nor the protocol supplies a usable one.

Not covered

Nine cases, one suite, one batch size (B=1, so the held-tail effect #726 fixes does not appear here), one checkpoint, one robot. Enough to show the pair runs and what it trades; not a leaderboard result.

Limit utilization for both arms over all nine measured cases, log scale, 1.0 = exactly at the limit. Every nominal case is outside both limits; every retimed case is under them, with acceleration at 0.998–1.000 throughout.

Validation

Proportional to the change, which is confined to the benchmark package and its tests:

• python -m pytest tests/benchmark/motion_generation/ tests/sim/motion/planners/test_neural_planner.py — 163 passed, 1 skipped.
• black . — clean.
• python docs/scripts/check_api_docs.py — 2315/2315. No public API changed.
• context.py affected reports motion-planning and simulation-system, but only through the lower layers' files; this layer's own diff is entirely under scripts/benchmark/, which no context topic registers. BENCHMARK_DESIGN.md section 3.3 is updated here to describe the paired arm and why its limits are shared.

Checklist

• I have run the black . command to format the code base.
• I reviewed affected documentation and agent context, updated it where needed, or explained why no update was needed.
• Public API changes are reflected in the API docs (python docs/scripts/check_api_docs.py), if applicable
• I have added tests that prove my fix is effective or that my feature works
• Dependencies have been updated, if applicable. No dependency changes were required.

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Adds a paired retimed variant to the motion generation benchmark.

The PR appears safe to merge; no new actionable issue or outstanding previous finding remains.

Findings

1.  Retimed arm cannot solve ▶

Fix with agent prompt

### Issue 1
scripts/benchmark/motion_generation/planners/nmg_onnx.py:undefined-117
When the NMG row runs with `retime: true`, this passes a one-dimensional list of velocity limits to TOPPRA. The shared adapter turns scalar limits into lower/upper-bound pairs but passes lists through unchanged. TOPPRA cannot construct the velocity constraint, so the retimed arm returns failed trajectories instead of producing the paired comparison.

```suggestion
            "velocity": [
                [-limit, limit] for limit in velocity.detach().cpu().tolist()
            ],
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR adds a retimed NMG benchmark row alongside the nominal row, passes suite motion limits to the adapter, and applies CLI checkpoint overrides to both rows.

• The checkpoint-override and jerk-documentation findings are fixed.
• The previously reported velocity-limit shape is accepted by the current retiming implementation.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Suite[Free-space suite] --> Nominal[nmg: nominal timing]
  Suite --> Retimed[nmg_retimed: solved timing]
  Limits[Asset velocity and protocol acceleration limits] --> Retimed
  Nominal --> Report[Benchmark report]
  Retimed --> Report

Loading

_{Reviews (7) · Last reviewed commit: "fix(bench): refuse to retime against a p..."}

[Yuan-Xinyi]

Both findings were valid; fixed in ce76faa7.

1. CLI overrides reached only the nmg row — fixed, and this one defeated the PR. --nmg-onnx-path selected spec.id == "nmg", so nmg_retimed kept a null checkpoint, failed its availability check, and was skipped — the paired comparison this change exists to run would never have run, silently. The overrides now apply to every nmg_onnx row. --nmg-pos-eps and --nmg-rot-eps had the same problem, and leaving those to diverge would have made the pair differ in tolerance as well as timing. Covered by test_nmg_cli_overrides_reach_every_nmg_row.

2. Jerk is validated but not retimed against — fixed in the docs. Correct: TOPPRA constrains velocity and acceleration, while the suites also state a jerk limit that the metrics check. BENCHMARK_DESIGN now says so explicitly, and frames a jerk failure as a real result about the policy's path — which retiming cannot smooth, only stretch in time — rather than a gap in the velocity and acceleration guarantee.

[greptile-apps]

Retimed arm cannot solve

When the NMG row runs with retime: true, this passes a one-dimensional list of velocity limits to TOPPRA. The shared adapter turns scalar limits into lower/upper-bound pairs but passes lists through unchanged. TOPPRA cannot construct the velocity constraint, so the retimed arm returns failed trajectories instead of producing the paired comparison.

Suggested change

	"velocity": velocity.detach().cpu().tolist(),
	"velocity": [
	[-limit, limit] for limit in velocity.detach().cpu().tolist()
	],

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/motion_generation/planners/nmg_onnx.py
Line: 117

Comment:
**Retimed arm cannot solve**

When the NMG row runs with `retime: true`, this passes a one-dimensional list of velocity limits to TOPPRA. The shared adapter turns scalar limits into lower/upper-bound pairs but passes lists through unchanged. TOPPRA cannot construct the velocity constraint, so the retimed arm returns failed trajectories instead of producing the paired comparison.

```suggestion
            "velocity": [
                [-limit, limit] for limit in velocity.detach().cpu().tolist()
            ],
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[yangchen73]

The constrains of each env should be the same, right?