diagnostic_bridge: keep the numbers behind a fault, and stop confirming every planned STALE

docs/api/rest.rst

@@ -1514,7 +1514,19 @@ Query and manage faults.
 
    **Snapshot Types:**
 
-   - ``freeze_frame``: Data captured at fault confirmation. Entity frames for
+   - ``freeze_frame``: Data captured at fault confirmation: one entry per
+     captured topic, plus one entry named ``freeze_frame`` whose ``data`` maps
+     each topic to its value. That entry can also hold **reporter evidence**
+     under ``"x-medkit": {"reported": {"<source_id>": {"<key>": {"value":
+     "37", "reported_at": "<ISO-8601>"}}}}``: the key-value pairs reporters
+     sent with the FAILED reports of the current occurrence (for example the
+     ``DiagnosticStatus`` values from the diagnostic bridge), as string values,
+     last value per key. Topic keys start with ``/``, so they cannot clash with
+     it. Evidence is written on every FAILED report, so a fault with no capture
+     config can still have a ``freeze_frame`` entry with only evidence in it.
+     When per-topic entries are listed, the ``freeze_frame`` entry holds only
+     the evidence, since the topic values are already in those entries. Its
+     ``captured_at`` is then the time of the newest evidence. Entity frames for
      faults that were already confirmed when the gateway started are captured
      at gateway start instead and carry ``"capture_origin": "startup"`` in
      their ``x-medkit`` block. An entity frame also carries ``source`` in

docs/config/diagnostic-bridge.rst

@@ -37,7 +37,30 @@ The diagnostic bridge:
      - SEVERITY_ERROR (2)
    * - STALE
      - FAILED
-     - SEVERITY_ERROR (2)
+     - ``stale_severity``, SEVERITY_CRITICAL (3) by default
+
+.. warning::
+
+   STALE reports SEVERITY_CRITICAL by default, and CRITICAL **bypasses debounce** in the fault
+   manager, so a STALE status confirms a CRITICAL fault on its first sample.
+
+   That matters because STALE is the one level a node can reach by design: a GPS goes STALE
+   in every tunnel, an IMU reports covariance -1 while it settles. Left unconfigured, every
+   such outage is a confirmed CRITICAL fault. Use ``stale_severity`` to set the level, or
+   ``stale_severity_overrides`` to name just the diagnostic names that go STALE on purpose;
+   those statuses then debounce like any other.
+
+.. important::
+
+   A non-CRITICAL STALE is only debounced if the fault manager is set up for it. With the
+   default ``confirmation_threshold: -1``, the first FAILED report confirms the fault. The
+   bridge's local filter only waits for 3 reports within 10 s. So with both defaults, a
+   planned STALE still becomes a confirmed fault after about three cycles. Set
+   ``confirmation_threshold`` to ``-3`` or lower to debounce it.
+
+   This threshold is picked by ``source_id``. All statuses from the bridge share one
+   ``source_id`` (``/diagnostic_bridge``, unless ``use_hardware_id_as_source_id`` is on), so
+   a threshold for the bridge applies to **every** diagnostic it forwards, not only STALE ones.
 
 Parameters
 ----------
@@ -49,6 +72,9 @@ Parameters
        diagnostics_topic: "/diagnostics"   # Topic to subscribe to
        auto_generate_codes: true           # Auto-generate fault codes from names
        keyvalue_codes: ["fault_code"]      # Take the code from these key-value keys
+       stale_severity: "WARN"              # What a STALE status reports at
+       # Per diagnostic name, longest matching prefix wins:
+       "stale_severity_overrides.gps": "WARN"
 
 .. list-table::
    :header-rows: 1
@@ -71,6 +97,67 @@ Parameters
        as its value, so a publisher can name its own code instead of relying on
        a mapping. Checked after ``name_to_code`` and before auto-generation.
        Empty strings in the list are ignored.
+   * - ``stale_severity``
+     - ``CRITICAL``
+     - Severity a STALE status reports at, one of ``INFO``, ``WARN``, ``ERROR``,
+       ``CRITICAL`` (case-insensitive). Applies to STALE only: the other levels are
+       facts about the status, not deployment decisions. A name that does not parse
+       is reported and ``CRITICAL`` is used.
+   * - ``stale_severity_overrides.<name>``
+     - ``-``
+     - Severity for STALE statuses whose name starts with ``<name>``. Diagnostic names
+       are conventionally ``<component>: <check>``, so a component prefix covers every
+       check it publishes. The **longest matching prefix** wins. An override that does
+       not parse is reported and **ignored**, leaving ``stale_severity`` in force -
+       applying ``CRITICAL`` to a typo would restore the immediate-confirm behaviour the
+       operator was configuring their way out of.
+
+Evidence
+--------
+
+A FAILED report carries the ``DiagnosticStatus`` key-values to the fault manager, which keeps
+them per occurrence of the fault and serves them in the fault's freeze frame from
+``GET /api/v1/apps/{app}/faults/{code}``. The fault is reported under the bridge's
+``source_id``, so with the bridge defaults the app is ``diagnostic_bridge``:
+
+.. code-block:: console
+
+   $ curl -s localhost:8080/api/v1/apps/diagnostic_bridge/faults/FUSION_DIVERGED | jq '.environment_data.snapshots[] | select(.type == "freeze_frame" and .name == "freeze_frame") | .data["x-medkit"].reported'
+   {
+     "/diagnostic_bridge": {
+       "nis": { "value": "0.03", "reported_at": "2026-10-19T07:54:03.120Z" },
+       "rejected_fixes": { "value": "37", "reported_at": "2026-10-19T07:54:03.120Z" }
+     }
+   }
+
+The evidence is grouped by ``source_id``, then by key. Each key has its value and the time of
+the last report that set it. If topics were also captured for the code, the list has one entry
+per topic before the freeze frame. Those entries also have type ``freeze_frame``, so the filter
+checks the name too. The freeze frame then holds only the evidence, because the topic values
+are already in those entries.
+
+Evidence is written on **every** FAILED report, not only the one that confirms the fault, so a
+code that never confirms still has evidence. Only the **last** value of each key is kept, and
+near-miss entries do not store evidence. So a code with several near misses shows only the
+numbers from its latest report.
+
+A new occurrence of the fault (a FAILED report after it was cleared) starts with no evidence.
+With ``snapshots.retain_on_clear`` on, the fault manager keeps the evidence of older
+occurrences too, up to ``snapshots.max_evidence_occurrences``, but serves only the current one.
+
+The frame's other keys are topic names sampled by the fault manager, always fully qualified
+and so always starting with ``/``. ``x-medkit`` cannot collide with one, which is what lets a
+reader tell a value the reporter asserted from one the fault manager sampled.
+
+.. note::
+
+   Evidence limits per occurrence of a fault code: at most 32 keys over all sources together,
+   128 bytes per key and 512 bytes per value (bytes, not characters). There is no limit on
+   total size. Bytes that are not valid UTF-8 are stored as U+FFFD, and the limits are checked
+   after that. An entry over a limit is dropped, not cut short, and a warning is logged. If
+   the dropped entry updates a key already stored, that key is removed, so an old value is not
+   shown as current. The fault is recorded either way. A key already stored can always be
+   updated, so a steady reporter at the bound can still refresh its own numbers.
 
 Custom Fault Code Mappings
 --------------------------

docs/config/fault-manager.rst

@@ -331,6 +331,7 @@ Basic Snapshot Settings
          config_file: ""                   # Path to YAML config file
          recapture_cooldown_sec: 60.0      # Min seconds between snapshot captures per fault
          max_per_fault: 10                 # Max snapshots stored per fault code (0 = unlimited)
+         max_evidence_occurrences: 10      # With retain_on_clear, occurrences whose evidence is kept
          capture_pool_size: 2              # Max concurrent capture threads (>= 1)
          capture_queue_depth: 16           # Max pending captures before policy applies (>= 1)
          capture_queue_full_policy: reject_newest  # reject_newest | drop_oldest
@@ -382,7 +383,16 @@ Basic Snapshot Settings
        historical behaviour: clearing a fault deletes them. Turn it on together
        with ``rosbag.max_bags_per_fault``, or acknowledging leaves the fault
        holding recordings whose matching readings are gone. Independent of
-       ``max_per_fault``, which still bounds growth either way.
+       ``max_per_fault``, which still bounds growth either way. Also keeps the
+       reporter evidence of older occurrences of a fault (see
+       ``max_evidence_occurrences``); without it a new occurrence deletes the
+       evidence of the older ones.
+   * - ``snapshots.max_evidence_occurrences``
+     - ``10``
+     - With ``retain_on_clear`` on, how many occurrences per fault code keep
+       their reporter evidence. The oldest are dropped first. ``GetFault`` serves
+       only the current occurrence. Set to 0 for unlimited. Has no effect
+       without ``retain_on_clear``.
    * - ``snapshots.capture_pool_size``
      - ``2``
      - Max concurrent capture threads under a fault storm (>= 1). The capture pool is

src/ros2_medkit_diagnostic_bridge/CMakeLists.txt

@@ -117,6 +117,8 @@ if(BUILD_TESTING)
   # repo convention) so it runs only in the integration phase.
   medkit_add_launch_test(test_integration test/test_integration.test.py TIMEOUT 60
     LABELS "integration")
+  medkit_add_launch_test(test_stale_and_evidence test/test_stale_and_evidence.test.py TIMEOUT 90
+    LABELS "integration")
   ros2_medkit_relax_vendor_warnings()
 endif()
 

src/ros2_medkit_diagnostic_bridge/README.md

@@ -15,7 +15,72 @@ to the FaultManager as faults.
 | OK (0) | - | Sends PASSED event (healing) |
 | WARN (1) | WARN (1) | Reports fault |
 | ERROR (2) | ERROR (2) | Reports fault |
-| STALE (3) | CRITICAL (3) | Reports fault |
+| STALE (3) | `stale_severity` (CRITICAL by default) | Reports fault |
+
+STALE is the one level whose severity is a deployment decision rather than a fact. A GPS in
+every tunnel and a dead sensor both publish STALE, and CRITICAL
+[bypasses debounce](../ros2_medkit_fault_manager/README.md), so an expected outage confirms a
+CRITICAL fault on its first sample. Set `stale_severity`, or name the noisy diagnostic names
+with `stale_severity_overrides`, and those statuses go through debounce like any other.
+
+This only helps if the fault manager debounces. With the default `confirmation_threshold: -1`
+the first FAILED report confirms, and the bridge's local filter only waits for 3 reports in
+10 s, so a planned STALE still confirms after about three cycles. Set `confirmation_threshold`
+to `-3` or lower. The threshold is picked by `source_id`, and all statuses from the bridge
+share one (`/diagnostic_bridge`, unless `use_hardware_id_as_source_id` is on), so it applies
+to every diagnostic the bridge forwards.
+
+## Evidence
+
+A FAILED report carries the status's key-values to the fault manager, which keeps them per
+occurrence of the fault and serves them in the fault's freeze frame under `x-medkit.reported`,
+from `GET /api/v1/apps/{app}/faults/{code}` (with the bridge defaults the app is
+`diagnostic_bridge`):
+
+```jsonc
+{
+  "environment_data": {
+    "snapshots": [
+      // ...one entry per captured topic, if the code has a capture configured...
+      {
+        "type": "freeze_frame",
+        "name": "freeze_frame",
+        "data": {
+          "x-medkit": {
+            "reported": {
+              "/diagnostic_bridge": {
+                "nis": { "value": "0.03", "reported_at": "2026-10-08T09:15:02.123Z" },
+                "rejected_fixes": { "value": "37", "reported_at": "2026-10-08T09:15:02.123Z" }
+              }
+            }
+          }
+        },
+        "x-medkit": {
+          "topic": "",
+          "message_type": "",
+          "full_data": { "x-medkit": { "reported": { "...": "same as data" } } },
+          "captured_at": "2026-10-08T09:15:02.123Z"
+        }
+      }
+    ]
+  }
+}
+```
+
+The evidence is grouped by `source_id`, then by key, and each key has the time of the last
+report that set it. When the code has captured topics, the frame holds only the evidence,
+because the topic values are already in the per-topic entries. The frame's other keys are
+topic names sampled by the fault manager, which are always fully qualified and start with
+`/`, so a reader can always tell a value the reporter asserted from one the fault manager
+sampled.
+
+`keyvalue_codes` still reads the same values to pick a fault code; the two uses are
+independent and either can be used without the other. Only the last value of each key is
+kept, and near-miss entries do not store evidence. A new occurrence (a FAILED report after a
+clear) starts with no evidence. Limits per occurrence: 32 keys over all sources, 128 bytes per
+key, 512 bytes per value, no limit on total size. An entry over a limit is dropped with a
+warning, not cut short, and the fault is recorded either way. If it updates a stored key, that
+key is removed. Bytes that are not valid UTF-8 are stored as U+FFFD.
 
 ## Quick Start
 
@@ -39,6 +104,8 @@ ros2 run ros2_medkit_diagnostic_bridge diagnostic_bridge_node
 | `max_tracked_sources` | integer | `512` | Maximum number of per-source FaultReporter instances retained by the bridge |
 | `name_to_code.<name>` | string | - | Custom mapping from diagnostic name to fault code |
 | `keyvalue_codes` | string[] | - | List of keys used to search the diagnostic values for the fault code |
+| `stale_severity` | string | `CRITICAL` | Severity a STALE status reports at. One of `INFO`, `WARN`, `ERROR`, `CRITICAL` |
+| `stale_severity_overrides.<name>` | string | - | Severity for STALE statuses whose name starts with `<name>`. Longest matching prefix wins |
 
 ### Example Configuration
 

src/ros2_medkit_diagnostic_bridge/config/diagnostic_bridge.yaml

@@ -14,6 +14,19 @@ diagnostic_bridge:
     # Maximum number of per-source FaultReporter instances retained by the bridge.
     max_tracked_sources: 512
 
+    # Severity a STALE status reports at. STALE is the one level a node can reach by
+    # design - a GPS in every tunnel, an IMU reporting covariance -1 - and CRITICAL
+    # bypasses debounce, so leaving it CRITICAL confirms a fault on the first sample.
+    # One of INFO, WARN, ERROR, CRITICAL.
+    stale_severity: "CRITICAL"
+
+    # Per-diagnostic STALE severity. Matches on the start of the diagnostic name, so a
+    # component prefix covers every check it publishes. Longest matching prefix wins.
+    # Format: "stale_severity_overrides.<diagnostic name or prefix>": "<SEVERITY>"
+    # Example:
+    # "stale_severity_overrides.gps": "WARN"
+    # "stale_severity_overrides.gps: antenna": "ERROR"
+
     # Custom diagnostic name to fault code mappings
     # Format: "name_to_code.<diagnostic_name>": "<FAULT_CODE>"
     # Example:

src/ros2_medkit_diagnostic_bridge/design/index.rst

@@ -48,12 +48,14 @@ The following diagram shows the data flow and component relationships.
            +name: string
            +message: string
            +hardware_id: string
+           +values: KeyValue[]
        }
    }
 
    package "ros2_medkit_fault_reporter" {
        class FaultReporter {
            +report(fault_code, severity, description)
+           +report(fault_code, severity, description, evidence)
            +report_passed(fault_code)
        }
    }
@@ -62,8 +64,10 @@ The following diagram shows the data flow and component relationships.
 
        class DiagnosticBridgeNode {
            +map_to_fault_code(status): string
-           +map_to_severity(level): optional<uint8>
-           +is_ok_level(level): bool
+           +map_to_severity(level, name): optional<uint8>
+           +stale_severity_for(name): uint8
+           +{static} parse_severity_name(name): optional<uint8>
+           +{static} is_ok_level(level): bool
            --
            -diagnostics_callback(msg)
            -process_diagnostic(status)
@@ -102,9 +106,10 @@ Data Flow
        Reporter -> FM : ReportFault(PASSED)
        note right: Triggers healing
    else level == WARN/ERROR/STALE
-       Bridge -> Reporter : report(fault_code, severity, msg)
-       Reporter -> FM : ReportFault(FAILED)
-       note right: Creates/updates fault
+       Bridge -> Reporter : report(fault_code, severity, msg, values)
+       Reporter -> FM : ReportFault(FAILED, evidence)
+       note right: Creates/updates fault,
+keeps evidence in freeze frame
    end
 
    @enduml
@@ -131,8 +136,9 @@ The bridge maps DiagnosticStatus levels to FaultManager severities:
      - ERROR (2)
      - Error-level fault
    * - STALE (3)
-     - CRITICAL (3)
-     - Stale data treated as critical
+     - ``stale_severity`` (CRITICAL by default), per diagnostic name via
+       ``stale_severity_overrides``
+     - See `Configurable STALE severity`_
 
 Main Components
 ---------------
@@ -148,7 +154,8 @@ Main Components
 
 3. **FaultReporter Integration** - Bridges to FaultManager
    - OK status → ``report_passed()`` for healing
-   - WARN/ERROR/STALE → ``report()`` with mapped severity
+   - WARN/ERROR/STALE → ``report()`` with mapped severity and the status ``values`` as
+     evidence
 
 Configuration
 -------------
@@ -172,6 +179,13 @@ Parameters
    * - ``name_to_code.<name>``
      - (none)
      - Custom mapping from diagnostic name to fault code
+   * - ``stale_severity``
+     - ``CRITICAL``
+     - Severity a STALE status reports at (``INFO``, ``WARN``, ``ERROR``, ``CRITICAL``)
+   * - ``stale_severity_overrides.<name>``
+     - (none)
+     - Severity for STALE statuses whose diagnostic name starts with ``<name>``; longest
+       prefix wins
 
 Example Launch Configuration
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -208,8 +222,24 @@ Examples:
 - ``"/robot/arm"`` → ``"ROBOT_ARM"``
 - ``"sensor: status"`` → ``"SENSOR_STATUS"``
 
-STALE as CRITICAL
-~~~~~~~~~~~~~~~~~
+Configurable STALE severity
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+STALE maps to ``stale_severity``, CRITICAL by default, because stale data often means a lost
+connection or a crashed node. But some nodes go STALE on purpose (a GPS in a tunnel, an IMU
+while it settles), and CRITICAL skips debounce in the fault manager. So the severity is
+configurable. ``stale_severity_overrides.<name>`` sets it per diagnostic name, and the longest
+matching prefix wins. That is why ``map_to_severity`` takes the diagnostic name as well as the
+level. An override that cannot be parsed is logged and ignored, and ``stale_severity`` is used.
+
+A non-CRITICAL STALE is only held back if the fault manager's ``confirmation_threshold`` needs
+more than one report. That threshold is picked by ``source_id``, which all statuses from the
+bridge share, so it applies to every diagnostic the bridge forwards.
+
+Evidence
+~~~~~~~~
 
-STALE diagnostics are mapped to CRITICAL severity because stale data typically indicates
-a communication failure or node crash, which is a serious system health issue.
+The status ``values`` are sent as ``ReportFault.evidence`` with every FAILED report. The fault
+manager keeps them per occurrence and serves them in the fault's freeze frame under
+``"x-medkit": {"reported": ...}``, grouped by ``source_id``. See
+:doc:`/config/diagnostic-bridge`.