FEAT-0007: Manifest-Backed Dashboard Feeds

Header

1. Summary

For each Dashboard, maintain a manifest tag dashboard-{dashboard_id} on every sensor the dashboard actually reads, kept in sync as the dashboard is created, edited, and deleted. This lets a future control-api-ui change subscribe to one GET /api/manifests/dashboard-{id}/events SSE stream per dashboard instead of polling each panel’s sensors independently.

2. Context & Motivation

#265. FEAT-0005 gave control-api-ui a persisted, multi-dashboard store, but layout is an intentionally opaque, unvalidated JSON blob (FEAT-0005 §3, “Out of Scope” — “Validating the internal shape of layout”). control-api’s existing manifest system (ManifestHub, GET /api/manifests/{tag} //events) already does exactly the kind of tag-scoped live feed control-api-ui would want per dashboard — but nothing currently populates a dashboard-{id} tag on any sensor, and control-api-ui does not currently subscribe to the manifest SSE endpoint at all (confirmed by inspection — usePanelSeries/useHealthStatsPanel poll getSensor/sensorHistoryQueryOptions directly via TanStack Query). This spec produces the tag machinery on the control-api side; wiring control-api-ui’s panels to consume it is separate, future frontend work.

References:

• app/runtime/manifest_hub.py, app/services/manifest_service.py, app/routers/manifests.py (existing tag-scoped manifest/SSE system this spec feeds)
• app/repositories/sensor_repo.py:123-126 (get_sensors_with_tag, the JSONB .contains() query pattern this spec’s reconciliation reuses)
• app/services/sensor_service.py:42-84,131-174 (upsert_sensor / _fan_out_upsert — precedent for both the reserved-tag-prefix guard and the “best-effort, must never fail the primary write” fan-out posture this spec copies)
• docs/specs/FEAT-0005-dashboard-store.md (the Dashboard CRUD resource this spec extends; establishes the opaque-layout contract this spec deliberately does not touch)
• control-api-ui src/lib/dashboardLayout.ts, src/hooks/usePanelSeries.ts, src/hooks/useHealthStatsPanel.ts (confirms why layout can’t be parsed server-side for this — see Design Decision below)

3. Scope

In Scope

• A new sensors field on Dashboard/DashboardCreate/DashboardRead/ DashboardPatch: the explicit, client-supplied list of (device_id, sensor_ref, data_type) triples the dashboard’s current layout reads. See Design Decision.
• Reconciling the dashboard-{id} tag on app.sensors rows to match sensors whenever a dashboard is created, has sensors change via PATCH, or is deleted.
• Atomic, race-free tag add/remove at the repository layer (new SensorRepository.add_tag_to_sensors/remove_tag_from_sensors), replacing the read-then-full-replace pattern SensorPatch.tags uses today for this one purpose.
• A reserved dashboard- tag-name prefix: SensorService.upsert_sensor preserves a sensor’s existing dashboard-* tags when a caller’s PATCH .../sensors/... supplies a tags list, instead of letting it silently overwrite them (see FR-6).
• Fanning newly-added/removed dashboard tags out to ManifestHub, so an already-open SSE subscription to dashboard-{id} sees the sensor appear/disappear without a reconnect — mirroring SensorService._fan_out_upsert.
• One Alembic migration adding the sensors column to app.dashboards.

Out of Scope

• Parsing control-api-ui’s layout blob server-side. Rejected explicitly — see Design Decision.
• control-api-ui actually populating sensors or consuming the manifest SSE stream. Separate, tracked frontend work; this spec only makes the backend capability exist.
• Reconciling tags when a not-yet-reporting sensor referenced by sensors later starts reporting. See NFR-3 — accepted limitation, self-heals on the dashboard’s next save.
• Fully race-proofing the reserved-tag-prefix preserve against a concurrent dashboard-tag reconciliation on the same sensor. See NFR-4 — narrowed, not solved.
• Any endpoint or schema change to the manifest system itself. dashboard-{id} is just a tag value; GET /api/manifests/{tag} and .../events already handle any tag.

Design Decision: Client-Supplied sensors, Not Server-Side layout Parsing

The obvious-looking alternative — have control-api decode layout itself and derive the sensor set — does not actually work cleanly, for one panel type specifically. chart/gauge panels store series[].{deviceId,sensorRef} plus a panel-level dataType, which would parse cleanly. But a health_stats panel stores only healthStats: {bodyDeviceId?, heartRateDeviceId?} (confirmed in serializeDashboardLayout) — its 7 sensor_refs are frontend constants (BODY_SENSOR_REFS, HR_SENSOR_REF in useHealthStatsPanel.ts), and each role’s data_type is discovered at render time from listDeviceSensors, not stored anywhere. Reproducing that resolution server-side would mean duplicating a constant list that lives in control-api-ui with no shared source of truth — a second copy that silently goes stale (manifest quietly missing sensors, no error) the moment control-api-ui’s panel schema evolves, exactly the kind of cross-repo coupling layout’s opacity (FEAT-0005) was meant to avoid.

Decision: add an explicit sensors: [{device_id, sensor_ref, data_type}, ...] field, supplied by the client alongside layout, not derived from it. control-api-ui already resolves every panel’s sensors — including health_stats’s dynamic role lookups — to issue its own polling queries (usePanelSeries, useHealthStatsPanel); it can supply this list exactly, with no guessing on control-api’s side. This keeps layout fully opaque (FEAT-0005’s contract, untouched) and has no version-drift failure mode: sensors is a flat, stable shape independent of layout’s internal schema. The frontend work this requires is not “extra” — control-api-ui does not consume the manifest system at all today, so some frontend change is required regardless before this feature has any observable effect.

4. Functional Requirements

• FR-1: Dashboard/DashboardCreate/DashboardRead gain sensors: list[{ device_id: int, sensor_ref: str, data_type: str}], stored verbatim (deduplication happens implicitly at reconciliation time, not at storage time).
• FR-2: DashboardPatch gains sensors: list[...] | None. A PATCH that supplies layout MUST also supply sensors — rejected with 422 (schema-level, model_validator) if layout is present without sensors. This prevents a layout change from silently leaving the tag set stale. The reverse is allowed: sensors may be supplied alone (no layout) — this is the recovery path NFR-3 relies on for a dashboard whose referenced sensor didn’t exist yet at save time.
• FR-3: On POST /api/dashboards/ success, the tag dashboard-{id} MUST be added to every sensor in sensors (order-independent; empty sensors is valid — no tags applied).
• FR-4: On PATCH /api/dashboards/{id} success where the patch includes sensors, the tag dashboard-{id} MUST end up applied to exactly the sensors in the new sensors list: added to sensors newly present, removed from sensors no longer present, left untouched on sensors present in both.
• FR-5: Reconciliation (FR-4) determines “currently tagged” by querying get_sensors_with_tag(f"dashboard-{id}") at reconciliation time — not by diffing against the dashboard’s previous sensors/layout value. This makes reconciliation idempotent and self-healing: any prior drift (e.g. NFR-3’s not-yet-reporting case) is corrected on the next save regardless of cause.
• FR-6: SensorService.upsert_sensor MUST treat the dashboard- prefix as reserved: when a caller’s patch includes tags, any of the sensor’s existing dashboard-* tags are preserved (re-appended) even if absent from the caller’s list, and any dashboard-* entries in the caller’s supplied list are stripped (silently — not a validation error) before the write. A caller can never add or remove a dashboard-* tag directly through PATCH .../sensors/...; only dashboard reconciliation (FR-3–FR-5) may.
• FR-6a: The prior-tags read FR-6 depends on (old_tags, upsert_sensor) is already wrapped in try/except Exception: logfire.exception(...) so it can never break the upsert — but under FR-6, silently treating a failed read as old_tags = [] would mean preserved comes back empty and the caller’s list wins outright, wiping every dashboard-* tag on that sensor: the exact outcome FR-6 exists to prevent. If that read fails, the tags field MUST be dropped from the patch entirely (every other field in the patch still applies) rather than proceeding with an empty preserved set — a tags-edit that silently no-ops on a transient read failure is an acceptable, rare degradation; silently deleting dashboard membership is not.
• FR-7: On DELETE /api/dashboards/{id} success, the tag dashboard-{id} MUST be removed from every sensor currently carrying it (queried the same way as FR-5). Unlike FR-3/FR-4, this reconciliation MUST run before the dashboard row is deleted, not after (see NFR-3a) — the delete only proceeds once the untag step has been attempted.
• FR-8: Tag add/remove (FR-3, FR-4, FR-7) MUST be atomic, idempotent DB-level operations (JSONB ||/- on app.sensors.tags, scoped by WHERE to rows not already in the target state) — not an application-level read-full-list-then-write cycle. This is simpler and self-evidently idempotent, not primarily a concurrency fix (see NFR-4 for the residual, accepted race).
• FR-9: Every sensor actually changed by FR-3/FR-4/FR-7 (i.e. rows the WHERE clause matched — not the full target/removal set, which may include no-ops) MUST be fanned out to ManifestHub: publish_sensor_update for each newly-tagged sensor (using the post-add SensorRead, so ManifestHub sees the new tag), and publish_sensor_removed(sensor, ["dashboard-{id}"]) for each newly-untagged one (the pre-removal SensorRead is sufficient — only its composite key is used).
• FR-10: Reconciliation for create/update (FR-3, FR-4) MUST be best-effort: a failure (DB error on the tag write, or a ManifestHub fan-out error) MUST be logged via logfire.exception and MUST NOT fail the request, whose dashboard row is already committed. Mirrors SensorService._fan_out_upsert’s posture. This relies on FR-5’s self-healing diff — a swallowed failure here is corrected on the dashboard’s next save.
• FR-10a: Reconciliation for delete (FR-7) is deliberately not best-effort in the same way, because FR-10’s self-healing argument doesn’t apply once the dashboard row is gone (see FR-7, NFR-3a): the untag step MUST be attempted and MUST succeed before the dashboard row is deleted; if it raises, the exception propagates and the delete does not proceed (dashboard survives, request fails, retryable). A ManifestHub fan-out error during that untag step, however, stays best-effort/logged-only, same as create/update — only the tag write itself gates the delete.

5. Non-Functional Requirements

• NFR-1: Follow existing layering — reconciliation logic lives in DashboardService, which gains a SensorRepository alongside its existing DashboardRepository (both constructed from the same injected db: AsyncSession, per this repo’s multi-repository DI convention — get_dashboard_service already takes db directly, so this is additive, not a signature change).
• NFR-2: Tag mutation SQL casts app.sensors.tags (JSON) to JSONB per-query, mirroring get_sensors_with_tag’s existing cast(Sensor.tags, JSONB).contains(...) — the column itself stays plain JSON (FEAT-0005’s “no JSONB columns” convention is about column type, not query-time casts, which already exist in this file).
• NFR-3: No reconciliation is attempted for a sensors entry with no matching app.sensors row (sensor hasn’t reported yet). Accepted limitation: that sensor simply doesn’t join the manifest until it exists, and there is no background sweep to catch it up automatically. Recovery is re-PATCHing the dashboard with (at least) sensors (FR-2 allows sensors alone, without layout) once the sensor has reported — FR-5’s live-diff design makes this a correct, ordinary reconciliation, not a special repair path.
• NFR-3a: NFR-3’s “recovery is the next save” argument is exactly why FR-10’s best-effort posture is safe for create/update but not for delete: there is no next save for a dashboard that no longer exists, so a swallowed untag failure on delete would orphan dashboard-{id} tags permanently and leave GET /api/manifests/dashboard-{id} serving a dead dashboard’s feed indefinitely. FR-7/FR-10a’s reconcile-before-delete, fail-the-request-on-error ordering exists specifically to avoid that.
• NFR-4: FR-6’s read-merge-write (existing sensor’s dashboard-* tags, merged with the caller’s non-dashboard-* tags, written as one replaced list) is not fully race-proof against a PATCH .../sensors/... tags-edit landing concurrently with a dashboard reconciliation on the same sensor — the reconciliation’s atomic add (FR-8) could land between the operator patch’s read and write and be overwritten. Accepted as a narrow, low-frequency edge case (this repo’s sensor ingest path never sends tags — see sensor_reading() — so the only real trigger is a human editing tags in the UI at the same moment a dashboard is saved); not solved by this spec. A future fix would need FR-6’s preserve step to also be a DB-level atomic operation rather than app-level merge.
• NFR-5: No change to GET /api/manifests/{tag} or .../events — both already accept any tag string; dashboard-{id} requires no new routing.

6. Interface Contract

DashboardSensorRef: { device_id: int, sensor_ref: str, data_type: str }

POST   /api/dashboards/            body now requires `sensors: [DashboardSensorRef, ...]`
                                    alongside `name`/`layout` (may be `[]`)
PATCH  /api/dashboards/{id}        `sensors` optional, but required together with
                                    `layout` (both-or-neither) -> 422 if only one given
DashboardRead                      gains `sensors: [DashboardSensorRef, ...]`

No new routes. Manifest access is the existing:
GET /api/manifests/dashboard-{id}          -> Manifest snapshot
GET /api/manifests/dashboard-{id}/events   -> SSE snapshot + delta stream

7. Acceptance Criteria

• AC-1: Creating a dashboard with a non-empty sensors list results in every listed sensor carrying tag dashboard-{id}, verifiable via GET /api/manifests/dashboard-{id}.
• AC-2: PATCHing a dashboard’s sensors to a list that drops sensor A and adds sensor B results in A no longer carrying dashboard-{id} and B carrying it; sensors present before and after are untouched (their other tags, if any, unaffected).
• AC-3: PATCH supplying layout without sensors returns 422; PATCH supplying sensors without layout succeeds (FR-2’s one-directional constraint — this is NFR-3’s recovery path). POST requires both name and layout regardless (FEAT-0005, unchanged) plus sensors (FR-1, defaults to [] if omitted rather than being required-and-absent-triggers-422, since DashboardAttrs.sensors has a default_factory).
• AC-4: Deleting a dashboard removes dashboard-{id} from every sensor that had it; the dashboard row itself is gone (existing FEAT-0005 behavior, unchanged).
• AC-5: A PATCH .../devices/{id}/sensors/{ref}/{type} supplying tags: ["dashboard-3", "custom"] on a sensor whose existing tags include dashboard-3 and dashboard-7 results in stored tags ["custom", "dashboard-3", "dashboard-7"] (order not asserted) — dashboard-7 preserved, the caller’s literal dashboard-3 deduplicated against the preserved one, custom applied.
• AC-6: Referencing a sensor in sensors that has no app.sensors row yet does not error the dashboard create/update; that entry is simply not tagged (NFR-3).
• AC-7: An open SSE subscription to dashboard-{id} receives a delta event adding a sensor within one flush interval of a PATCH that adds it to sensors, and a removed_sensor_ids entry when a PATCH removes it — no reconnect required.
• AC-8: A dashboard create/update still succeeds (2xx) even if tag reconciliation raises internally (simulate via a monkeypatched repository method) — verifies FR-10’s best-effort posture. Conversely, a dashboard delete whose untag step raises does not delete the dashboard row — the request fails and the dashboard is still fetchable afterward (FR-10a, NFR-3a).
• AC-8a: A sensor PATCH .../sensors/... supplying tags whose prior-tags read fails (simulate via a monkeypatched repository get) still applies every non-tags field in the patch, and leaves the sensor’s existing dashboard-* tags untouched (FR-6a) — the patch does not silently wipe them.
• AC-8b: Adding the sensors migration column to a pre-populated app.dashboards table (seed at least one row before running the migration in a test/staging apply) succeeds without a manual backfill step (§8’s server_default requirement).
• AC-9: uv run pytest tests/ green; black . clean.

8. Data & State Changes

• Schema: app.dashboards gains sensors — JSON, nullable=False, default=list, matching layout’s column shape.
• Migration: new Alembic revision (next available id after the current head at implementation time — re-verify head immediately before creating; do not assume 20260801_01_add_sensor_cadence.py is still the head). app.dashboards is a populated table by this point (FEAT-0005 shipped), unlike 20260730_01_add_dashboards.py (which created the table fresh) — a plain nullable=False add_column with only a Python-side default=list fails against existing rows, since ORM-level defaults don’t backfill. Follow 20260801_01_add_sensor_cadence.py’s cadence_samples column instead: op.add_column("dashboards", sa.Column("sensors", sa.JSON(), nullable=False, server_default=sa.text("'[]'::json")), schema=_schema) — a real DB-side default, applied to existing rows at add-time.
• No migration needed for app.sensors — tags already exists (FEAT-0005-era); this spec only changes how it’s written for the dashboard-* subset, via new repository methods, not a new column.

9. Implementation Guidance

1. app/schemas/dashboards.py:
   • Add DashboardSensorRef(UTCBaseModel): device_id: int, sensor_ref: str, data_type: str, extra: "forbid". (A local BaseModel, not the trigger subsystem’s SensorRefJSON TypedDict in app/trigger_expr/types.py — that type is internal to the trigger-expression tree, not a public API schema; this resource follows this file’s own existing pattern instead.)
   • Add sensors: list[DashboardSensorRef] = Field(default_factory=list, ...) to DashboardAttrs (covers Create/CreateInternal/Read via inheritance).
   • Add sensors: list[DashboardSensorRef] | None = Field(None, ...) to DashboardPatch and DashboardPatchInternal.
   • Add a model_validator(mode="after") on DashboardPatch enforcing FR-2 (("layout" in self.model_fields_set) == ("sensors" in self.model_fields_set)).
2. app/models/dashboard.py — add sensors: Mapped[list[dict]] (JSON, nullable=False, default=list), same shape as layout.
3. app/repositories/protocols.py (SensorRepository) + app/repositories/sensor_repo.py — two new methods:
   • add_tag_to_sensors(keys: list[SensorId], tag: str) -> list[SensorRead]: single UPDATE ... WHERE (device_id, sensor_ref, data_type) IN (...) AND NOT (tags::jsonb @> :tag_json) SET tags = (tags::jsonb || :tag_json)::json RETURNING * (via SQLAlchemy Core update().returning()); empty keys short- circuits to [] without a query.
   • remove_tag_from_sensors(keys: list[SensorId], tag: str) -> list[SensorRead]: same shape, WHERE tags::jsonb @> :tag_json, SET tags = (tags::jsonb - :tag)::json.
   • Both reuse the tuple_(...).in_(...) pattern already imported in this file (get_tags_for_sensor_keys) for the key filter.
4. app/services/dashboard_tags.py (new, small) — DASHBOARD_TAG_PREFIX = "dashboard-" and def dashboard_tag(dashboard_id: int) -> str. Imported by both DashboardService (FR-3/4/7) and SensorService (FR-6) so the literal prefix exists in exactly one place.
5. app/services/sensor_service.py (upsert_sensor) — FR-6/FR-6a: the existing old_tags read is already inside a try/except Exception: logfire.exception(...); old_tags = [] — add an explicit flag (e.g. old_tags, tags_read_ok = [...], True / False on the except branch) rather than reusing the empty-list fallback as a signal. When "tags" in patch_fields: if tags_read_ok, compute preserved = [t for t in old_tags if t.startswith(DASHBOARD_TAG_PREFIX)] and caller_tags = [t for t in patch_fields["tags"] if not t.startswith(DASHBOARD_TAG_PREFIX)], and set patch_fields["tags"] = sorted(set(preserved) | set(caller_tags)) (note: this dedupes and reorders the caller’s entire tags list, not just the dashboard-* subset — an intentional, minor behavior change from today’s verbatim passthrough, not just a dashboard-scoped tweak). If not tags_read_ok, drop "tags" from patch_fields entirely before calling self.repo.upsert (FR-6a) — every other field in the patch still applies.
6. app/services/dashboard_service.py:
   • Constructor takes sensor_repo: SensorRepository alongside the existing dashboard_repo.
   • Add async def _reconcile_tags(self, dashboard_id: int, target: list[ DashboardSensorRef]) -> None implementing FR-4/FR-5/FR-9 (diff against get_sensors_with_tag, call add_tag_to_sensors/remove_tag_from_sensors, fan out via get_manifest_hub()).
   • create_dashboard/update_dashboard (only if "sensors" in update.model_fields_set) call _reconcile_tags wrapped in a broad try/except Exception: logfire.exception(...) per FR-10 — never re-raised, called after the row is committed.
   • delete_dashboard calls _reconcile_tags(dashboard_id, []) unwrapped (FR-7/FR-10a — the tag-write portion must propagate) before calling self._dashboard_repo.delete(dashboard_id); only the ManifestHub fan-out portion inside _reconcile_tags stays try/except-guarded even on this path (FR-10a).
7. app/dependencies.py — get_dashboard_service already takes db: AsyncSession = Depends(get_db) directly (FEAT-0005’s shape); just add SQLAlchemySensorRepository(db) as a second constructor argument. No signature change needed elsewhere.
8. Alembic — new revision file adding the sensors column (§8).
9. Tests:
   • tests/unit/repositories/test_sensor_repo.py (or integration, since this needs real Postgres JSONB operators) — add_tag_to_sensors/remove_tag_from_sensors idempotency and the RETURNING-only-changed-rows behavior.
   • tests/unit/services/test_sensor_service.py — FR-6/AC-5 (preserve + strip + dedupe).
   • tests/unit/services/test_dashboard_service.py — FR-3/4/5/7/10 (mock both repos; assert add_tag_to_sensors/remove_tag_from_sensors called with the right diffed key sets; assert reconciliation exceptions are swallowed).
   • tests/integration/test_dashboard_repo.py or a new tests/integration/test_dashboard_manifest_reconciliation.py — AC-1, AC-2, AC-4, AC-6 end-to-end against real Postgres.
   • tests/api/test_dashboards.py — AC-3 (422 on mismatched layout/sensors).
   • A ManifestHub-integration test (real hub, fake session factory) — AC-7.

Things to avoid

• Parsing layout server-side to derive sensors — see Design Decision.
• Diffing against the dashboard’s previous sensors value to decide what to add/remove — FR-5 requires diffing against get_sensors_with_tag’s live result, for self-healing idempotency.
• A read-then-full-list-write for the dashboard-* tag itself (FR-8) — only FR-6’s caller-facing merge (a different, narrower, already-accepted race per NFR-4) may still do that.
• Letting a reconciliation failure raise out of create_dashboard/update_dashboard/ delete_dashboard — FR-10 requires it stay best-effort, like SensorService._fan_out_upsert.
• Reusing app.trigger_expr.types.SensorRefJSON as the Pydantic field type for DashboardSensorRef — it’s a TypedDict owned by an unrelated subsystem, not this resource’s schema file.
• Adding the sensors column with only nullable=False + a Python-side ORM default=list and no server_default — app.dashboards already has rows by this point; the migration fails without a real DB-side default (§8).
• Running delete-path reconciliation after DashboardRepository.delete (mirroring create/update’s ordering) or swallowing its tag-write failure — both leave orphaned dashboard-{id} tags with no dashboard left to re-save and self-heal them (FR-7, FR-10a, NFR-3a).
• Treating a failed prior-tags read in upsert_sensor as old_tags = [] for purposes of FR-6’s preserve step — that silently wipes every dashboard-* tag on the sensor instead of the intended no-op-on-tags degradation (FR-6a).

10. Dependencies

• Consumer: control-api-ui, which must start (a) computing and sending sensors alongside layout on every create/update, and (b) actually subscribing to GET /api/manifests/dashboard-{id}/events to get any benefit from this — both tracked separately in that repo. Until that lands, this spec is inert (dashboards get tagged; nothing consumes it).
• No new libraries. Postgres JSONB ||/- operators are used via raw casts, same as get_sensors_with_tag already does — no new extension/dependency.

11. Definition of Done

• sensors field added to Dashboard/schemas, with the layout↔sensors both-or-neither PATCH constraint (FR-1–FR-2, AC-3).
• Tag reconciliation on create/update/delete, diffed against live get_sensors_with_tag state (FR-3–FR-5, AC-1, AC-2, AC-4).
• Atomic add_tag_to_sensors/remove_tag_from_sensors repository methods, no read-modify-write for the dashboard-tag path (FR-8).
• dashboard- reserved-prefix preserve/strip in SensorService.upsert_sensor (FR-6, AC-5).
• Reconciliation fans out to ManifestHub; best-effort for create/update but failure-propagating (tag-write portion only) for delete (FR-9, FR-10, FR-10a, AC-7, AC-8).
• Prior-tags read failure in upsert_sensor degrades to skipping the tags field, never to wiping dashboard-* tags (FR-6a, AC-8a).
• Not-yet-reporting sensors documented as a self-healing limitation, not an error (NFR-3, AC-6); sensors-alone PATCH is the documented recovery path.
• Migration adds sensors column with a real server_default (not just an ORM default), safe against app.dashboards’ existing rows (AC-8b).
• Unit + integration + api tests cover AC-1–AC-8.
• uv run pytest tests/ green; black . clean.