SenseBridge Docs

Awareness feature completion + on-device/local/cloud reasoning tiers

Design for finishing the Awareness feature (ObstacleAwarenessView + AmbientAwarenessSession) to production/investor-demo quality, and building the reasoning backend the owner asked for: On-Device (Apple Foundation Models), Local — self-hosted endpoint (user’s own LAN/internet server), and Cloud (BYOK: Anthropic, OpenAI, NVIDIA NIM). A fourth bucket, a bundled on-device model, is named in the UI as not-yet-available rather than built — see “Bundled local model” below for why that’s a change from the first draft.

Owner decisions this design is built on:

Scope boundary, stated plainly

Everything below ships as real, working code this session except the bundled local model. First draft scaffolded a BundledLLMSceneComposer conformance and a disabled Settings enum case for it; this revision drops both (see “Bundled local model”). The follow-up (model choice, Core ML conversion, license ledger entry, on-device benchmarking) is tracked in TODO.md as its own item.

Architecture

Unify on SceneComposer, retire CloudReasoningAdapter

CloudReasoningAdapter (CloudOptional/CloudReasoningAdapter.swift) has zero conformances today and an identical shape to SceneComposer (compose(from:) async throws -> String). Delete it and CloudOptional/; every backend conforms to SceneComposer.

Collapsing on-device and network composers into one protocol does lose the ability to distinguish them by type — needed for the in-flight/cost-control UI below. That distinction lives as data, not a type hierarchy:

extension ReasoningBackend {
    var usesNetwork: Bool { self == .localEndpoint || self == .cloud }
}

The resolver already knows which backend it chose; nothing needs a second protocol to ask “is this one a network call.”

Where the new composers live

AnthropicSceneComposer and OpenAICompatibleSceneComposer go in SenseBridgeCore/Reasoning/, not the App layer — unlike FoundationModelsSceneComposer, nothing about them requires a framework the core package can’t depend on (URLSession is Foundation). Each takes an injected URLSession so the whole request/response/validator matrix runs under swift test, no Xcode or simulator required — the same reason the SenseBridgeCore package seam exists at all (docs/ARCHITECTURE.md). Credential storage follows the existing SettingsStore/ UserDefaultsSettingsStore split: an APICredentialStore protocol in the package, a Keychain-backed implementation in the App layer.

OpenAICompatibleSceneComposer covers three backends

Base URL + optional key + model name is the whole delta between OpenAI cloud, NVIDIA NIM, and a self-hosted endpoint (Ollama, LM Studio, vLLM all expose an OpenAI-compatible /v1/chat/completions route) — one implementation serves all three. Two things this revision adds that the first draft assumed away:

The output validator — the actual safety enforcement point

This is the correction to the first draft’s central claim. The first draft said the network composers stay safe because they’re prompted the same way FoundationModelsSceneComposer constrains its model. That’s not true: the on-device composer is safe because @Generable constrains the reply structurally, at the decoding layer — Apple’s runtime, not a request you sent. Over HTTP there is no decoding layer you control. response_format is a request to a server you don’t run, and for the self-hosted tier the server is arbitrary. A remote returning a full, confident, unhedged sentence and Phrasing.describe (String(format:), which wraps anything) would produce something like “it looks like there’s There is a car about 2 feet ahead — dangerous” — a distance and danger claim the detector never earned, spoken to a walking blind user. That’s the worst-bug archetype this project’s audit guide names, reachable through an ordinary non-malicious path (a self-hosted model ignoring the schema; an OpenAI-compat layer silently dropping it).

Required addition: a local, deterministic ReasoningOutputValidator between the HTTP response and Phrasing, in SenseBridgeCore/Reasoning/, used by every network composer before the phrase reaches Phrasing.describe. Reject (→ throw, caught by the resolver, falls back to on-device — same contract as FoundationModelsSceneComposer.modelPhrase(for:) returning nil today) when the reply:

Required test, and it is the regression test for the worst bug in this project: a URLProtocol mock returning an unhedged, distance-and-danger full sentence must deterministically throw or resolve to a phrase wrapped in a known hedge template — never the raw sentence.

The wire contract: what actually gets serialized

SceneComposer.compose(from:) takes [PerceptionRecord], whose .kind includes .recognizedText (OCR) alongside .detectedObject. FoundationModelsSceneComposer filters to .detectedObject only (FoundationModelsSceneComposer.swift:84-87) as a side effect of its implementation, not a stated rule — the first draft inherited the protocol without restating it. If a future change ever routes Reading through the resolver, a user’s OCR’d bank statement or medical letter would go to Anthropic under a design that only ever said “labels only” in prose.

Stated rule, enforced at the boundary, and tested: every network composer serializes .detectedObject labels only. .recognizedText, .detectedSound, and .depthReading records are dropped before the request is built. Confidence values stay on-device — the hedge is computed from them locally; the provider never receives them and has no use for them.

Resolution, concurrency, and fallback

ReasoningComposerResolver (App layer — needs AppEnvironment for Keychain and Settings) picks the active composer from Settings.

Composition must not block the depth cadence — required change. AmbientAwarenessSession.run → tick → describeIfDue → the composer, all awaited in sequence (AmbientAwarenessSession.swift:185-194,293). On-device that costs a few hundred ms; over a network — cellular, a slow self-hosted box, a stalled connection — the 750 ms depth loop stalls for the whole round-trip, breaking the stated invariant that depth sampling never waits on a narration cadence chosen for comfort (docs/ARCHITECTURE.md, “Two cadences, one loop”). That’s a doctrine 1 defect, invisible in the simulator.

Fix: composition runs as a tracked child Task, single-flight (a tick that finds one already in flight is skipped, not queued — matching the existing environment.speech.isSpeaking skip-not-queue pattern at AmbientAwarenessSession.swift:264-266), cancelled alongside loopTask in stop(). Add a staleness guard: if the response arrives after more than 2× narrationIntervalSeconds from when its source frame was captured, discard it — describing a room the user already walked out of is a false statement about the present, the same principle that already governs stale detection outlines (:207-217).

Fallback disclosure — three distinct cases, not one “silent” rule. The first draft’s blanket “any failure falls back silently” undersells doctrine 4’s “never let a limitation go unstated.” A user who configured and is paying for a cloud tier and is silently getting free-tier quality has had a limitation go unstated.

  1. Never configured / consent declined — on-device is simply the active backend; the Settings row already names it. Nothing to announce.
  2. Configured but failing (timeout, expired key, rate limit, endpoint down) — the user believes they’re getting a tier they’re not. Announce once, through NarrationThrottle’s urgent path so it isn’t swallowed: “Cloud descriptions aren’t responding, so SenseBridge is continuing with on-device descriptions.” Do not repeat per-failure — that trains the user to ignore the channel, the same reasoning that already governs routine narration.
  3. Configured and working — nothing to say per request.

Circuit breaker, doing double duty as disclosure and cost control: 2 consecutive network failures trips it, dropping to on-device for the rest of the session with the one announcement above. While tripped, probe the network composer again every 5th tick rather than every tick (bounds cost against a still-down endpoint) or every tick (would hammer it); a successful probe resets the breaker and announces recovery once, so the user isn’t left wondering indefinitely whether the good tier ever came back.

The in-flight cue is state, not speech. A cue fired on every request during hands-free awareness would repeat every narrationIntervalSeconds (default 6s) into the one channel the user is listening to — exactly what NarrationThrottle exists to prevent elsewhere. Instead: an observable isAwaitingNetworkResponse property (for accessibilityValue/visual state), plus the active backend named once, spoken, at session start — extending the existing “Descriptions are composed on-device by Apple Intelligence” footnote pattern (ObstacleAwarenessView.swift:129-136) to name all three backends.

Both call sites move off their hardcoded FoundationModelsSceneComposer.init() onto the resolver: AmbientAwarenessSession.start(environment:) and SceneDescriptionView.captureAndDescribe().

Data model (Settings.swift)

Replace cloudReasoningEnabled: Bool with:

public enum ReasoningBackend: String, Sendable, Codable {
    case onDevice, localEndpoint, cloud
}
public enum CloudProvider: String, Sendable, Codable {
    case anthropic, openai, nvidiaNIM
}
public var reasoningBackend: ReasoningBackend = .onDevice
public var cloudProvider: CloudProvider?
public var localEndpointURL: String?

(No .localBundledModel case — see “Bundled local model” below. No localEndpointModelName field either; the model name is part of the request body, not a durable setting worth its own field yet — pass a sane default per composer and revisit if a real need for per-endpoint model overrides shows up.)

Migration — corrects a bug in the first draft. Settings.init(from:) currently decodes cloudReasoningEnabled with plain try container.decode (non-optional, Settings.swift:97) — the first draft didn’t account for this. The new decode must:

Existing tests referencing the old key need updating in the same change: SettingsTests.swift:11,25,32,41,64, CrashReportingTests.swift:28.

API keys never enter Settings/UserDefaults. New APICredentialStore (protocol in SenseBridgeCore, Keychain implementation in the App layer), keyed by CloudProvider (plus one slot for the self-hosted endpoint’s optional bearer token). Required Keychain attributes the first draft omitted:

localEndpointURL is a destination, not a secret, and stays in Settings (plain UserDefaults) as originally planned — but see URL validation below for what has to happen before it’s ever used.

One ReasoningBackendSettingsView, reached from Settings. Segmented picker: On-Device / Local / Cloud.

Per TODO’s already-approved spec: default off, one-time explicit screen naming exactly what leaves the device before any network backend can be turned on, persistent off switch, per-provider ToS acknowledgment (unchecked, real Toggle with its own label/hint — never a bare tap target), link labeled with its destination, re-shown on every provider switch. The self-hosted copy says data goes to the address the user typed, on whatever network that reaches, and SenseBridge has no visibility past that point; if the URL is http://, the copy states the labels travel unencrypted on that network (doctrine 4 corollary 2 — cleartext isn’t a footnote).

Key entry accessibility — the biggest usability gap in the first draft. A blind user cannot proofread a SecureField character by character through VoiceOver. Primary path is a “Paste key from clipboard” button; the SecureField is the fallback. Either path ends in a “Test connection” round trip — one lightweight real request — that speaks success or the specific failure before the backend can be enabled, so the user isn’t left guessing whether a pasted key actually worked.

Safety framing (highest-severity surface — safety-framing-reviewer sign-off required)

Every network composer is still prompt-constrained the same way FoundationModelsSceneComposer constrains its model — bare noun phrase, no sentence, no hedge word, no distance/direction/danger claim — but the prompt is now correctly framed as a quality measure, not the safety mechanism. The validator is the safety mechanism: it runs after every network response, regardless of what the provider claims to support, and nothing reaches Phrasing.describe without passing it. Phrasing supplies the hedge from the detector’s confidence exactly as today; a network provider can degrade phrasing quality (via a rejected/fallback response) but cannot make the app assert certainty it hasn’t earned.

Pre-existing gap, worth fixing in the same pass since the sibling file is already open: FoundationModelsSceneComposer.instructions never pins the output language, while Phrasing wraps it in a localized template — an es/vi user can get "có vẻ như có a chair and a doorway." today. Both the on-device prompt and the new network prompts should state the target language explicitly; the validator’s language check above is the backstop.

Self-hosted endpoint — network-layer specifics the first draft didn’t cost

URLSession configuration (applies to every network composer)

waitsForConnectivity = false (the platform default of true would queue a request until connectivity returns — exactly the “walking, connectivity drops mid-session” scenario, resolved tens of seconds later into a description of somewhere the user no longer is), per-context request timeouts (~4s for hands-free composition, ~8s for one-shot Describe — a narration cadence can’t afford the platform’s 60s default), allowsConstrainedNetworkAccess = false (honors Low Data Mode), and no automatic retries anywhere in this stack — retries are the mechanism that turns a flaky endpoint into a runaway BYOK bill.

Cost controls (BYOK-specific — required, not optional)

At the default 6s narration interval, one hour of hands-free awareness is ~600 requests against the user’s own API budget. Required:

Bundled local model — dropped from this pass, kept as a disclosure

First draft scaffolded BundledLLMSceneComposer (a real SceneComposer conformance whose compose always throws) plus a .localBundledModel enum case, presented as a disabled Settings option. This revision drops both. Reasoning: ReasoningBackend is Codable and persisted — a case that exists now is a value that can be persisted now and needs migrating forever, and when a real bundled model ships it will need its own settings (model identifier, quantization, download state) that don’t fit this enum anyway, so the case gets rewritten regardless. A disabled radio button that presents as selectable is also a worse investor-demo experience than a plain disclosure row — this project already has the right pattern for “planned but unbuilt” at DiagnosticsSettingsSection.swift:27-36: a warning row stating the reason, not a control the presenter has to explain away.

What ships instead: a static, non-interactive row in the Local section — “Bundled on-device model — not yet available. Needs a benchmarked model; tracked as a follow-up.” Satisfies doctrine 4 (named, not hidden) without persisting a value nothing backs. TODO.md already tracks the real follow-up (model choice, Core ML conversion, license ledger, on-device benchmarking).

Awareness feature hardening

Testing

Docs/legal sync (same change) — expanded from the first draft

Shipping any network reasoning tier makes several currently-true “nothing leaves the device” statements false. First draft’s list only covered the architecture/privacy docs; this revision adds every surface that actually asserts the old claim:

Guardrails for the new tooling

Per AGENTS.md, every new tool/script ships with guardrails. The i18n completeness check below (if built) needs the same treatment: a narrow, read-only script that fails CI on a missing catalog key, nothing that writes or auto-translates.

What stays explicitly unverified after this session

Machine-checkable: build green, unit tests including the validator regression test, VoiceOver label presence via the existing XCUITest accessibility audit pattern. Not machine-checkable, stated rather than implied:

npm run app:install at the end proves the build runs on Kevin’s device; it proves none of the above.