prototype.md

OpenAware v0.4.0 developer prototype

OpenAware has a sandboxed Electron dashboard, private supervised service, continuous selected-feed previews and independent local AI agents. The current handoff records verification and delivery. Historical v0.3.3, v0.3.2 audit and v0.3.1 evidence applies to those earlier builds.

Implemented behavior

• One flat dashboard of independently movable/resizable source, assistant, activity and Agent desk/Memory panes. Two monitors start side by side; additional feeds form two-column rows above the desk. Source sizing follows uncropped video proportions. Live connections persist through pane movement, resizing and agent selection. Layout is temporary within Overview.
• Up to sixteen selected sources and four independent agents, each assigned up to four feeds. Each owns an explicitly discovered, selected and synthetically verified LM Studio, Ollama or llama.cpp connection, its own bounded inference queue, conversation, captions, rules and history. A feed is captured once and routed only to assigned engines. Selecting another seat changes visible model/chat/settings aliases without transferring bindings or histories. Each engine admits one inference at a time; separate engines can dispatch independently, with no hardware throughput promise.
• Agent 1 starts unconfigured as an operator that can also observe. Newly added agents start unconfigured/unassigned. New sources autoassign to Agent 1 while it has room, until its first explicit Feeds edit; other assignments are explicit. Unverified seats have the cyan glowing outline and exact Needs agent label. Per-seat Start/Pause preserve shared previews and other agents; global Stop releases every capture owner and invalidates every engine.
• Demo, monitor, window, camera and virtual-camera sources, plus local video files, direct HTTP(S) media and isolated web-player pages. Add source exposes clear file/link controls and Open player permits Play/Pause/seek or site sign-in. Video files/URLs are main-process registrations referenced by opaque IDs; registered paths/URLs are omitted from source/frame metadata and model request envelopes. The link form holds the entered URL while preparing it. Web links watch rendered page content, with no platform downloader, extraction, existing-browser login reuse or protected playback guarantee. See agents and video.
• Fixed native desktop capture owners and fixed file/direct-video decoders are sandboxed nonpersistent documents with no Node, preload or navigation. Direct video is streamed through an exact authorized same-origin route with Range support and bounded frame transport. Web player pages have separate nonpersistent sessions and no dashboard preload, Node, capture permissions, downloads or external protocol bridge. Remote resources are available to these explicitly opened players; the dashboard remains network-denied.
• Black privacy masks apply before image analysis. Original capture timestamps, exact source revisions, agent identity/configuration revisions, binding revisions and epochs preserve provenance. Background temporal monitoring uses up to three chronological masked images spanning four seconds. Caption time denotes capture time rather than a video playback-time index. Preview is continuous; sampled image analysis does not establish native every-frame video understanding.
• Each agent supports motion monitoring, up to eight scoped semantic rules, caption-text search and asynchronous historical text summaries. Alerts require fresh, revision-matched evidence and two distinct matching observations; uncertainty does not alert. Search/history are bounded in memory and are not raw video recording or vector search. These original workflows need no NVIDIA VSS server, code, model or containers.
• Only an active operator can propose a bounded typed action plan for an assigned desktop. Native execution still requires approval for every step, a fresh unchanged monitor/window inspection and exact identity/digest/revision checks. Switching agents, reassignment, role/model changes and Stop revoke authority. Video sources are not native action targets. No arbitrary shell, model-approved input, automatic trades or external messages are added.
• Background hides the same dashboard while configured captures and agents continue. Tray Show/Stop/Quit remain available. Background/headless launch starts idle in an interactive desktop session, restoring no previous source/model/watch. Configuration, tokens, captions, media registrations and player sign-ins are memory-only.
• The opt-in authenticated local CLI retains its whitelist; it cannot acquire sources, manage agents/providers, inject pixels or approve native input. Status exposes redacted per-agent identity, scope, model and readiness. Existing workflow commands default to the currently selected agent unless an allowed typed request pins agentId. Local workflows.

Use

1. Run npm ci and npm start with Node.js 24 or later, or open the tested Windows build.
2. Click + Add source, select monitor/window/camera/video, and connect. Live preview does not need a model or Start watching. For web videos, open the player and press Play or sign in if required; a normal browser Window source is the fallback when a site refuses this player.
3. Use Add agent and Feeds to choose the seat's role and up to four source assignments. Select the seat to use its assistant and Connections.
4. Start your local LM Studio/Ollama/llama.cpp server, discover models, explicitly select a vision model and run the displayed synthetic test. Each new agent requires its own verification. Then Start that agent's watch. Pausing one seat leaves other seats/capture running.
5. Configure per-agent temporal monitoring/rules or use Memory to search its retained captions. Rules/observations never authorize computer actions.
6. For an operator, request a plan on an assigned desktop and review each step natively. Stop all or Ctrl+Shift+F12 revokes remaining operations, but cannot undo a sent event.
7. Background continues the current session in the tray. Quit releases all owners and exits. Restart begins empty and idle.

Evidence and remaining gates

Current checks, counts, review verdicts, screenshots, package identities and delivery status are recorded in the v0.4.0 handoff. Synthetic file/HTTP/page playback validates decoder/capture/routing lifecycle. Mock models establish per-agent protocol isolation, not visual accuracy or multi-model hardware performance. Platform authentication, protected content, arbitrary codecs, actual personal cameras/videos, mixed-DPI input effects and Bionic interoperability remain live acceptance gates.

Background caption/summary jobs allow 60 seconds; probes/current chat/Operator allow 20 seconds. The newest frame must be within five seconds at dispatch. Current chat/Operator evidence must finish within fifteen seconds; delayed background captions may remain history but older semantic evidence cannot alert. The earlier tested Ollama model took about 18–20 seconds for a single synthetic image and timed out on temporal/summary requests; useful sustained monitoring is not claimed. There is no cloud fallback or bundled model.