Native player plan

The software we install on the screen device so it registers itself, plays the loop unattended and reports back. This is the development plan for that software: what it must do, how it is built, the exact shell per platform, the registration flow, offline and update behaviour, hardware to buy, the testing plan and a week by week build order for one developer.

Status (2026-10-24): plan written; the Mac and Windows shell (player-desktop/) runs; the Android shell (player-android/) builds from the command line (debug, unsigned release and lint pass, 2026-10-26; Gradle wrapper 8.11.1, AGP 8.7.2) and has not yet run on a device.

Related: web/content/docs/flow.md section "Native player instead of a browser" (product intent), docs/API_CONTRACT.md section "Device API" (the contract the player codes to), web/PLAYER.md (the browser player runbook), player-desktop/README.md, player-android/README.md.

1. Goals

The device must, with nobody in the mall touching it:

1. Start on boot. Power comes back after an outage, the device boots, the player is on screen within 60 seconds, no login, no click.
2. Survive power loss. Pairing, the day's schedule, the cached media and the queue of play records are on disk, not in a browser tab's memory.
3. Keep playing offline for hours. A wifi drop or a backend outage must not blank the screen. The player plays today's cached loop from local media and keeps recording plays.
4. Upload queued play records afterwards. When the network returns, batches go up oldest first in batch_seq order. Late uploads are the normal case, never an error (contract clock drift rule).
5. Self restart on stall. If the page hangs, the renderer crashes, or the video element stops advancing, the shell reloads the page. If the shell itself dies, a watchdog relaunches it.
6. Update the engine at night. The web engine updates itself from the server (the existing build id check). The shell updates itself from a signed manifest, at a quiet hour, never mid play.
7. Report health. The check-in (heartbeat) carries what the shell knows: uptime, free disk, temperature where available, camera present or not, shell version, last reload reason.
8. Keep the camera permission. Granted once at install, never prompted again, surviving reboots.
9. Stay in kiosk mode. No status bar, no home button, no notifications, no other app, no way for a shopper to leave the player.
10. Same pairing code flow as today. The screen shows a six character pairing code; ops types it in /ops/pair; nothing changes for the ops person or the backend.

Non goals for the first version: a second codebase for playback (the web engine stays the only player), a native vision model (the web engine's face counting runs inside the WebView), a store listing (side load first).

2. Architecture decision: one web engine, thin native shells

Decision: the player logic stays in the web engine (web/app/player plus web/lib/player). Each platform gets a thin native shell that hosts a browser view, points it at the player URL, and adds only what a browser tab cannot do: boot start, kiosk, camera grant, watchdog, health, updates.

Why one engine:

• The engine already implements the whole device contract: pairing, schedule pull, Cache API media cache, a persisted play record queue with strict batch_seq, heartbeat every 15 s, 401 handling, the games embed protocol, spins and cricket. Rewriting that three times in Kotlin, C# and Swift is months of work and three sets of bugs for the same play record format.
• The play ledger is the invoice. One code path producing play records means one thing to audit.
• The feat/devices branch is extending the same engine with a longer offline cache, queued play records that survive longer outages, health fields in the check-in and remote commands (reload, restart loop, clear cache). All of that lands in the shells for free the moment the engine updates on the server.
• Ops and the browser player keep working for tests; the shell is optional, not a fork.

Served from the server or bundled locally? Both, in this order:

• Phase 1 (now): the shell loads https://freebeeads.com/player from the server. Chromium keeps the page in its HTTP cache and the engine keeps state in localStorage plus the Cache API, so a reboot while offline still boots the last loaded page in practice, but it is not guaranteed.
• Phase 2: the shell registers a service worker aware boot. The web app ships /player as a PWA with a service worker that precaches the player shell (the feat/devices offline cache work). Then an offline boot is guaranteed: the service worker serves the page, the engine serves the media from the Cache API.
• Phase 3 (only if needed): bundle a built copy of /player inside the shell as a fallback origin, used only when the network page fails to load twice in a row. Costs a build step per release; defer until a mall shows it is needed.

The shell and the engine talk through one small JavaScript bridge, window.freebee, with the same methods on every platform (section 6).

3. The shell per platform

3.1 Android (first to build)

Devices: Android TV box or signage box behind the wall, or an Android tablet as the kiosk. One APK covers both.

Stack: Kotlin, Gradle, minSdk 26 (Android 8.0), target the latest stable SDK, AndroidX, no Compose (a WebView does not need it). Project skeleton is in player-android/.

Components:

• MainActivity: a single WebView filling the screen. Settings: JavaScript on, DOM storage on, mediaPlaybackRequiresUserGesture = false (autoplay), mixed content never, cache mode default, setSupportMultipleWindows(false). Immersive sticky fullscreen through WindowInsetsControllerCompat. Keeps the screen on with FLAG_KEEP_SCREEN_ON. Overrides WebChromeClient.onPermissionRequest to grant camera to the player origin only. WebViewClient.onReceivedError and onRenderProcessGone reload the page. The JS bridge is added with addJavascriptInterface(FreebeeBridge(...), "freebee").
• BootReceiver: ACTION_BOOT_COMPLETED plus QUICKBOOT_POWERON starts WatchdogService, which starts MainActivity. RECEIVE_BOOT_COMPLETED permission in the manifest.
• WatchdogService: a foreground service (notification channel "Player", low importance) that every 30 s checks that the activity is alive and in front and that the page heartbeat arrived in the last 60 s; otherwise it restarts the activity. The service is sticky (START_STICKY) so the OS restarts it if it is killed.
• FreebeeBridge: the @JavascriptInterface class with the methods in section 6.
• Camera: CAMERA permission in the manifest; with device owner it is granted at setup with adb shell pm grant or by the device owner policy, so no runtime prompt. USB cameras: Android TV boxes with a USB host port expose a UVC webcam as a normal camera on most signage firmware (RK3568 and RK3588 vendor images do; stock Android TV from Xiaomi and the like usually do not). Test on the exact box before buying twenty. If the box does not expose UVC to the WebView, the fallback is the libuvc based UVCCamera library feeding a local MJPEG stream the engine reads from http://127.0.0.1, which is a week of work; avoid by choosing a box that works.
• Kiosk: device owner mode. After a factory reset, before adding any Google account, run adb shell dpm set-device-owner in.freebee.player/.AdminReceiver. The app then calls setLockTaskPackages, startLockTask(), disables the status bar, keyguard and system error dialogs, and sets itself as the persistent preferred home activity. Exact commands are in player-android/README.md.
• Packaging: a signed release APK, side loaded with adb install or by copying to a USB stick and installing from the file manager, then device owner setup. Later: Android Enterprise managed Play (a private app in a managed Google Play account) so updates and the kiosk policy come from the MDM instead of adb.

3.2 Windows (second)

Devices: mini PC (Intel N100 class) driving the LED wall or a large display through the LED controller.

Stack: the same Electron shell as the Mac (player-desktop/), packaged with electron-builder as an NSIS installer. Electron, not "Chrome in kiosk mode", because the shell needs the preload bridge, the watchdog, the log, the camera auto grant and the auto updater, and Chrome kiosk gives none of those.

Boot: a scheduled task at log on of the kiosk user (scripts/install-autostart-windows.ps1), the kiosk user set to auto log on (netplwiz or the registry AutoAdminLogon keys), power plan set to never sleep, never turn off the display. The script sets all three.

Kiosk: Windows assigned access (Set-AssignedAccess) works only with UWP apps and Edge kiosk, so for the Electron shell the practical kiosk is: a dedicated standard user, auto log on, the shell as the only startup item, the shell window always on top, Win key and Ctrl+Alt+Del not blocked (they cannot be from user space) but the shell relaunches itself when it loses focus. Good enough for a mini PC behind a wall; a kiosk tablet in reach of shoppers should be Android.

Watchdog service: Electron relaunches its own renderer; a second line of defence is the scheduled task set to restart on failure (-RestartCount 999 -RestartInterval 1min), which relaunches the whole app if it exits.

3.3 Mac (third, the test rig today)

Devices: Mac mini driving the wall.

Stack: the same Electron shell, packaged as a DMG. Boot: scripts/install-autostart-mac.sh writes a launch agent (~/Library/LaunchAgents/in.freebee.player.plist, RunAtLoad, KeepAlive) so launchd restarts the shell whenever it exits, plus turns on auto login for the current user (System Settings, General, Login Items and Users, the script prints the two toggles it cannot flip without the password). Sleep is prevented with caffeinate style assertions from inside the shell (powerSaveBlocker) and sudo pmset -a sleep 0 displaysleep 0 from the script.

Camera: macOS asks once per app signature; the shell's Info.plist carries NSCameraUsageDescription, the first launch prompt is accepted by the installer, after that the shell grants the permission to the page origin itself (section 7 of the desktop README).

4. Device registration flow, step by step

Nothing changes on the backend. The shell only makes the existing flow unattended.

1. First boot. The shell starts (boot receiver on Android, launch agent on Mac, scheduled task on Windows), opens the player URL fullscreen. The engine finds no device token in localStorage, generates a device_uid, calls POST /v1/device/pairing/start {device_uid} and shows the six character pairing code at 19vw.
2. Ops enters the code. The ops person opens /ops/pair, types the code next to the screen it belongs to, which calls POST /v1/ops/devices/claim {code, screen_id}.
3. The device learns it is claimed. The engine polls POST /v1/device/pairing/poll every few seconds; a 200 answers {device_token, screen}. On a 410 (code expired) it starts over and shows a fresh code.
4. The token is stored. Today the engine keeps it in localStorage (dooh.player.state.v1). The shell makes that storage durable: on Android the WebView data directory is app private storage (not readable by other apps, cleared only by uninstall or "clear cache" from ops); on Electron the user data path is per user and the token is additionally mirrored, encrypted with safeStorage, into the shell config so a corrupted Chromium profile does not force a re pair (phase 2 of the shell; not in the first slice).
5. Schedule pull. The engine calls GET /v1/device/schedule?date=today, preloads every creative into the Cache API, and starts the slot loop. It refetches every 300 s, on the UTC date rollover, and whenever the heartbeat answer's schedule_version changes.
6. Heartbeat. Every 15 s the engine posts POST /v1/device/heartbeat {playing_creative_id, watchers_now} and the shell adds its health fields (section 8). Ops sees "last check-in" on the fleet page.
7. Revocation. Ops revokes the device; the next call answers 401; the engine drops the token, keeps the queue, and shows a new pairing code. The shell does nothing special.

What the shell adds to the screen during pairing: nothing visual. The pairing code stays the only thing on screen. The device identity line at the bottom (device_uid) is extended by the web player with the hostname from the bridge so ops can tell two unpaired boxes apart (one line change, section 6).

5. Offline model and the play record queue

What the engine does today (web/lib/player/engine.ts, storage.ts, media.ts):

• Schedule: cached in localStorage with its UTC date; a reload during an outage keeps playing today's loop.
• Media: every creative file and game asset is stored in the Cache API, keyed by creative id and checksum, fetched with the device token through the same origin proxy /player/api.
• Play records: one batch per play, appended to a persisted queue in localStorage before anything else, batch_seq strictly increasing and persisted; the drain uploads oldest first, one at a time, and dequeues on recorded, duplicate or rejected (rejected is terminal and kept for the overlay). Unreachable or 5xx: back off and keep the queue.
• Heartbeat: every 15 s, silently skipped offline.

What the feat/devices branch adds (being built in parallel; the shells build on it, they do not duplicate it):

• A longer offline cache: tomorrow's schedule and media prefetched the evening before, so a box that boots offline at 10 am still has today's loop.
• Queued play records hardened for multi day outages (quota aware, oldest never dropped, upload throttled on reconnect).
• Health fields in the check-in body.
• Remote commands answered in the heartbeat response: reload, restart_loop, clear_cache.

How the shell maps onto that:

Rule for both sides: the shell never writes a play record. Only the engine does.

6. The bridge: window.freebee

One object, the same methods on Android (addJavascriptInterface) and Electron (contextBridge.exposeInMainWorld). Every method is synchronous and returns a string or nothing, because Android's bridge cannot return objects or promises. JSON strings carry structured data.

Also present: shell() returns "electron" or "android" (a method, because the Android bridge cannot expose plain properties). The web player treats window.freebee as optional and keeps working in a plain browser.

The one line change the web player needs (not made here; it belongs with the feat/devices work on the heartbeat tick in web/lib/player/engine.ts, inside sendHeartbeat, or in the page's start effect):

// web/lib/player/engine.ts, at the top of sendHeartbeat():
(window as unknown as { freebee?: { heartbeat(): void } }).freebee?.heartbeat();

Because the engine heartbeats every 15 s regardless of network (the network call is skipped offline but the tick still runs), this keeps the shell watchdog satisfied while playing, offline or not. Until that line lands, the desktop shell's watchdog accepts a successful page load as the first heartbeat and then only reloads on crash, hang (unresponsive event) or load failure; the 60 s bridge timeout is armed only after the page has called heartbeat() once, so an older engine is never reload looped.

A second small change, optional: PairingScreen appends JSON.parse(window.freebee.deviceInfo()).hostname to the device_uid line so ops can tell boxes apart.

7. Update strategy

Web engine: unchanged. The player polls /build-id.txt every 5 minutes and reloads at the next slot boundary when the deployed build changes. The shell adds nothing; a reload inside the WebView picks up the new engine.

Shell, Android: a signed APK over the air.

• The web app serves https://<site>/player/updates/android.json: {version, versionCode, url, sha256, minVersionCode, publishedAt}, signed by a server side key (Ed25519, public key baked into the APK, libsodium or Tink). The engine, or a tiny poller in the shell (UpdateChecker work in WatchdogService), checks nightly between 02:00 and 04:00 local time.
• Download to app private storage, verify the sha256 and the signature, then install. With device owner the install is silent: PackageInstaller session with the app as the installer, no prompt. Without device owner Android shows an install prompt, which nobody in the mall will tap; so device owner is not optional.
• Roll back: the previous APK is kept; if the new version does not call heartbeat() within 5 minutes of its first start, the watchdog reinstalls the previous one.
• Later: managed Play handles all of this with a track per mall.

Shell, desktop: electron-updater (part of electron-builder) against a generic HTTPS provider: the web app's /player/updates/<platform>/latest.yml plus the signed installer. Mac requires code signing for auto update (an Apple Developer account, see section 15); Windows works unsigned but SmartScreen warns on first manual install, so sign with an OV certificate before the second mall. Updates install at the quiet hour and only when the engine reports phase !== "playing" or at a slot boundary; the shell asks the page through freebee only after that bridge method exists (phase 2).

8. Health and remote commands

Health fields the shell can provide (read by the engine through freebee.health() and sent in the heartbeat body that the feat/devices branch extends):

Remote commands. The feat/devices branch returns commands in the heartbeat answer (for example {"schedule_version": 7, "commands": [{"id": "...", "kind": "reload"}]}). The engine executes them. Shell level commands that the engine cannot do alone map to bridge methods: reload calls freebee.reload() if a hard reload is wanted; restart_app (phase 2) calls freebee.restartApp(). The shell never polls the backend itself; one network client per device keeps the token in one place.

9. Camera counting on the device

What runs where: the vision module stays in the web engine (web/lib/player/vision.ts, @vladmandic/face-api, tiny face detector plus age and gender, weights self hosted under /public/models). The WebView runs it against getUserMedia. The shell's only jobs are to grant the permission without a prompt and to keep the camera from being taken by another app.

Platform notes:

• Android WebView supports getUserMedia since Android 5; the box must expose the camera (built in on tablets; USB UVC on signage boxes, see 3.1). onPermissionRequest grants RESOURCE_VIDEO_CAPTURE only when the origin matches the configured player URL.
• Electron: session.setPermissionRequestHandler grants media (video only) to the player origin; setPermissionCheckHandler the same. On macOS the OS prompt appears once per app bundle; the installer runbook accepts it.
• Performance: face-api at 2.5 fps at 640 px needs about 15 percent of one core on an N100 and runs on an RK3568 at reduced fps; the RK3588 is comfortable. A box below RK3566 class should run with vision off.

Privacy (CLAUDE.md rule 3): frames are processed in memory and never written to disk or sent. The shell adds no capture of its own; capturePage in the desktop shell is used once, by the developer, for the README screenshot and is not wired to any runtime path. The mall notice about counting (legal gap in flow.md) belongs with the legal slice, not here.

10. Security

• Device token: opaque, hashed at rest server side, stored only in the engine's localStorage inside app private storage. Never in the shell config in clear text (phase 2 mirrors it with safeStorage on Electron and EncryptedSharedPreferences on Android).
• No inbound ports. The device only makes outbound HTTPS calls to the web origin; the same origin proxy means it never needs the backend host. No adb over network in production (adb tcpip off), no remote desktop, no SSH.
• Origin pinning: the shell only grants camera and only loads the configured player origin; navigation to any other origin is cancelled in will-navigate (Electron) and shouldOverrideUrlLoading (Android). window.open is blocked.
• Signed updates: APK signature verified by Android itself, plus the manifest signature; desktop installers code signed; the update manifest served over HTTPS from the web origin only.
• Kiosk: device owner lock task on Android; dedicated non admin user plus auto restart on desktop. Physical: the box behind the wall or in a locked enclosure; the kiosk tablet in a locked case with the USB port covered.
• Logs: the rotating log holds no play records, no tokens (the shell redacts Authorization), and stays on the device; ops read it through the status overlay or a future "fetch log" command.

11. Testing plan

Bench rig (week 1): one Android box, one tablet, the Mac mini, a USB camera, a smart plug per device for remote power cuts, a wifi router the developer controls, a HDMI monitor. Backend and web run locally (uvicorn on 8116, next dev on 3116, see web/PLAYER.md) or against the Vercel and Railway deployments.

Tests, each with a pass rule written down before running:

1. Cold boot test: power on from cold; the pairing code or the loop is on screen within 60 s; no prompt, no desktop visible. Repeat 10 times.
2. Pairing test: pair from /ops/pair; the loop starts within 20 s of the claim; the fleet page shows the device and a last check-in under 30 s old.
3. Power pull test: pull the plug mid video; power back; the loop resumes within 60 s with the same token (no new pairing code); the play that was interrupted is either recorded with its real duration or not recorded, never duplicated. Repeat 20 times over a day using the smart plug on a schedule.
4. Wifi pull test: cut the wifi for 4 hours during play; the loop keeps playing from cache; the queue depth in the overlay climbs; wifi back; the queue drains to zero within 10 minutes; the ledger shows every play once, in order, with batch_seq gaps equal to zero.
5. Backend down test: stop the backend for 30 minutes with wifi up; same pass rule as 4.
6. Stall test: inject a hang (?debug=1 overlay has a "freeze" button in phase 2, or kill -STOP the renderer on desktop); the watchdog reloads within 90 s; the log says why.
7. Crash test: kill the shell process (adb shell am force-stop, kill -9); the OS watchdog relaunches within 30 s.
8. 24 hour soak: a full day at the bench; pass is zero manual touches, memory of the WebView process flat (check hourly from the log), no missed heartbeats longer than 2 minutes except during the deliberate wifi pull.
9. 7 day soak before the first mall install, same rules, with a schedule change and an engine deploy in the middle.
10. Update test: publish a new shell version to the manifest; it installs between 02:00 and 04:00; the version in the overlay changes; the token survives.
11. Camera test: vision on; a person walks past; watchers now updates on the fleet page; cover the camera; camera_present stays true, counts go to zero; unplug the USB camera; health reports camera_present: false within 2 check-ins.
12. Revoke test: revoke in ops; the screen shows a new pairing code within 30 s; the queue is still there (overlay); re pair; the queue drains.

Automation: tests 3, 4 and 8 run from a laptop script that toggles the smart plug and the router through their APIs and reads the fleet endpoint. The script lives in simulator/ next to the existing simulator (phase 2).

12. Hardware recommendations and bill of materials

Prices are rough Indian street prices in October 2026, before GST where the seller quotes that way, and must be checked before ordering. Buy one of each class first, test with the plan above, then buy for the mall.

Android box classes:

Desktop classes:

Cameras:

Per screen bill of materials (wall screen, Android RK3568 class):

The screen itself and the LED wall are the mall's or the partner's. For a kiosk tablet replace the first line with the tablet and an enclosure (about 15,000 plus 3,000).

Buy for the bench now: one RK3568 box, one RK3588 box, one Galaxy Tab A9, two C920 cameras, two smart plugs. About 50,000.

13. Build order and week by week plan (one developer, Android first)

Week 1: Android shell to first pairing.

• Install Android Studio and the SDK, open player-android/, build the debug APK, run on the bench box.
• Get the WebView loading the Vercel player, pairing code on screen, camera grant, immersive mode.
• Boot receiver and foreground watchdog running; cold boot test passing.
• Land the one line freebee.heartbeat() call in the engine (coordinate with feat/devices).

Week 2: Android kiosk and reliability.

• Device owner setup script and README verified on the box; lock task; status bar and keyguard off.
• Power pull and wifi pull tests passing with the current engine.
• Health bridge fields; engine sends them once feat/devices has the heartbeat fields.
• Signed release APK; a player/updates/android.json manifest endpoint in the web app; nightly update check.

Week 3: Desktop shell to production grade.

• player-desktop/ (already runs): add electron-updater, the safeStorage token mirror, health() fields, the update window logic.
• Windows: build the NSIS installer on a Windows machine or in GitHub Actions (electron-builder --win cross builds from Mac for NSIS without signing), test the scheduled task and auto log on on the mini PC.
• Mac: launch agent and pmset; DMG; decide on signing.

Week 4: soak and the first install.

• 24 hour soak on the bench for Android and Windows; fix what it finds.
• Update test and revoke test.
• 7 day soak starts; the first mall install at the end of the week with the RK3568 box and a C920.

Week 5 and 6: hardening from the field.

• Service worker offline boot with feat/devices.
• Remote commands wired to the bridge.
• Log fetch command; fleet page shows shell version and health.
• Managed Play evaluation; GitHub Actions builds for all three shells.

14. File layouts

player-desktop/                 Electron shell (Mac, Windows)
  package.json                  scripts: start, build, build:mac, build:win
  src/main.ts                   window, kiosk, watchdog, permissions, log
  src/preload.ts                window.freebee bridge
  src/config.ts                 JSON config at userData/config.json, env, --url
  src/log.ts                    rotating log at userData/logs/
  scripts/install-autostart-mac.sh
  scripts/install-autostart-windows.ps1
  docs/first-run.png            screenshot from the first run on the bench Mac
  README.md

player-android/                 Android shell (Kotlin, uncompiled here)
  settings.gradle.kts, build.gradle.kts, gradle.properties
  app/build.gradle.kts
  app/src/main/AndroidManifest.xml
  app/src/main/java/in/freebee/player/
    MainActivity.kt             WebView, fullscreen, camera grant, lock task
    FreebeeBridge.kt            window.freebee
    BootReceiver.kt             starts the watchdog on boot
    WatchdogService.kt          foreground service, relaunches the activity
    AdminReceiver.kt            device owner receiver
    PlayerConfig.kt             player URL from SharedPreferences or BuildConfig
  app/src/main/res/             strings, device admin policy, icons
  README.md                     build steps, device owner adb commands

15. Open questions for the founder

1. Hardware: approve buying the bench kit (about 50,000) now, and the per screen BOM (about 17,500 to 23,500) for the first mall. Which Android box class for the first install: RK3568 at 9,000 or RK3588 at 18,000?
2. Google Play Console: open a developer account (one time USD 25) now, so managed Play is an option in month 2, or stay on side load through the pilot?
3. Signing: an Apple Developer Program membership (USD 99 per year) is needed for a Mac DMG that opens without a Gatekeeper warning and for Mac auto update; a Windows OV code signing certificate (about 25,000 to 35,000 per year) removes the SmartScreen warning. Both can wait if the Mac stays a test rig and Windows installs are done by hand.
4. Android keystore: who holds the release signing key and where is it backed up? Losing it means every box must be re installed by hand.
5. Connectivity: do we rely on mall wifi or put a 4G router per screen (300 per month per screen)?
6. Vision on the wall box: is counting on the RK3568 at reduced accuracy acceptable for the pilot, or do we spend the extra 9,000 per screen for the RK3588?
7. Who does the install in the mall, and is a printed one page runbook (plug in, wait for the code, call ops, read the code) enough?
8. Quiet hour for updates: is 02:00 to 04:00 local fine for every mall, or do some malls power the screens down at night (then updates happen at the next boot)?