feat: the VDK, Visor Development Kit

CLAUDE.md

@@ -21,7 +21,8 @@ the frame style and language packs. Works on real glasses as of 2026-10-05.
 ## Working on it
 
 - `reference/lensbreak`, a snapshot of the desktop Python client, already does most of what
-  this app needs. Look there first; the protocol write-ups are on the GitHub wiki.
+  this app needs. Look there first; the protocol write-ups are in `../visor-site/src/content/protocol`
+  (visor.macedonga.com/protocol), the only copy. The wiki is retired.
 - The glasses are the test. Builds get tried on hardware, so there are no test suites.
 - Keep it small: short comments, no planning documents.
 - No backend, analytics or crash reporting.

README.md

@@ -4,7 +4,7 @@
 
 <p align="center">
   <b>An Android app for Meta glasses that replaces the Meta AI app.</b><br>
-  Your photos stay on your phone, and "Hey Meta" answers with the model you choose.
+  Your photos stay on your phone, "Hey Meta" answers with the model you choose, and your apps use the glasses too.
 </p>
 
 <p align="center">
@@ -16,6 +16,7 @@
 <p align="center">
   <a href="https://visor.macedonga.com">Website</a> |
   <a href="#what-it-does">Features</a> |
+  <a href="#apps-for-your-glasses">Apps</a> |
   <a href="#install">Install</a> |
   <a href="#what-it-sends-and-where">Privacy</a> |
   <a href="#good-to-know">Good to know</a> |
@@ -39,10 +40,23 @@ Visor works with Ray-Ban Meta, Oakley Meta and the other Meta glasses. It speaks
 | **Notifications** | Message notifications go to the glasses, which announce the sender by the name in your contacts. |
 | **Settings** | Every setting the glasses have, the way the Meta app shows them. Advanced holds the rest, assistant language packs and the firmware's feature flags. |
 | **Outside the app** | A home screen widget, a Quick Settings tile, launcher shortcuts, and a Live Update notification while the Live view runs. |
-| **Developer mode** | Other apps and automations can take photos and videos, import captures and hear about new ones. See [Developer mode](https://visor.macedonga.com/developers/docs/developer-mode/). |
 | **More than one pair** | Keep several pairs on one phone. Home swipes between them, each with its name, batteries and a Connect button, and ends with a page to set up another. |
-| **Meta Ray-Ban Display** | A teleprompter, live captions and translation, the Neural Band's gestures, the Briefing panel and saved Wi-Fi networks. The glasses' own apps reach the internet through the phone, with Instagram's Reels and messages among them. |
-| **Neural Band** | Set up from the Display's settings, with its battery on Home and its wrist in Settings. |
+| **Meta Ray-Ban Display** | A teleprompter, live captions and translation, the Neural Band's gestures, the Briefing panel with your apps' own cards, and saved Wi-Fi networks. The glasses' own apps reach the internet through the phone, with Instagram's Reels and messages among them. Web apps get the phone's location, motion sensors and nearby Bluetooth devices, and can give your assistant tools. |
+| **Neural Band** | Set up from the Display's settings, with its battery on Home and its wrist in Settings. Share it with the phone, and apps buzz it at once and read its gestures and motion while the Display keeps using it. |
+
+## Apps for your glasses
+
+Visor is also where your apps meet the glasses. Through the [Visor Development Kit](https://visor.macedonga.com/developers/)
+(VDK), apps on your phone, programs and AI agents on your computer, and web apps on the Display use the camera and
+captures, the Display's layouts and Briefing cards, the Neural Band's gestures and motion, voice, the assistant's
+tools and the glasses' settings. No Meta account, no Meta AI app, no developer review.
+
+| To | Do this |
+|---|---|
+| **Use an app** | It asks for what it needs the first time, on Visor's consent screen. **Settings, Apps** lists every app, program and web app with access, and takes any of it back. |
+| **Connect a program or an AI agent** | In **Settings, Apps**, turn on **Let programs connect** and add the program, with its own token. Agents get the glasses over MCP. |
+| **Build your own** | For Android, Python, TypeScript, or a web app on the Display: start with the [getting started guide](https://visor.macedonga.com/developers/docs/getting-started/). `visor fake` stands in for the glasses while you build. |
+| **Automate** | Automation apps such as Tasker take photos and videos and hear about new captures through [Developer mode](https://visor.macedonga.com/developers/docs/developer-mode/)'s intents. |
 
 ## Requirements
 

app/build.gradle.kts

@@ -73,6 +73,9 @@ dependencies {
     implementation(libs.androidx.lifecycle.viewmodel.compose)
     implementation(libs.androidx.compose.material3)
     implementation(libs.androidx.browser)
+    implementation(libs.visor.kit)
+    implementation(libs.ktor.server.cio)
+    implementation(libs.ktor.server.websockets)
     implementation(libs.androidx.media)
     implementation(libs.kotlinx.coroutines.android)
 

app/src/debug/kotlin/com/macedonga/visor/debug/AppLinkProbe.kt

@@ -0,0 +1,149 @@
+package com.macedonga.visor.debug
+
+import android.content.Context
+import com.macedonga.visor.core.identity.BundleCredentials
+import com.macedonga.visor.core.identity.BundleState
+import com.macedonga.visor.core.identity.IdentityBundle
+import com.macedonga.visor.core.identity.nowIso
+import com.macedonga.visor.core.protocol.Dat
+import com.macedonga.visor.core.session.DataXError
+import com.macedonga.visor.core.session.GlassesSession
+import com.macedonga.visor.core.session.LinkState
+import com.macedonga.visor.core.session.LinkStateChanged
+import com.macedonga.visor.core.session.ProtocolError
+import com.macedonga.visor.core.session.ServiceData
+import com.macedonga.visor.core.session.SessionCommand
+import com.macedonga.visor.core.transport.BluetoothLeTransport
+import com.macedonga.visor.log.EventLog
+import java.security.PrivateKey
+import java.util.Locale
+import java.util.UUID
+import java.util.concurrent.atomic.AtomicInteger
+import kotlinx.coroutines.CoroutineScope
+import kotlinx.coroutines.Dispatchers
+import kotlinx.coroutines.Job
+import kotlinx.coroutines.SupervisorJob
+import kotlinx.coroutines.delay
+import kotlinx.coroutines.launch
+import kotlinx.coroutines.runBlocking
+
+/**
+ * Debug only: a link of Visor's own to the Display under its app key, trusted on svc79 against the constellation
+ * manifest as Meta's toolkit's links are, and a toolkit session on it (svc7009, then the toolkit app on svc6500).
+ * Every step goes to the event log as "AppLink:".
+ */
+internal object AppLinkProbe {
+    private val scope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
+    private var session: GlassesSession? = null
+    private var job: Job? = null
+    private val ids = AtomicInteger()
+    private val samples = AtomicInteger()
+    private val bandSamples = AtomicInteger()
+
+    fun start(context: Context, address: String, pem: ByteArray, version: Long, salt: ByteArray, tag: Int, sign: Int, dam: Boolean, log: EventLog): String {
+        val credentials = BundleCredentials(IdentityBundle(pem, address, "visor-app", nowIso(), "app", BundleState.VERIFIED))
+        return start(context, address, credentials.appEcKey, credentials.appPublicKey, version, salt, tag, sign, dam, log)
+    }
+
+    fun start(context: Context, address: String, key: PrivateKey, publicKey: ByteArray, version: Long, salt: ByteArray, tag: Int, sign: Int, dam: Boolean, log: EventLog): String {
+        stop()
+        val trust = ProbeConstellationTrust(key, publicKey, version, salt, tag, sign) { log.add("AppLink: $it") }
+        val opened = GlassesSession(BluetoothLeTransport(context, address), trust)
+        session = opened
+        job = scope.launch {
+            opened.events.collect { event ->
+                when (event) {
+                    is LinkStateChanged -> {
+                        log.add("AppLink: link ${event.state} ${event.reason}".trim())
+                        if (event.state == LinkState.READY) {
+                            log.add("AppLink: trust ${opened.trustResult}")
+                            send(Dat.SESSION_SERVICE, Dat.SESSION_START_REQUEST, Dat.sessionStart("com.macedonga.visor", UUID.randomUUID().toString(), "Visor", "1.0.1", dam))
+                        }
+                    }
+                    is ProtocolError -> log.add("AppLink: ${event.detail}")
+                    is DataXError -> {
+                        log.add("AppLink: svc${event.serviceId} status 0x%04x".format(Locale.ROOT, event.errorId))
+                        // A refused channel is dead: the next request opens a fresh one.
+                        if (event.serviceId == Dat.DWA_SERVICE) dwaChannel = null
+                    }
+                    is ServiceData -> heard(event, log)
+                    else -> Unit
+                }
+            }
+        }
+        opened.start(scope)
+        return "connecting to $address with manifest $version"
+    }
+
+    /** Adds a capability on the toolkit app, then sends its first request: motion at [rateHz], or a test layout. */
+    fun capability(capability: Int, payload: ByteArray?): String {
+        val now = System.currentTimeMillis()
+        send(Dat.DWA_SERVICE, Dat.REQUEST, Dat.requests("add-${ids.incrementAndGet()}", capability, Dat.addCapability(capability), null, now).single())
+        if (payload != null) {
+            Dat.requests("req-${ids.incrementAndGet()}", capability, null, payload, now).forEach { send(Dat.DWA_SERVICE, Dat.REQUEST, it) }
+        }
+        return "sent to capability $capability"
+    }
+
+    fun stop(): String {
+        val running = session ?: return "no app link"
+        runBlocking { running.stop() }
+        job?.cancel()
+        session = null
+        return "stopped; ${samples.getAndSet(0)} motion samples"
+    }
+
+    @Volatile
+    private var dwaChannel: Int? = null
+
+    private fun send(service: Int, type: Int, payload: ByteArray) {
+        session?.submit(
+            SessionCommand { s ->
+                val connection = s.connection ?: return@SessionCommand
+                if (service != Dat.DWA_SERVICE) return@SessionCommand connection.sendOnService(service, type, payload)
+                // The toolkit app's own channel, opened once it is up, so a refusal before then does not stick. Only the
+                // opening frame names the service: one that names it again reads as a new open.
+                val open = dwaChannel
+                if (open != null) return@SessionCommand connection.send(open, payload, messageType = type)
+                val channel = connection.openChannel(service).also { dwaChannel = it }
+                connection.send(channel, payload, messageType = type, serviceId = service)
+            },
+        )
+    }
+
+    private fun heard(event: ServiceData, log: EventLog) {
+        when (event.serviceId) {
+            Dat.SESSION_SERVICE -> if (event.messageType == Dat.SESSION_START_RESPONSE) {
+                val started = Dat.sessionStarted(event.payload)
+                log.add("AppLink: session status ${started.status}, toolkit app ${started.dwaVersion}, launch ${started.dwaLaunch}, compatibility ${started.compatibility}")
+                // The toolkit app needs about a second to come up after the launch before it hears svc6500.
+                if (started.status == Dat.START_OK && started.dwaLaunch == 1) scope.launch {
+                    delay(2_000)
+                    log.add("AppLink: ${capability(Dat.IMU, Dat.motionStart(10))}")
+                }
+            } else {
+                log.add("AppLink: svc7009 type ${event.messageType} ${event.payload.toHexString()}")
+            }
+            Dat.DWA_SERVICE -> when (event.messageType) {
+                Dat.RESPONSE -> Dat.response(event.payload).let {
+                    log.add("AppLink: response ${it.requestId} capability ${it.capability} status ${it.status} active ${it.active} error ${it.error} denied ${it.denied}")
+                }
+                Dat.EVENT -> Dat.event(event.payload).let { e ->
+                    val motion = if (e.capability == Dat.IMU) e.payload?.let(Dat::motionSamples).orEmpty() else emptyList()
+                    val n = samples.addAndGet(motion.size)
+                    bandSamples.addAndGet(motion.count { it.band })
+                    when {
+                        motion.isNotEmpty() -> if (n % 50 < motion.size) {
+                            val a = motion.last().accelerometer
+                            log.add("AppLink: $n motion samples (${bandSamples.get()} from the band), last accelerometer ${a?.x} ${a?.y} ${a?.z}, orientation ${motion.last().orientation != null}")
+                        }
+                        e.heartbeat -> Unit
+                        else -> log.add("AppLink: event capability ${e.capability} shutdown ${e.shutdown} ${e.payload?.toHexString()}")
+                    }
+                }
+                else -> log.add("AppLink: svc6500 type ${event.messageType} ${event.payload.toHexString()}")
+            }
+            else -> log.add("AppLink: svc${event.serviceId} type ${event.messageType} ${event.payload.size} bytes")
+        }
+    }
+}