Track OpenBot tasks, capability use, and result consumption by davidmckayv · Pull Request #972 · CopilotKit/OpenBot

davidmckayv · 2026-11-03T18:36:10Z

What this changes

Adds the requested OpenBot tracking through the existing @copilotkit/shared sender → CopilotKit telemetry sink → PostHog: scoped actor/deployment and run context, observed outcomes, capability/friction signals, foreground result views, and successful answer copies. Also fixes Google/xAI OAuth and keyless-compatible provider tracking.

No new Postgres tables, migrations, telemetry writes, identity store, event ledger, outbox, replay, signing service or health framework. Existing deployment configuration and privacy key provide metadata; PostHog queries handle deduplication and first observed success. Existing opt-outs apply; richer self-hosted collection remains opt-in.

Contract and coverage · PostHog query examples

Where it runs

Persistent state: none added; best-effort emission through the existing transport.
Second replica: derives metadata from existing configuration and emits independently. Reporting deduplicates stable event IDs in PostHog.
Serialization: no new locks or exactly-once producer guarantees; first use is computed from observed successful human root runs.
Browser fanout: existing rendered results emit bounded observations through authenticated API ingress; PostHog joins message IDs to terminal result_message_ids.
Listener/port/schedule: no new port, scheduler, replay loop or retention job.

Boundary and audit

Acting-call permission and audit boundaries remain intact.
Closed projections use existing audit outcomes and exclude raw audit payloads, prompts, answers, credentials, tool arguments/results and customer content.
Actor metadata is server-derived; browser identifiers are observations, not execution-success or authorization claims.

Changelog

Unreleased entry reflects the final implementation.

Proof

Corrected server: 23 focused tests, typecheck, scoped Biome and diff checks passed.
Corrected UI: 15 component tests, fresh app typecheck and production build passed. Coverage includes actual SDK clipboard success, foreground/settled visibility, and one best-effort attempt without retries.
One bounded correction review verified zero net schema/migration/database changes and removal of persistence/signing/replay machinery.
Unchanged native provider fix retains its 68 TypeScript tests, 18 native tests, builds and Mac/Windows/Linux CI evidence.
CI is running on the corrected revision; earlier results for the removed architecture do not establish this revision's status.

Coverage and delivery limits

PostHog receipt and query-count validation remain unverified. Earlier investigation found a growing shared queue and Reo HTTP 429 failures; no infrastructure was changed. Shared sender completion is not destination acknowledgement.

Unknown linkage remains unknown; a stop request is distinct from terminal cancellation. View/copy does not imply usefulness. Activity-only outputs, nested code/table copies, native-mobile/external-channel consumption and notification result opens are not claimed. Intelligence context validation/export, installed dashboards and acquisition handoffs remain external dependencies within the agreed single-OpenBot-PR scope.

…bot-telemetry

guidovizoso

Review notes below. The most important are the telemetry context reaching downstream agents (product.ts:401), events being dropped once 16 sends are in flight (product.ts:296), and the unhandled rejection from the async audit tap (index.ts:321).

guidovizoso · 2026-11-03T20:02:22Z

+          `accepted:${context.user_id}:${input.threadId}:${input.runId}`,
+          { ...context, thread_id: input.threadId },
+        );
+        return next.run(input).pipe(


The middleware passes input to next.run with forwardedProps.openbotTelemetry still attached. A remote agent (CloningRemoteAgent → HttpAgent) then receives the hashed user_id, deployment_id, root_task_id and parent_run_id over HTTP. On headless turns the same context is also stored with the run record on the Intelligence platform via RUN_STARTED.input. Remove the key before calling next.run.

guidovizoso · 2026-11-03T20:02:22Z

+    properties: Record<string, unknown>,
+  ): Promise<void> {
+    if (!enabled) return Promise.resolve();
+    if (pending.size >= 16) {


Every event type shares this one limit of 16 sends in flight, /api/telemetry/product-events has no rate limit, and events over the limit are dropped with only a log counter. A client posting result_viewed in a loop, or just a busy multi-user deployment, pushes out run_accepted, run_terminal and capability events, which skews success rates. Consider giving run and outcome events their own limit, and rate-limiting the public route.

guidovizoso · 2026-11-03T20:02:22Z

-        ? { hooks: { onBeforeHandler: trajectoryCapture.gate } }
-        : {}),
+      hooks: {
+        onBeforeHandler: async (context) => {


productTelemetry is always passed in, so this hook rewrites every chat request even when export is off (the default outside desktop). Each run copies the request, parses the full body (history and attachments), calls identifyActor and builds a new body, all for nothing. Check productTelemetry.enabled() first.

guidovizoso · 2026-11-03T20:02:22Z

+  privacyKey: config.keyEncryptionKey,
+  deploymentId: config.deploymentId,
+});
+addAuditTap(productTelemetry.recordAudit);


recordAudit is async, but AuditTap returns void, so the sync try/catch in createAuditStore never sees its rejections. If anything inside throws (an unexpected payload in projectAuditMilestone, or user() failing), the result is an unhandled rejection. Bun may treat that as fatal, and it breaks the rule that 'a tap that throws is logged and ignored'. Wrap it with .catch(...), or let the tap type return a promise and handle it there.

guidovizoso · 2026-11-03T20:02:22Z

+        .insert(auditEvents)
+        .values(row)
+        .returning({ id: auditEvents.id, createdAt: auditEvents.createdAt });
      for (const tap of auditTaps) {


Taps now send product telemetry from inside the caller's transaction, before it commits. Example: approval.person_decided is written via createAuditStore(tx). If the later workItems insert fails and the transaction rolls back, the 'capability approval completed/rejected' event has already gone to the external sink. Send after commit, or queue the event on the transaction.

guidovizoso · 2026-11-03T20:02:22Z

+      runId: input.runId,
+      trigger:
+        lineage?.trigger ??
+        (turn.continuation ? "followup" : triggerForInitiator(turn.initiator)),


The trigger comes from hostTurn.initiator, the value the caller passed in, not from the initiator createTurnRunner actually falls back to ({kind:'routine', id: routineId}). A headless routine turn with no initiator and no lineage is reported as trigger unknown with root_task_id and is_root both null, so it drops out of the root-task and routine metrics.

guidovizoso · 2026-11-03T20:02:22Z

+        kind,
+        threadId: reference.threadId,
+        messageId: reference.messageId,
+        surface: "web",


surface is always 'web', even in the desktop build, while the server tags the same runs desktop via OPENBOT_DISTRIBUTION. Every desktop view and copy of a result is counted as web, which breaks any breakdown of usage by surface. Detect the Tauri runtime here, or let the server fill this in.

guidovizoso · 2026-11-03T20:02:22Z

-                forwardedProps: { openbotRun: assertion },
-              },
+              input: await (
+                options.prepareRunInput ?? (async (input) => input)


prepareRunInput is awaited here with no guard, while run-turn catches the same failure and keeps going (headless_context_failed). An HMAC or key error therefore fails the Bot-to-Bot handoff and makes the work item retry. The hook is also typed Promise<unknown>, so a hook that returns the wrong shape compiles and goes straight to runner.run. Catch the error and fall back to the original input, and give the hook a proper return type.

guidovizoso · 2026-11-03T20:02:23Z

+                    !message.subagentRunId &&
+                    typeof message.id === "string" &&
+                    boundedId.safeParse(message.id).success &&
+                    !input.messages.some((old) => old.id === message.id)


For each assistant message in a MESSAGES_SNAPSHOT, input.messages.some(...) scans the whole input list. Cost grows with snapshot size × history size, on the event-stream hot path. Build a Set of input message ids once per run instead.

guidovizoso · 2026-11-03T20:02:23Z

+  deploymentId: string,
+  actorUserId: string,
+) {
+  return `ob_${createHmac("sha256", key).update(`openbot-product\0${deploymentId}\0${actorUserId}`).digest("hex")}`;


Two things here:

User ids are HMACed directly with the deployment's master key-encryption key, so one secret serves two purposes. Rotating the KEK quietly changes every analytics identity. Derive a separate key (for example with HKDF and a 'product-telemetry' label).

createdAt from the audit row is passed through but not used for occurred_at, which is the send time, so delayed audit events get the wrong timestamp.

Each read a field through `?.` and asserted it was undefined, so it passed whether or not the event or Activity entry existed. The assertion-shape guard refuses that shape, which kept CI red on this branch and on main since #972. Each now fails when the event or entry is missing, then checks the field directly.

…utcome (#978) Two tests from #972 asserted `f.sent[0]?.properties?.outcome` was undefined, which also passes when no event was sent at all. The assertion-shape guard rejects that form, and main's tests job has failed on it since #972 landed. Each test now proves the event's properties exist, then that they hold no outcome.

main 把 createTurnRunner 改成先收 hostTurn 再解構(CopilotKit#972 telemetry);保留這個寫法,加回 historyFilter。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

davidmckayv added 9 commits November 3, 2026 10:19

feat: add scoped OpenBot product telemetry

aa72902

Merge remote-tracking branch 'origin/main' into david/2026-11-03/open…

6e95c42

…bot-telemetry

feat: record visible results and successful answer copies

168a459

fix(desktop): cover all provider telemetry paths

5c5a63f

fix: preserve observed connection verification outcomes

215a0ea

chore: format product telemetry migration metadata

eb17630

test: close product telemetry database pool

821e471

simplify OpenBot telemetry to the existing shared sink

5339e24

fix: keep result telemetry forwarding stateless

f6ad10b

davidmckayv marked this pull request as ready for review November 3, 2026 19:23

davidmckayv requested review from MikeRyanDev, dustywusty, guidovizoso, mxmzb, samjulien and tylerslaton as code owners November 3, 2026 19:23

guidovizoso reviewed Nov 3, 2026

View reviewed changes

davidmckayv merged commit 6bfee78 into main Nov 3, 2026
23 of 25 checks passed

davidmckayv deleted the david/2026-11-03/openbot-telemetry branch November 3, 2026 20:08

davidmckayv mentioned this pull request Nov 3, 2026

Check that a telemetry event was sent before checking it carries no outcome #978

Merged

davidmckayv mentioned this pull request Nov 4, 2026

Send OpenBot's run context to Intelligence under the shared contract key #992

Merged

Navigation Menu

Repository navigation

Track OpenBot tasks, capability use, and result consumption - #972

Track OpenBot tasks, capability use, and result consumption#972
davidmckayv merged 9 commits into
mainfrom
david/2026-11-03/openbot-telemetry

davidmckayv commented Nov 3, 2026 •

edited

Loading

Uh oh!

guidovizoso left a comment

Uh oh!

guidovizoso Nov 3, 2026

Uh oh!

guidovizoso Nov 3, 2026

Uh oh!

guidovizoso Nov 3, 2026

Uh oh!

guidovizoso Nov 3, 2026

Uh oh!

guidovizoso Nov 3, 2026

Uh oh!

guidovizoso Nov 3, 2026

Uh oh!

guidovizoso Nov 3, 2026

Uh oh!

guidovizoso Nov 3, 2026

Uh oh!

guidovizoso Nov 3, 2026

Uh oh!

guidovizoso Nov 3, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

davidmckayv commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this changes

Where it runs

Boundary and audit

Changelog

Proof

Coverage and delivery limits

Uh oh!

guidovizoso left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

davidmckayv commented Nov 3, 2026 •

edited

Loading