Skip to content

Keep an attached file as a file after a Bot on a harness answers - #1499

Open
davidmckayv wants to merge 2 commits into
mainfrom
fix/attachment-raw-text-after-reply
Open

davidmckayv wants to merge 2 commits into
mainfrom
fix/attachment-raw-text-after-reply

Conversation

@davidmckayv

Copy link
Copy Markdown
Contributor

What this changes

On a Bot OpenBot runs in a harness (the Personal Assistant on Desktop's LangChain harness, and any remote AG-UI Bot whose integration ends its run with a MESSAGES_SNAPSHOT), a file a person attached turned into the file's raw contents the moment the answer finished: a text file became one giant block of text, an image's thumbnail disappeared, and a reload showed the same.

Cause: the remote middleware in server/src/copilot.ts inlines each attachment's contents on the way to the endpoint (inlineAttachments). ag-ui-langgraph ends every run with a MESSAGES_SNAPSHOT of the thread as it was sent to it, so the person's message came back holding Attached file "x.txt": ... (or an image's base64) instead of the /api/attachments/<id> reference. The browser replaced its messages with that snapshot, and Intelligence stored it as the thread.

Fix: the remote middleware keeps each person's message that carried a file, as sent, and puts it back by id in every MESSAGES_SNAPSHOT the endpoint emits (withAttachmentsAsSent, applied last over every step's stream). The model is still sent the file's contents, and later turns inline it again from the reference as they already do for history. Built-in Bots never emit a snapshot and were not affected (checked in the browser).

Messages already stored as text in an existing conversation stay as they are.

Where it runs

  • New state that outlives a request? None. The as-sent copy lives for one run's stream.
  • What happens on the second replica? Nothing different: the rewrite happens inside the run's own event stream, before the runner stores it.
  • Anything serialised? No.
  • Anything fanned out to a browser? The same snapshot event as before, with the person's message as sent.
  • New listener, port, or schedule? None.

Boundary and audit

  • Every acting call still goes through the gateway: resolve, decide, audit, then act.
  • New refusals and new failures each write a row. (No new refusals or failures.)
  • Nothing new is trusted from the client that the server can resolve itself. The restored content is the content the same request carried in.

Changelog

  • A line in CHANGELOG.md under Unreleased, Conversations.

Proof

  • New regression test server/tests/attachment-snapshot.test.ts: a remote Bot built through buildAgents, dialling a real local AG-UI endpoint over HTTP that ends its run with a snapshot of the messages it was sent, as ag-ui-langgraph does. It checks the model still receives the text file's contents and the image's bytes, and that the emitted snapshot, and the messages the client holds after applying it, keep the person's message exactly as sent (text file, image, files earlier in the thread, plain messages untouched). On origin/main: 3 fail, 1 pass. With the fix: 4 pass.
  • server/tests/copilot.test.ts, attachment-parts, attachment-routes, headless-tools, learning-remote, bot-knows-its-model: all pass. tsc --noEmit (server), typecheck:tests and biome clean.
  • Real product on 127.0.0.1: OpenBot server and app from this branch, a local Intelligence, and the LangGraph harness (agent-langgraph-agui) as the Personal Assistant, the way Desktop runs it. Headless Chromium attached a text file and an image, watched the message during and after the answer, then reloaded. Before: the text message became the file's contents after the answer and after reload, and the image vanished. After: both stay as file tiles during, after and after reload, in light and dark, including a second file later in the same conversation; the stored thread holds the /api/attachments/<id> references and the Bot's answer shows it read the whole file. A PDF is still refused by the composer, unchanged.

A Bot on an endpoint (the LangGraph harness Desktop runs, among others)
ends its run with a MESSAGES_SNAPSHOT of the thread as it was sent to it.
The server inlines a file's contents on the way to the endpoint, so the
snapshot carried the file's text, or an image's base64, where the
person's reference to the file was. The browser applied it and
Intelligence stored it: a text file became one giant block of its own
contents when the answer finished, an image's thumbnail vanished, and a
reload showed the same.

The remote middleware now puts each person's message that carried a file
back as they sent it in every snapshot the endpoint emits. The model is
still sent the file's contents.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant