Dashboard playground UI

[ianmacartney]

Visualize tool calling graphs.
Inspect and replay past conversations while tuning prompts.
Interactively debug failed generations.
Search messages while tuning search parameters for your usecase.
Replaying failed steps and exporting evals to prevent regressions once you get it working.

[ianmacartney]

more idea dump:

show by order top to bottom, with step order left to right

show graph based on parent ID (maybe only if non-linear?)

show input context

allow re-running with different input context?

allow re-running sub-steps? how to find the tool to call?

time travel replay of bad messages: surface/interactive playground?

log search history results

[ghandic]

This would be huge - are you open to community PR's for this or internally developed?

[nipunn1313]

@ianmacartney is out for a bit - he'll be back soon.

Def open to community PRs!
That being said, the larger the task/PR, the bar to get it in will be higher (make sure it's aligned with what get-convex/agent is trying to do). Ian can say more when he's back.

[ianmacartney]

@ghandic community PRs would be great. I started a prototype with ReactFlow, but then I realized that the vast majority of usecases will not have much branching / flow diagrams, so a typical chat UI would suffice.
Ideally with dropdowns for sub-tool calls, and sub-threads (often from agents used as tools).
and a side panel with the info I posed above.
This is on my radar but not at the top

[ghandic]

Any pointers on how to enable an extension to the dashboard based on if a component is present?

[ghandic]

Opened up a draft PR to work on, if you could check the scope and give any feedback as I go that would be great!

[ghandic]

Hit a roadblock - since its not a system table (as its a component) I'd need access to the component db to surface any data. Is there any prior work done on exposing functionality to the dashboard through components?

[ianmacartney]

Wow, thanks for jumping in! Sorry for not providing more guidance before.
At a high level, while the dashboard may eventually have some integration points for components to surface data & actions, for the time being, more complex UIs like the playground will need to be provided outside of the dashboard.

This is the architecture direction I'm thinking of:

1. The component has a utility function to export the functions for the public API for the playground, e.g. export const { listThreads, ... } = createDashboardApi(components.agent);. These functions expose all the functionality the playground needs. The internals are in src/component/dashboard.ts and the public API exporting is in src/client/dashboard.ts.
2. The playground is a standalone React app that uses the exported query/mutation/actions that you can run locally, or host. , The frontend is maybe run like npx @convex-dev/agent playground running vite under the hood, or just serves some static files (simple SPA).
3. In order to authenticate the playground functions, there's a function you can run on the CLI / Convex dashboard to generate an API key, and on the playground when you log in it asks for the API key. Each query/mutation internally validates the API key, so no one can access the data except developers who have permission to run the CLI / access the Convex dashboard.

[ianmacartney]

Ok here's where I landed:

Next up, see whether v0, chef, or ...Cursor can make a more functional UI to extricate