Skip to content

Implement workflow completion monitoring with extensible hooks#21532

Merged
jmchilton merged 28 commits intogalaxyproject:devfrom
mvdbeek:claude/plan-workflow-completion-8L7O9
Jan 27, 2026
Merged

Implement workflow completion monitoring with extensible hooks#21532
jmchilton merged 28 commits intogalaxyproject:devfrom
mvdbeek:claude/plan-workflow-completion-8L7O9

Conversation

@mvdbeek
Copy link
Copy Markdown
Member

@mvdbeek mvdbeek commented Jan 4, 2026

User facing changes

This PR adds the ability to configure actions that automatically execute when a workflow invocation completes. Users can now:

  • Export results to a remote file source - Automatically export the invocation outputs to a configured file source (e.g., Dropbox, S3) when the workflow finishes
Screenshot 2026-01-21 at 12 38 53
  • Send a notification - Receive a notification when the workflow completes
Screenshot 2026-01-21 at 12 39 28
  • Inspect recent history and invocation exports
Screenshot 2026-01-21 at 12 39 42
  • Chose a history name when selecting "Send results to a new history"
Screenshot 2026-01-19 at 10 15 47

The workflow run form includes a new "On Completion" section where users can select and configure these actions before launching a workflow.

Workflow Completion Monitor

The system uses a Celery-based polling mechanism to detect workflow completion:

  1. WorkflowCompletionMonitor (lib/galaxy/workflow/completion_monitor.py) - A background task that periodically queries for invocations where all jobs have reached terminal states
  2. WorkflowInvocationCompletion - A new database table that tracks completion status and prevents duplicate hook execution
  3. Invocations gain a new completed state (distinct from scheduled) indicating all jobs finished

The monitor is limited to run only on the invocation handler to avoid contention in multi-process deployments.

Extensible Hook System

Hooks are registered in WorkflowCompletionHookRegistry (lib/galaxy/workflow/completion_hooks/init.py). Each hook extends WorkflowCompletionHook and implements:

  • validate_config() - Validate user-provided configuration
  • execute() - Perform the completion action

Built-in hooks:

  • export_to_file_source - Exports invocation to a file source using the model store system
  • send_notification - Sends a user notification via Galaxy's notification system

/api/exports Endpoint

New API endpoint for listing recent user exports:

  • GET /api/exports?limit=50&days=30 - Returns recent exports to remote file sources
  • Tracks both preparing (in-progress) and ready (completed) exports
  • Used by the frontend to show export status in the Downloads panel

GCheckbox Component Refactoring

A new reusable GCheckbox component (client/src/components/BaseComponents/GCheckbox.vue) replaces BFormCheckbox usage in the workflow run form:

  • Supports v-model with Vue 2.7-style value/input events
  • Optional toggle prop renders a switch-style slider instead of a checkbox
  • Includes data-test-id attributes for Selenium testing
  • HelpText component gains an infoIcon prop for rendering help as an icon that doesn't interfere with click events

How to test the changes?

(Select all options that apply)

  • I've included appropriate automated tests.
  • This is a refactoring of components with existing test coverage.
  • Instructions for manual testing are as follows:
    1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

  • I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

Comment thread lib/galaxy/app.py Outdated
@mvdbeek
Copy link
Copy Markdown
Member Author

mvdbeek commented Jan 5, 2026

FAILED lib/galaxy_test/api/test_workflows.py::TestWorkflowsApi::test_run_workflow_subworkflow_conditional_step - AssertionError: Final state - completed - not okay.

😆 working too good

@mvdbeek mvdbeek force-pushed the claude/plan-workflow-completion-8L7O9 branch 6 times, most recently from d72e7db to b4ad649 Compare January 6, 2026 17:09
@mvdbeek
Copy link
Copy Markdown
Member Author

mvdbeek commented Jan 14, 2026

Here's how that look like currently:

export_notification.mov

@mvdbeek mvdbeek force-pushed the claude/plan-workflow-completion-8L7O9 branch 6 times, most recently from 376abc9 to 4eb5d62 Compare January 18, 2026 14:41
@mvdbeek mvdbeek marked this pull request as ready for review January 19, 2026 08:33
@github-actions github-actions Bot added this to the 26.0 milestone Jan 19, 2026
@mvdbeek mvdbeek added the highlight Included in user-facing release notes at the top label Jan 19, 2026
@mvdbeek mvdbeek requested a review from jmchilton January 19, 2026 18:13
@jmchilton
Copy link
Copy Markdown
Member

I mean the bottom right of this screenshot is kind of ugly - but I've opened many PRs with worse or no UI at all 😆. This isn't a blocker at all to me but I'm pointing it out if you want to clean it up. I'll review the backend again - give me a second.

Screenshot 2026-01-19 at 2 34 20 PM

Comment thread lib/galaxy/workflow/completion_hooks/notification.py Outdated
Comment thread lib/galaxy/workflow/completion_hooks/notification.py Outdated
Changes made:

1. ExportOnCompleteWizard.vue:
  - Replaced ExportOnCompleteConfig with WriteStoreToPayload from @/api/schema
  - Renamed format to model_store_format throughout
  - Added ModelStoreFormat type for proper typing of the ref
2. OnCompleteActions.vue:
  - Imported WriteStoreToPayload from the wizard
  - Removed local ExportConfig interface
  - Updated all references from format to model_store_format
3. WorkflowRunFormSimple.vue:
  - Imported WriteStoreToPayload from the wizard
  - Replaced inline type definition with WriteStoreToPayload
@mvdbeek mvdbeek force-pushed the claude/plan-workflow-completion-8L7O9 branch 3 times, most recently from 98ca035 to 1244d1b Compare January 21, 2026 11:48
@mvdbeek
Copy link
Copy Markdown
Member Author

mvdbeek commented Jan 21, 2026

Alright, I think I addressed all of your comments (hope i didn't forget anything ...), and I updated the screenshots with the new state.

The notification now simply reports that the workflow completed with a
job summary, letting users interpret success themselves.
Replace manual hook registration with automatic discovery using
Galaxy's plugin_config.plugins_dict() framework. Hooks are now
discovered via their plugin_type class attribute.
@mvdbeek mvdbeek force-pushed the claude/plan-workflow-completion-8L7O9 branch from b1e413f to 96eb733 Compare January 21, 2026 14:18
@mvdbeek mvdbeek requested a review from jmchilton January 25, 2026 13:18
@mvdbeek mvdbeek force-pushed the claude/plan-workflow-completion-8L7O9 branch 2 times, most recently from 0998996 to c3917c3 Compare January 26, 2026 16:26
@mvdbeek mvdbeek force-pushed the claude/plan-workflow-completion-8L7O9 branch from c3917c3 to 564cc8f Compare January 26, 2026 17:27
@jmchilton jmchilton merged commit e2d5a41 into galaxyproject:dev Jan 27, 2026
73 of 78 checks passed
@mvdbeek mvdbeek mentioned this pull request Jan 27, 2026
4 tasks
@guerler guerler added the release-testing-26.0 PRs marked for testing for the 26.0 release and issues stemming from release testing label Jan 29, 2026
guerler added a commit to guerler/galaxy that referenced this pull request Feb 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/API area/database Galaxy's database or data access layer area/testing/api area/testing/integration area/testing area/workflows highlight Included in user-facing release notes at the top kind/feature release-testing-26.0 PRs marked for testing for the 26.0 release and issues stemming from release testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants