Add gateway-plugin support to generate image and video #1603

varungup90 · 2025-09-24T19:56:17Z

Pull Request Description

Add user APIs /v1/image/generations and /v1/video/generations.

Samples

curl -v "http://localhost:8888/v1/image/generations" \
	-H "Content-Type: application/json" \
	-d '{
	   "prompt": "a cute cat",
	   "model": "<model-name>"
	 }'

curl -v "http://localhost:8888/v1/video/generations" \
	-H "Content-Type: application/json" \
	-H "routing-strategy: random" \
	-d '{
	   "prompt": "dog chasing cat",
	   "model": "<model-name>",
	   "save_disk_path": "/tmp"
	 }'

Related Issues

Resolves: #[Insert issue number(s)]

Important: Before submitting, please complete the description above and review the checklist below.

Contribution Guidelines (Expand for Details)

We appreciate your contribution to aibrix! To ensure a smooth review process and maintain high code quality, please adhere to the following guidelines:

Pull Request Title Format

Your PR title should start with one of these prefixes to indicate the nature of the change:

[Bug]: Corrections to existing functionality
[CI]: Changes to build process or CI pipeline
[Docs]: Updates or additions to documentation
[API]: Modifications to aibrix's API or interface
[CLI]: Changes or additions to the Command Line Interface
[Misc]: For changes not covered above (use sparingly)

Note: For changes spanning multiple categories, use multiple prefixes in order of importance.

Submission Checklist

PR title includes appropriate prefix(es)
Changes are clearly explained in the PR description
New and existing tests pass successfully
Code adheres to project style and best practices
Documentation updated to reflect changes (if applicable)
Thorough testing completed, no regressions introduced

By submitting this PR, you confirm that you've read these guidelines and your changes align with the project's contribution standards.

gemini-code-assist · 2025-09-24T19:56:38Z

Summary of Changes

Hello @varungup90, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces comprehensive support for image and video generation functionalities within the existing gateway plugin. It updates external and internal routing configurations to accommodate new API endpoints, implements path rewriting for these requests, and refactors the response and request body processing to handle the specific requirements of image and video generation models. The changes aim to seamlessly integrate these new capabilities into the gateway's request flow.

Highlights

New API Endpoints for Image and Video Generation: Added /v1/image/generations and /v1/video/generations path prefixes to the gateway plugin configuration and HTTP routes, enabling external access to new image and video generation functionalities.
Internal Path Rewriting: Implemented logic within the gateway plugin to rewrite incoming external API paths (e.g., /v1/image/generations) to internal service paths (e.g., /generate) for image and video generation requests, facilitating proper routing to backend services.
Broader Gateway Routing Prefix: Updated the main gateway configuration to use a broader / prefix instead of /v1 for the original_route, potentially simplifying future API versioning and expanding routing capabilities for the gateway.
Refactored Response Body Processing: The response body handling in gateway_rsp_body.go has been refactored to introduce isLanguageRequest and processLanguageResponse functions, making the processing more modular and adaptable to different response types beyond just language models.
Request Body Validation for New Endpoints: Added specific request body validation for the new image and video generation endpoints, including unmarshalling the payload into openai.ImageGenerateParams to correctly extract model information.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request adds support for image and video generation endpoints to the gateway. The changes include adding new routes in the gateway configuration, rewriting paths in the gateway plugin, and handling the new request types.

My review has identified a critical issue in pkg/plugins/gateway/gateway_req_body.go where a path is unconditionally rewritten, which will break existing functionality. I've also noted a significant change in the gateway's path matching that could have unintended side effects and should be carefully considered. Additionally, there are a few opportunities for refactoring to improve code maintainability and future-proofing the design for video generation parameters. Please review the detailed comments.

pkg/plugins/gateway/gateway_req_body.go

gemini-code-assist · 2025-09-24T19:57:47Z

config/gateway/gateway.yaml

        name: original_route
        match:
-          prefix: "/v1"
+          prefix: "/"


Changing the prefix match from /v1 to / is a significant change that broadens the scope of this route considerably. It now matches all paths for requests that include a routing-strategy header. This could have unintended side effects on other routes that might not be intended to be handled by the original_destination_cluster. Please add a comment explaining why this change is necessary or consider if a more specific set of path prefixes could be used to avoid accidentally capturing unintended traffic.

the new added api endpoints all comes with v1. what's the reason to change from /v1 to /?

User requests are for /v1/image/generations and /v1/video/generations, but for xDit engine requests path are modified to /generate and /generatevideo respectively.

@happyandslow is trying to work with xdit team, if they can also use open ai protocol.

@happyandslow did you create any issues or talk to any person? any estimated DDL?

dist/chart/templates/gateway-instance/gateway.yaml

pkg/plugins/gateway/gateway_rsp_body.go

gemini-code-assist · 2025-09-24T19:57:47Z

pkg/plugins/gateway/util.go

+	case "/v1/image/generations", "/v1/video/generations":
+		imageGenerationObj := openai.ImageGenerateParams{}
+		if err := json.Unmarshal(requestBody, &imageGenerationObj); err != nil {
+			klog.ErrorS(err, "error to unmarshal image generations object", "requestID", requestID, "requestBody", string(requestBody))
+			errRes = buildErrorResponse(envoyTypePb.StatusCode_BadRequest, "error processing request body", HeaderErrorRequestBodyProcessing, "true")
+			return
+		}
+		model = imageGenerationObj.Model


Using the same openai.ImageGenerateParams struct for both /v1/image/generations and /v1/video/generations might be limiting if the video generation API has different parameters than the image generation one. For example, video generation might involve parameters like duration or frames per second, which are not present in ImageGenerateParams. If the APIs are expected to diverge, consider defining a separate struct for video generation to improve type safety and clarity.

right now, open ai does not have /v1/video/generations API, hence re-using the /v1/image/generations for now.

Jeffwan · 2025-09-24T20:20:29Z

pkg/controller/modelrouter/modelrouter_controller.go

+						{
+							Path: &gatewayv1.HTTPPathMatch{
+								Type:  ptr.To(gatewayv1.PathMatchPathPrefix),
+								Value: ptr.To("/generatevideo"),


what's the api sepc on the engine side now? is it finalized?

#1595 we can have the discussion in this issue

/cc @happyandslow

Jeffwan · 2025-09-24T20:22:48Z

pkg/plugins/gateway/gateway_rsp_body.go

-			code := envoyTypePb.StatusCode_InternalServerError
-			if res.Code >= 100 && res.Code < 600 {
-				code = envoyTypePb.StatusCode(res.Code)
+		if isLanguageRequest(routerCtx.ReqPath) {


is there any else logic?

Yes, there is no else right now, if request is off type image or video, then ignore response processing and return response to user.

Once we add more features for example support for async request then there will be else case.

Signed-off-by: varungupta <[email protected]>

Jeffwan · 2025-09-25T17:16:10Z

overall LGTM

varungup90 marked this pull request as draft September 24, 2025 19:56

gemini-code-assist bot reviewed Sep 24, 2025

View reviewed changes

Jeffwan reviewed Sep 24, 2025

View reviewed changes

varungupta and others added 2 commits September 24, 2025 14:03

Add gateway-plugin support to generate image and video

56bd9c3

Signed-off-by: varungupta <[email protected]>

fix lint error

773ef6e

Signed-off-by: varungupta <[email protected]>

varungup90 force-pushed the add-image-video branch from 3b1c99e to 773ef6e Compare September 24, 2025 21:10

varungup90 marked this pull request as ready for review September 24, 2025 21:24

Jeffwan approved these changes Sep 25, 2025

View reviewed changes

Merge branch 'main' into add-image-video

0485f48

varungup90 merged commit 82df5d2 into vllm-project:main Sep 25, 2025
12 checks passed

Jeffwan deleted the add-image-video branch September 25, 2025 17:54

Add gateway-plugin support to generate image and video #1603

Add gateway-plugin support to generate image and video #1603

Uh oh!

Conversation

varungup90 commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Description

Related Issues

Pull Request Title Format

Submission Checklist

Uh oh!

gemini-code-assist bot commented Sep 24, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

gemini-code-assist bot Sep 24, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot Sep 24, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

varungup90 Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jeffwan commented Sep 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

varungup90 commented Sep 24, 2025 •

edited

Loading

varungup90 Sep 24, 2025 •

edited

Loading