[batch] Support File List API, configurable job pool size and error file handling #1700
Conversation
Signed-off-by: Jiaxin Shan <[email protected]>
Signed-off-by: Jiaxin Shan <[email protected]>
Signed-off-by: Jiaxin Shan <[email protected]>
Signed-off-by: Jiaxin Shan <[email protected]>
Signed-off-by: Jiaxin Shan <[email protected]>
Summary of ChangesHello @Jeffwan, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request delivers Phase 2 improvements to the batch API, focusing on enhancing file management, job execution control, and API endpoint coverage. It introduces a new file listing API for better visibility into uploaded data, allows for dynamic configuration of batch job concurrency, and refines error handling by providing dedicated output files for failed requests. Additionally, the batch API's capabilities are expanded to support more OpenAI-compatible endpoints, all while maintaining backward compatibility. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces several enhancements to the batch API, including support for file listing, configurable job pool size, and error file handling. The changes include modifications to documentation, constants, storage adapters, and API endpoints. I have provided comments addressing a potential issue with exception handling in the list_files function.
| except Exception as e: | ||
| logger.error("Failed to list files", error=str(e)) # type: ignore[call-arg] | ||
| raise HTTPException(status_code=500, detail="Failed to list files") |
There was a problem hiding this comment.
The current implementation catches all exceptions but raises an HTTPException with a generic message. This obscures the original error, making debugging difficult. It would be better to either re-raise the original exception or include the original error message in the HTTPException.
Consider logging the full exception traceback for better debugging.
logger.exception("Failed to list files", error=str(e))| except Exception as e: | |
| logger.error("Failed to list files", error=str(e)) # type: ignore[call-arg] | |
| raise HTTPException(status_code=500, detail="Failed to list files") | |
| logger.exception("Failed to list files", error=str(e)) # Log the full exception with traceback | |
| raise HTTPException(status_code=500, detail=f"Failed to list files: {str(e)}") |
…ile handling (vllm-project#1700) * Make DEFAULT_JOB_POOL_SIZE configurable * Implement the file listing API * Added comprehensive docstring to prepare_job_ouput_files() * Create tests to cover batch API supported endpoints * Update the batch api docs Signed-off-by: Jiaxin Shan <[email protected]> Signed-off-by: Artyom Voronin <[email protected]>
1 participant
Pull Request Description
This PR implements Phase 2 improvements for the batch API, addressing GitHub issues #1651, #1652, #1653, and #1654. All changes maintain full backward compatibility with the OpenAI Batch API specification.
Related Issues
Resolves: part of #1644
Important: Before submitting, please complete the description above and review the checklist below.
Contribution Guidelines (Expand for Details)
We appreciate your contribution to aibrix! To ensure a smooth review process and maintain high code quality, please adhere to the following guidelines:
Pull Request Title Format
Your PR title should start with one of these prefixes to indicate the nature of the change:
[Bug]: Corrections to existing functionality[CI]: Changes to build process or CI pipeline[Docs]: Updates or additions to documentation[API]: Modifications to aibrix's API or interface[CLI]: Changes or additions to the Command Line Interface[Misc]: For changes not covered above (use sparingly)Note: For changes spanning multiple categories, use multiple prefixes in order of importance.
Submission Checklist
By submitting this PR, you confirm that you've read these guidelines and your changes align with the project's contribution standards.