Skip to content

[Model] Add new model provider: AI/ML API #1093

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

D1m7asis
Copy link

From #1073

This PR adds first-class support for AI/ML API as a new unified model provider to VLMEvalKit.
AI/ML API provides access to 300+ models, including GPT-4o, Claude 3.5, Gemini 1.5, DeepSeek, LLaMA3, Mistral, and many others — via a single OpenAI-compatible interface.


🔧 What was added

  • ✅ New model wrapper: vlmeval/api/aimlapi.py
  • ✅ Registration in vlmeval/api/__init__.py
  • ✅ Multiple preconfigured model entries in vlmeval/config.py via ai_ml_api_models
  • ✅ Extended api_models with support for AIMLAPI
  • ✅ English and Chinese documentation in docs/en/Quickstart.md and docs/zh-CN/Quickstart.md
  • ✅ Ready-to-use AIML_API_KEY environment variable support

🚀 Usage Example

AIML_API_KEY=your_key_here python run.py --data CCBench --model aimlapi_gpt_4o

✅ Checklist

  • Code follows repo structure
  • Docs (EN + ZH) updated
  • Tested on: MMBench, CCBench, MME
  • Uses OpenAI-compatible API structure

Let me know if you'd like to adjust or split this PR into parts.

@kennymckormick
Copy link
Member

Hi, @D1m7asis

Would you please share me a temporary account on the AI / ML API platform for testing (w. $10 credit, e.g.)? My email is [email protected].

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants