Skip to content

Support vertex_ai global endpoints for chat #10658

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 21, 2025
Merged

Support vertex_ai global endpoints for chat #10658

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/my-website/docs/providers/vertex.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem';
| Description | Vertex AI is a fully-managed AI development platform for building and using generative AI. |
| Provider Route on LiteLLM | `vertex_ai/` |
| Link to Provider Doc | [Vertex AI ↗](https://cloud.google.com/vertex-ai) |
| Base URL | [https://{vertex_location}-aiplatform.googleapis.com/](https://{vertex_location}-aiplatform.googleapis.com/) |
| Base URL | 1. Regional endpoints<br/>[https://{vertex_location}-aiplatform.googleapis.com/](https://{vertex_location}-aiplatform.googleapis.com/)<br/>2. Global endpoints (limited availability)<br/>[https://aiplatform.googleapis.com/](https://{aiplatform.googleapis.com/)|
| Supported Operations | [`/chat/completions`](#sample-usage), `/completions`, [`/embeddings`](#embedding-models), [`/audio/speech`](#text-to-speech-apis), [`/fine_tuning`](#fine-tuning-apis), [`/batches`](#batch-apis), [`/files`](#batch-apis), [`/images`](#image-generation-models) |


Expand Down Expand Up @@ -832,7 +832,7 @@ OR

You can set:
- `vertex_credentials` (str) - can be a json string or filepath to your vertex ai service account.json
- `vertex_location` (str) - place where vertex model is deployed (us-central1, asia-southeast1, etc.)
- `vertex_location` (str) - place where vertex model is deployed (us-central1, asia-southeast1, etc.). Some models support the global location, please see [Vertex AI documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/locations#supported_models)
- `vertex_project` Optional[str] - use if vertex project different from the one in vertex_credentials

as dynamic params for a `litellm.completion` call.
Expand Down
10 changes: 8 additions & 2 deletions litellm/llms/vertex_ai/common_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,9 +84,15 @@ def _get_vertex_url(
endpoint = "generateContent"
if stream is True:
endpoint = "streamGenerateContent"
url = f"https://{vertex_location}-aiplatform.googleapis.com/{vertex_api_version}/projects/{vertex_project}/locations/{vertex_location}/publishers/google/models/{model}:{endpoint}?alt=sse"
if vertex_location== "global":
Copy link
Contributor

@krrishdholakia krrishdholakia May 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the difference between this and /{vertex_location}/..

this seems like it would have the same final url

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's the hostname that is different, in global mode you skip the vertex_location part if the hostname

Copy link

@abriko abriko May 19, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@krrishdholakia here is official doc
https://cloud.google.com/vertex-ai/generative-ai/docs/learn/locations#use_the_global_endpoint

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://aiplatform.googleapis.com/{vertex_api_version}/projects/{vertex_project}/locations/global/publishers/google/models/{model}:{endpoint}?alt=sse

based on the url - it looks like you still have a location it's just = global

so wouldn't this already work today if you set vertex_location="global" ?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @krrishdholakia
I believe the major change for this PR is domain name in URL.
Regional domain: https://{vertex_location}-aiplatform.googleapis.com
Global domain: https://aiplatform.googleapis.com

Currently the litellm not support global domain.

url = f"https://aiplatform.googleapis.com/{vertex_api_version}/projects/{vertex_project}/locations/global/publishers/google/models/{model}:{endpoint}?alt=sse"
else:
url = f"https://{vertex_location}-aiplatform.googleapis.com/{vertex_api_version}/projects/{vertex_project}/locations/{vertex_location}/publishers/google/models/{model}:{endpoint}?alt=sse"
else:
url = f"https://{vertex_location}-aiplatform.googleapis.com/{vertex_api_version}/projects/{vertex_project}/locations/{vertex_location}/publishers/google/models/{model}:{endpoint}"
if vertex_location == "global":
url = f"https://aiplatform.googleapis.com/{vertex_api_version}/projects/{vertex_project}/locations/global/publishers/google/models/{model}:{endpoint}"
else:
url = f"https://{vertex_location}-aiplatform.googleapis.com/{vertex_api_version}/projects/{vertex_project}/locations/{vertex_location}/publishers/google/models/{model}:{endpoint}"

# if model is only numeric chars then it's a fine tuned gemini model
# model = 4965075652664360960
Expand Down
43 changes: 43 additions & 0 deletions tests/litellm/llms/vertex_ai/test_vertex_ai_common_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
get_vertex_location_from_url,
get_vertex_project_id_from_url,
set_schema_property_ordering,
_get_vertex_url
)


Expand Down Expand Up @@ -516,3 +517,45 @@ def test_vertex_ai_complex_response_schema():
assert "additionalProperties" not in type2
assert "additionalProperties" not in type3
assert "additionalProperties" not in type3_prop3_items

@pytest.mark.parametrize(
"stream, expected_endpoint_suffix",
[
(True, "streamGenerateContent?alt=sse"),
(False, "generateContent"),
],
)
def test_get_vertex_url_global_region(stream, expected_endpoint_suffix):
"""
Test _get_vertex_url when vertex_location is 'global' for chat mode.
"""
mode = "chat"
model = "gemini-1.5-pro-preview-0409"
vertex_project = "test-g-project"
vertex_location = "global"
vertex_api_version = "v1"

# Mock litellm.VertexGeminiConfig.get_model_for_vertex_ai_url to return model as is
# as we are not testing that part here, just the URL construction
with patch("litellm.VertexGeminiConfig.get_model_for_vertex_ai_url", side_effect=lambda model: model):
url, endpoint = _get_vertex_url(
mode=mode,
model=model,
stream=stream,
vertex_project=vertex_project,
vertex_location=vertex_location,
vertex_api_version=vertex_api_version,
)

expected_url_base = f"https://aiplatform.googleapis.com/{vertex_api_version}/projects/{vertex_project}/locations/global/publishers/google/models/{model}"

if stream:
expected_endpoint = "streamGenerateContent"
expected_url = f"{expected_url_base}:{expected_endpoint}?alt=sse"
else:
expected_endpoint = "generateContent"
expected_url = f"{expected_url_base}:{expected_endpoint}"


assert endpoint == expected_endpoint
assert url == expected_url
Loading