Skip to content

Fix/issue 10113 embeddings use non default tokenizer #10629

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 15, 2025
Merged

Fix/issue 10113 embeddings use non default tokenizer #10629

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a48acd9
fix(embeddings): use non default tokenizer when passing list of lists…
camfarineau May 7, 2025
18f57d6
feat(embeddings): allow for passthrough of list of lists of tokens to…
camfarineau May 7, 2025
a11ea61
Revert "fix(embeddings): use non default tokenizer when passing list …
camfarineau May 12, 2025
9ae9166
refactor(embeddings): use a list to verify if provider accept as inpu…
camfarineau May 12, 2025
94955a7
fix(embeddings): verify the model name before validating if provider …
camfarineau May 12, 2025
cedacd1
test(embedding): add unit test to bypass decode for some providers wi…
camfarineau May 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions litellm/constants.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,12 @@
"meta_llama",
]

LITELLM_EMBEDDING_PROVIDERS_SUPPORTING_INPUT_ARRAY_OF_TOKENS = [
"openai",
"azure",
"hosted_vllm"
]


OPENAI_CHAT_COMPLETION_PARAMS = [
"functions",
Expand Down
35 changes: 19 additions & 16 deletions litellm/proxy/proxy_server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
from litellm.constants import (
DEFAULT_MAX_RECURSE_DEPTH,
DEFAULT_SLACK_ALERTING_THRESHOLD,
LITELLM_EMBEDDING_PROVIDERS_SUPPORTING_INPUT_ARRAY_OF_TOKENS
)
from litellm.types.utils import (
ModelResponse,
Expand Down Expand Up @@ -3808,24 +3809,26 @@ async def embeddings( # noqa: PLR0915
and isinstance(data["input"][0], list)
and isinstance(data["input"][0][0], int)
): # check if array of tokens passed in
# check if non-openai/azure model called - e.g. for langchain integration
# check if provider accept list of tokens as input - e.g. for langchain integration
if llm_model_list is not None and data["model"] in router_model_names:
for m in llm_model_list:
if m["model_name"] == data["model"] and (
m["litellm_params"]["model"] in litellm.open_ai_embedding_models
or m["litellm_params"]["model"].startswith("azure/")
or m["litellm_params"]["model"].startswith("hosted_vllm/")
):
pass
else:
# non-openai/azure embedding model called with token input
input_list = []
for i in data["input"]:
input_list.append(
litellm.decode(model="gpt-3.5-turbo", tokens=i)
)
data["input"] = input_list
break
if m["model_name"] == data["model"]:
if (m["litellm_params"]["model"] in litellm.open_ai_embedding_models
or any(
m["litellm_params"]["model"].startswith(provider)
for provider in LITELLM_EMBEDDING_PROVIDERS_SUPPORTING_INPUT_ARRAY_OF_TOKENS
)
):
pass
else:
# non-openai/azure embedding model called with token input
input_list = []
for i in data["input"]:
input_list.append(
litellm.decode(model="gpt-3.5-turbo", tokens=i)
)
data["input"] = input_list
break

### CALL HOOKS ### - modify incoming data / reject request before calling the model
data = await proxy_logging_obj.pre_call_hook(
Expand Down
Loading