Skip to content

Fix/issue 10113 embeddings use non default tokenizer #10629

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 15, 2025
Merged

Fix/issue 10113 embeddings use non default tokenizer #10629

Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a48acd9
fix(embeddings): use non default tokenizer when passing list of lists…
camfarineau May 7, 2025
18f57d6
feat(embeddings): allow for passthrough of list of lists of tokens to…
camfarineau May 7, 2025
a11ea61
Revert "fix(embeddings): use non default tokenizer when passing list …
camfarineau May 12, 2025
9ae9166
refactor(embeddings): use a list to verify if provider accept as inpu…
camfarineau May 12, 2025
94955a7
fix(embeddings): verify the model name before validating if provider …
camfarineau May 12, 2025
cedacd1
test(embedding): add unit test to bypass decode for some providers wi…
camfarineau May 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion litellm/proxy/proxy_server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3814,14 +3814,15 @@ async def embeddings( # noqa: PLR0915
if m["model_name"] == data["model"] and (
m["litellm_params"]["model"] in litellm.open_ai_embedding_models
or m["litellm_params"]["model"].startswith("azure/")
or m["litellm_params"]["model"].startswith("hosted_vllm/")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a unit test for this in test_proxy_server.py in tests/litellm -

litellm/tests/litellm/proxy/test_proxy_server.py

Line 4 in e2a9cd0

import socket

So there are no future regressions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally - this can be a list in constants.py - stating the list of Llmproviders which support an input of array of tokens

Constants.py file -

litellm/litellm/constants.py

Line 4 in 02efffc

DEFAULT_BATCH_SIZE = 512

example list -

litellm/litellm/constants.py

Line 103 in 02efffc

LITELLM_CHAT_PROVIDERS = [

):
pass
else:
# non-openai/azure embedding model called with token input
input_list = []
for i in data["input"]:
input_list.append(
litellm.decode(model="gpt-3.5-turbo", tokens=i)
litellm.decode(model=m["model_name"], tokens=i)
)
data["input"] = input_list
break
Expand Down
Loading