/utils/token_counter: get model_info from deployment directly

[chaofuyang]

Title

/utils/token_counter: get model_info from deployment directly

Relevant issues

It raises the "model is not mapped yet" error when we configured the the model as below. Since we only need the custom_tokenizer field in the model_info. We do not need to get the other info like model cost from the cost map.

{
  "model_name": "cf-test-2",
  "litellm_params": {
    "api_base": "http://ma-mix.cicero.dev-spark.ali-netbase.com/v1",
    "input_cost_per_token": 0.00001,
    "output_cost_per_token": 0.00001,
    "model": "openai/model",
    "use_in_pass_through": false
  },
  "model_info": {
    "id": "62fb4db3-7f9a-4031-b33a-aa6ceae4f98e",
    "db_model": true,
    "custom_tokenizer": {
      "revision": "main",
      "auth_token": "os.environ/HUGGING_FACE_HUB_TOKEN",
      "identifier": "meta-llama/Llama-3.1-8B-Instruct"
    },
    "input_cost_per_token": 1,
    "output_cost_per_token": 1
  },
}

Detailed error message:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self.scope, self.receive, self.send
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/lib/python3.13/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/lib/python3.13/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/lib/python3.13/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/lib/python3.13/site-packages/starlette/middleware/base.py", line 185, in __call__
    with collapse_excgroups():
         ~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3.13/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/usr/lib/python3.13/site-packages/starlette/middleware/base.py", line 187, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/litellm/proxy/middleware/prometheus_auth_middleware.py", line 47, in dispatch
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/starlette/middleware/base.py", line 163, in call_next
    raise app_exc
  File "/usr/lib/python3.13/site-packages/starlette/middleware/base.py", line 149, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/usr/lib/python3.13/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/lib/python3.13/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/lib/python3.13/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/lib/python3.13/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/lib/python3.13/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/lib/python3.13/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/usr/lib/python3.13/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/litellm/proxy/proxy_server.py", line 5292, in token_counter
    model_info = llm_router.get_router_model_info(
        deployment=deployment,
        received_model_name=request.model,
    )
  File "/usr/lib/python3.13/site-packages/litellm/router.py", line 4808, in get_router_model_info
    model_info = litellm.get_model_info(model=model_info_name)
  File "/usr/lib/python3.13/site-packages/litellm/utils.py", line 4701, in get_model_info
    _model_info = _get_model_info_helper(
        model=model,
        custom_llm_provider=custom_llm_provider,
    )
  File "/usr/lib/python3.13/site-packages/litellm/utils.py", line 4620, in _get_model_info_helper
    raise Exception(
    ...<3 lines>...
    )
Exception: This model isn't mapped yet. model=openai/model, custom_llm_provider=openai. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json.

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
• I have added a screenshot of my new test passing locally
• My PR passes all unit tests on (make test-unit)[https://docs.litellm.ai/docs/extras/contributing_code]
• My PR's scope is as isolated as possible, it only solves 1 specific problem

Test

A request was sent to utils/token_counter

Type

🐛 Bug Fix

Changes

• Get the model_info from deployment directly instead of getting the model info through llm_router.get_router_model_info

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 16, 2025 7:32am