Fixed Ollama Structured Response not working #10616

[imdigitalashish]

Title

feat(ollama): Allow direct Pydantic model usage in response_format for ollama_chat

Relevant issues

Fixes #10616

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
• have added a screenshot of my new test passing locally
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem

Type

🐛 Bug Fix

Changes

The key changes are:

1. Bug Resolution: This resolves the underlying issue that led to KeyError: 'schema' when users attempted to use Pydantic models without the precise nested structure, as reported in [Bug]: Structured Output not working with Ollama #10616. The PR ensures the schema is correctly processed by litellm before being passed to the Ollama API.

These changes enhance the developer experience for using structured outputs with Ollama via litellm and address the incompatibility noted in the referenced issue.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 8, 2025 9:50am

[CLAassistant]

All committers have signed the CLA.

[yelonek]

Hi @imdigitalashish and @krrishdholakia, the lack of structured output in ollama via litellm drove me crazy far enough that I looked into it today.

I see you check for type json_object and json_schema, but what is sent as a parameter to response_format is the result of model_json_schema() and that is a dict where type is set to "object", not "json_object".

I just added additional if and then it worked as expected:

if param == "response_format" and value["type"] == "object":
                optional_params["format"] = value

I am using pydantic '2.11.4' and litellm '1.69.1'

Let me know what you think!

[krrishdholakia]

Hey @yelonek is this an alternative to @imdigitalashish's PR?

if so - can you file it separately with a unit test in tests/litellm?

[yelonek]

Hey @yelonek is this an alternative to @imdigitalashish's PR?

if so - can you file it separately with a unit test in tests/litellm?

I think it is complementary.
First I need to figure out how to set up the dev environment for litellm. So far I figured out, that Windows will not cut it. ;)

[krrishdholakia]

What's the issue you see? @yelonek