Releases: BerriAI/litellm
Releases · BerriAI/litellm
v1.70.2.dev5
What's Changed
- [Feature] Add supports_computer_use to the model list by @slytechnical in #10881
- [Feat] - Add Support for Showing Passthrough endpoint Error Logs on LiteLLM UI by @ishaan-jaff in #10990
- added support to credential delete to support slashes in the curl by @jugaldb in #10987
- Add new gemini preview models + Fix cohere v2 embedding 'embedding_types' param mapping by @krrishdholakia in #10991
- Litellm add new gemini models by @krrishdholakia in #10998
- [Feat] Prometheus - Track
routeon proxy_* metrics by @ishaan-jaff in #10992 - fix: default role for JWT authentication by @ishaan-jaff in #10995
- fix(internal_user_endpoints.py): allow resetting spend/max budget on
/user/update+ Fix updating user budget across multiple instances by @krrishdholakia in #10993 - Improve response_id propagation logic and add tests for valid/empty I… by @daarko10 in #11006
- Support vertex_ai global endpoints for chat by @sorenmat in #10658
- Ollama wildcard support by @matthid in #10982
- Spend rounded to 4 decimal points on Teams page by @NANDINI-star in #11013
- Place organization and team create buttons at the top by @NANDINI-star in #10948
- Feat: add xai/grok-3 pricing by @ishaan-jaff in #11028
- [Feat] Add Image Edits Support to LiteLLM by @ishaan-jaff in #11020
- Updating the available VoyageAI models in the docs by @fzowl in #11003
- Fix UI - Call tool renders correctly when no arguments required by @wagnerjt in #11012
- Verbose error on admin add by @jugaldb in #10978
- Spend rounded to 4 for Organizations and Users page by @NANDINI-star in #11023
- Fix: Handle dict objects in Anthropic streaming response by @jgowdy-godaddy in #11032
- feat: add Databricks Llama 4 Maverick model cost by @bepotp in #11008
- Add LiteLLM Managed Files support for finetuning api
createendpoint by @krrishdholakia in #11039 - Fixes the InvitationLink Prisma find_many query by @jmorenoc-o in #11031
- Support passing
prompt_labelto langfuse + Log consistent modelParams to langfuse by @krrishdholakia in #11018 - Fix CI | Added closing tags for + indentation changes by @NANDINI-star in #11046
- Feat: add MCP to Responses API and bump openai python sdk by @ishaan-jaff in #11029
- Model filter on logs by @NANDINI-star in #11048
- [Feat] Add claude-4 model family by @ishaan-jaff in #11060
New Contributors
- @slytechnical made their first contribution in #10881
- @daarko10 made their first contribution in #11006
- @sorenmat made their first contribution in #10658
- @matthid made their first contribution in #10982
- @jgowdy-godaddy made their first contribution in #11032
- @bepotp made their first contribution in #11008
- @jmorenoc-o made their first contribution in #11031
Full Changelog: v1.70.2-nightly...v1.70.2.dev5
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.70.2.dev5
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 490.0 | 556.7359308349169 | 5.626256137716844 | 0.0 | 1684 | 0 | 437.79858300001706 | 2137.070654000013 |
| Aggregated | Failed ❌ | 490.0 | 556.7359308349169 | 5.626256137716844 | 0.0 | 1684 | 0 | 437.79858300001706 | 2137.070654000013 |
Contributors
sorenmat, matthid, and 11 other contributors
Assets 4
v1.70.2-nightly
Full Changelog: v1.70.1.dev2...v1.70.2-nightly
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.70.2-nightly
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 480.0 | 551.6473700786378 | 5.693462012007638 | 0.0 | 1704 | 0 | 435.9962309999901 | 1522.4978100000044 |
| Aggregated | Failed ❌ | 480.0 | 551.6473700786378 | 5.693462012007638 | 0.0 | 1704 | 0 | 435.9962309999901 | 1522.4978100000044 |
What's Changed
- add docs for spend logs by @jugaldb in #10913
- [QA/Guardrails] Allow editing guardrail entity configs by @ishaan-jaff in #10918
- [UI QA Guardrails] - Minor UI Fixes by @ishaan-jaff in #10920
- [CI/CD] Fix - linting cryptography error by @ishaan-jaff in #10921
- LiteLLM Stable release notes by @krrishdholakia in #10919
- docs apply guardrail endpoint by @ishaan-jaff in #10923
- [Docs] Pii masking with LiteLLM by @ishaan-jaff in #10924
- Add provider Featherless AI (access to 4200+ models) by @DarinVerheijke in #10596
- fix(vertex_llm_base.py): remove quota_project_id parameter from crede… by @krrishdholakia in #10915
- Enable structured JSON schema outputs in LM Studio by @estsauver in #10929
- update sambanova models and parameters by @jhpiedrahitao in #10900
- Fix Cohere Rerank Provider by @keyute in #10822
- add skip server startup flag to cli by @mohittalele in #10665
- Allow passed in vertex_ai credentials to be authorized_user type by @pselden in #10899
- fix: handle DB_USER, DB_PASSWORD, DB_HOST problem I faced, since this… by @unrealandychan in #10842
- Add keys and members count in team info by @NANDINI-star in #10950
- Update github.md by @dastaiger in #10946
- docs: add llms.txt and llms_full.txt for LLM-friendly scraping by @colesmcintosh in #10961
- [Fix] Invitation Email does not include the invitation link by @ishaan-jaff in #10958
- Enable key reassignment on UI + Show 'updated at' column for key in all keys table by @krrishdholakia in #10960
- Update existing model's model info on UI + Show model access groups when creating keys/teams on UI by @krrishdholakia in #10965
- Validate migrating keys to teams + Fix mistral image url on async translation by @krrishdholakia in #10966
- add cla to docs by @jugaldb in #10963
- [Fix] List Guardrails - Show config.yaml guardrails on litellm ui by @ishaan-jaff in #10959
- fix: vertex show clear exception on failed refresh by @ishaan-jaff in #10969
New Contributors
- @DarinVerheijke made their first contribution in #10596
- @estsauver made their first contribution in #10929
- @mohittalele made their first contribution in #10665
- @pselden made their first contribution in #10899
- @unrealandychan made their first contribution in #10842
- @dastaiger made their first contribution in #10946
Full Changelog: v1.70.1-stable...v1.70.2-nightly
Contributors
pselden, estsauver, and 11 other contributors
Assets 4
v1.70.1.dev8
What's Changed
- add docs for spend logs by @jugaldb in #10913
- [QA/Guardrails] Allow editing guardrail entity configs by @ishaan-jaff in #10918
- [UI QA Guardrails] - Minor UI Fixes by @ishaan-jaff in #10920
- [CI/CD] Fix - linting cryptography error by @ishaan-jaff in #10921
- LiteLLM Stable release notes by @krrishdholakia in #10919
- docs apply guardrail endpoint by @ishaan-jaff in #10923
- [Docs] Pii masking with LiteLLM by @ishaan-jaff in #10924
- Add provider Featherless AI (access to 4200+ models) by @DarinVerheijke in #10596
- fix(vertex_llm_base.py): remove quota_project_id parameter from crede… by @krrishdholakia in #10915
- Enable structured JSON schema outputs in LM Studio by @estsauver in #10929
- update sambanova models and parameters by @jhpiedrahitao in #10900
- Fix Cohere Rerank Provider by @keyute in #10822
- add skip server startup flag to cli by @mohittalele in #10665
- Allow passed in vertex_ai credentials to be authorized_user type by @pselden in #10899
- fix: handle DB_USER, DB_PASSWORD, DB_HOST problem I faced, since this… by @unrealandychan in #10842
- Add keys and members count in team info by @NANDINI-star in #10950
- Update github.md by @dastaiger in #10946
- docs: add llms.txt and llms_full.txt for LLM-friendly scraping by @colesmcintosh in #10961
- [Fix] Invitation Email does not include the invitation link by @ishaan-jaff in #10958
New Contributors
- @DarinVerheijke made their first contribution in #10596
- @estsauver made their first contribution in #10929
- @mohittalele made their first contribution in #10665
- @pselden made their first contribution in #10899
- @unrealandychan made their first contribution in #10842
- @dastaiger made their first contribution in #10946
Full Changelog: v1.70.1-stable...v1.70.1.dev8
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.70.1.dev8
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 490.0 | 562.6894453097084 | 5.610163428556303 | 0.0033413719050365115 | 1679 | 1 | 195.35745899997892 | 1568.1852209999647 |
| Aggregated | Failed ❌ | 490.0 | 562.6894453097084 | 5.610163428556303 | 0.0033413719050365115 | 1679 | 1 | 195.35745899997892 | 1568.1852209999647 |
Contributors
pselden, estsauver, and 11 other contributors
Assets 4
v1.70.1.dev6
Full Changelog: v1.70.1.dev4...v1.70.1.dev6
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.70.1.dev6
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 520.0 | 614.117503553192 | 5.499725111392901 | 0.003343297940056475 | 1645 | 1 | 470.59824999996636 | 40452.63952500005 |
| Aggregated | Failed ❌ | 520.0 | 614.117503553192 | 5.499725111392901 | 0.003343297940056475 | 1645 | 1 | 470.59824999996636 | 40452.63952500005 |
v1.70.1.dev4
Full Changelog: v1.70.1.dev2...v1.70.1.dev4
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.70.1.dev4
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 500.0 | 594.1888488711803 | 5.578781212015596 | 0.006685178204931811 | 1669 | 2 | 190.37631700001612 | 40183.55002499999 |
| Aggregated | Failed ❌ | 500.0 | 594.1888488711803 | 5.578781212015596 | 0.006685178204931811 | 1669 | 2 | 190.37631700001612 | 40183.55002499999 |
v1.70.1.dev2
Full Changelog: v1.67.0-stable.patch2...v1.70.1.dev2
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.70.1.dev2
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 530.0 | 603.0679202423511 | 5.573003547206886 | 0.0 | 1667 | 0 | 488.8812669999538 | 2023.6071630000083 |
| Aggregated | Failed ❌ | 530.0 | 603.0679202423511 | 5.573003547206886 | 0.0 | 1667 | 0 | 488.8812669999538 | 2023.6071630000083 |
v1.70.1.dev11
Full Changelog: v1.70.1.dev8...v1.70.1.dev11
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.70.1.dev11
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 510.0 | 588.5182217067622 | 5.584563874858318 | 0.0 | 1671 | 0 | 469.68324099998426 | 1859.8325959999897 |
| Aggregated | Failed ❌ | 510.0 | 588.5182217067622 | 5.584563874858318 | 0.0 | 1671 | 0 | 469.68324099998426 | 1859.8325959999897 |
v1.67.0-stable.patch2
What's Changed
- add docs for spend logs by @jugaldb in #10913
- [QA/Guardrails] Allow editing guardrail entity configs by @ishaan-jaff in #10918
- [UI QA Guardrails] - Minor UI Fixes by @ishaan-jaff in #10920
- [CI/CD] Fix - linting cryptography error by @ishaan-jaff in #10921
- LiteLLM Stable release notes by @krrishdholakia in #10919
- docs apply guardrail endpoint by @ishaan-jaff in #10923
- [Docs] Pii masking with LiteLLM by @ishaan-jaff in #10924
- Add provider Featherless AI (access to 4200+ models) by @DarinVerheijke in #10596
- fix(vertex_llm_base.py): remove quota_project_id parameter from crede… by @krrishdholakia in #10915
- Enable structured JSON schema outputs in LM Studio by @estsauver in #10929
- update sambanova models and parameters by @jhpiedrahitao in #10900
- Fix Cohere Rerank Provider by @keyute in #10822
- add skip server startup flag to cli by @mohittalele in #10665
- Allow passed in vertex_ai credentials to be authorized_user type by @pselden in #10899
- fix: handle DB_USER, DB_PASSWORD, DB_HOST problem I faced, since this… by @unrealandychan in #10842
- Add keys and members count in team info by @NANDINI-star in #10950
- Update github.md by @dastaiger in #10946
- docs: add llms.txt and llms_full.txt for LLM-friendly scraping by @colesmcintosh in #10961
- [Fix] Invitation Email does not include the invitation link by @ishaan-jaff in #10958
- Enable key reassignment on UI + Show 'updated at' column for key in all keys table by @krrishdholakia in #10960
- Update existing model's model info on UI + Show model access groups when creating keys/teams on UI by @krrishdholakia in #10965
- Validate migrating keys to teams + Fix mistral image url on async translation by @krrishdholakia in #10966
- add cla to docs by @jugaldb in #10963
- [Fix] List Guardrails - Show config.yaml guardrails on litellm ui by @ishaan-jaff in #10959
- fix: vertex show clear exception on failed refresh by @ishaan-jaff in #10969
New Contributors
- @DarinVerheijke made their first contribution in #10596
- @estsauver made their first contribution in #10929
- @mohittalele made their first contribution in #10665
- @pselden made their first contribution in #10899
- @unrealandychan made their first contribution in #10842
- @dastaiger made their first contribution in #10946
Full Changelog: v1.70.1-stable...v1.67.0-stable.patch2
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.67.0-stable.patch2
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 250.0 | 261.5294727693982 | 6.120234747528359 | 0.0 | 1830 | 0 | 217.36453699998037 | 1204.8032490000082 |
| Aggregated | Passed ✅ | 250.0 | 261.5294727693982 | 6.120234747528359 | 0.0 | 1830 | 0 | 217.36453699998037 | 1204.8032490000082 |
Contributors
pselden, estsauver, and 11 other contributors
Assets 4
v1.70.1-stable
What's Changed
- Handle gemini audio input by @krrishdholakia in #10739
- Fixed Ollama Structured Response not working #10616 by @imdigitalashish in #10617
- fix(factory.py): Add reasoning content handling for missing assistant… by @LouisShark in #10688
- [Feat] Add tools support for Nvidia NIM by @ishaan-jaff in #10763
- [Fix]: /messages - allow using dynamic AWS params by @ishaan-jaff in #10769
- fix: pass application/json for GenericAPILogger by @ishaan-jaff in #10772
- [Docs] Using litellm with Google ADK by @ishaan-jaff in #10777
- Update Nscale model providers to point to website by @OscarSavNS in #10764
- [Fix] Allow using dynamic aws_region with /messages on Bedrock by @ishaan-jaff in #10779
- [Feat] Option to force/always use the litellm proxy (#10559) (#10633) by @ishaan-jaff in #10773
- feat: Addded EU Anthropic Inference profile for Claude 3.7 by @wagnerjt in #10767
- Add new model provider Novita AI (#7582) by @krrishdholakia in #9527
- Support Anthropic web search tool + Add more google finish reason mapping by @krrishdholakia in #10785
- Fix azure dall e 3 call with custom model name + Handle
Bearer $LITELLM_API_KEYinx-litellm-api-keycustom header by @krrishdholakia in #10776 - [Refactor] Move LLM Guard, Secret Detection to Enterprise Pip packagea by @ishaan-jaff in #10782
- fix: URL encode OTEL_EXPORTER_OTLP_TRACES_HEADERS for Phoenix Integration by @arizedatngo in #10654
- [README] Add Llama API in supported providers table by @seyeong-han in #10802
- [Fix] Maximum Vertex Recursion Depth Reached when transforming response_format by @ishaan-jaff in #10798
- add ollama api base defaults by @jugaldb in #10805
- Add Azure gpt-4o-mini-tts by @emerzon in #10807
- Add Azure Cohere Embed v4 by @emerzon in #10806
- [Fix] Allow overriding all constants using a .env variable by @ishaan-jaff in #10803
- [Bug Fix] Presidio integration failing and making inference return 500 by @ishaan-jaff in #10197
- [Feat] Allow specifying PII Entities Config when using Presidio Guardrails by @ishaan-jaff in #10810
- [Bedrock] Fixes to only allow accepted fields for tool json schema by @ishaan-jaff in #10062
- Fix: Separate Sync/Async Azure Client Cache to Prevent Type Mismatches by @daikeren in #10781
- fix(litellm_logging.py): log custom headers in requester metadata by @krrishdholakia in #10818
- Migrate cohere to
/v2/embed- adds support for output_dimensions param by @krrishdholakia in #10809 - add support to parse metadata by @jugaldb in #10832
- Team id + Status filter on logs by @NANDINI-star in #10831
- Update Groq models on model_prices_and_context_window.json by @naliotopier in #10077
- [Feat] Add endpoints for adding, deleting, editing guardrails in DB by @ishaan-jaff in #10833
- [Feat UI] Add Guardrails on LiteLLM UI by @ishaan-jaff in #10834
- [UI] Fixes for bulk invite users - add validation for bad inputs by @ishaan-jaff in #10836
- Support OTLP standard environment variables for configuration by @damienpontifex in #10813
- [Feat] Presidio Improvements - Allow adding presidio api base on UI, Test presidio on Test Key, fixes for running presidio hook by @ishaan-jaff in #10840
- patch: validate_team_org_change by @ishaan-jaff in #10835
- Feat add retention config by @jugaldb in #10815
- fix failing test typo by @jugaldb in #10843
- Capture Gemini reasoning tokens usage in streaming mode by @Dima-Mediator in #10789
- Anthropic unified web search + tool cost tracking support by @krrishdholakia in #10846
- avoid error
AttributeError: 'Message' object has no attribute 'audio'by @igtm in #10814 - fix: bedrock-sonnet-caching-cost by @shibaboy in #10752
- Fix/issue 10113 embeddings use non default tokenizer by @camfarineau in #10629
- Add
models importcommand by @msabramo in #10581 - Add documentation on in-memory and disk caching by @user202729 in #10522
- Bedrock pixtral fix by @ajac-zero in #10439
- added fix to resolve delete API by @jugaldb in #10845
- build(deps): bump undici from 6.21.1 to 6.21.3 in /docs/my-website by @dependabot in #10855
- [Refactor] Make Pagerduty a free feature by @ishaan-jaff in #10857
- [Refactor] Move enterprise_routes within litellm_enterprise by @ishaan-jaff in #10860
- refactor: make send_management_endpoint_alert a free feature by @ishaan-jaff in #10863
- [Feat - Guardrails] Expose /apply_guardrail endpoint for directly calling guardrail by @ishaan-jaff in #10867
- [CI/CD] Test: fix mapped tests by @ishaan-jaff in #10870
- [Feat] UI: Add support for Deleting Guardrail on UI by @ishaan-jaff in #10869
- [UI] Allow adding Bedrock, Presidio, Lakera, AIM guardrails on UI by @ishaan-jaff in #10874
- fix(proxy/_types.py): add missing comma for
/v2/rerankby @krrishdholakia in #10861 - fix(streaming_handler.py): patch vertex ai meta llama streaming error by @krrishdholakia in #10878
- build(dependencies)📦: Add numpydoc to dependencies in pyproject.toml by @ericmjl in #10736
- [Feat]: Configure LiteLLM to Parse User Headers from Open Web UI by @damgem in #9802
- Feat/support anonymize in aim guardrail by @hxdror in #10757
- Gemini Multimodal Live API support by @krrishdholakia in #10841
- Feat log retention cleanup by @jugaldb in #10872
- Model filter added on logs tab. (#10877) by @ishaan-jaff in #10891
- fix: fix novita ai docs by @jasonhp in #10882
- [Feat] Add Tracing for guardrails in StandardLoggingPayload, Langfuse by @ishaan-jaff in #10890
- [UI] - Add Guardrail Tracing to LiteLLM SpendLogs by @ishaan-jaff in #10893
- Fix example commands in reliability.md by @wwwillchen in #10894
- Feat: add guardrail tracing to OTEL, Arize phoenix by @ishaan-jaff in #10896
- Key hash filter added by @NANDINI-star in #10903
- [Feat] UI: Allow clicking into a guardrail by @ishaan-jaff in #10904
- [Feat] Add Lakera v2 Guardrail Support by @ishaan-jaff in #10880
- Key alias filter on logs by @NANDINI-star in #10905
- [Feat] handle multi-pod deployment for SpendLogs Retention by @jugaldb in #10895
- [UI] Fixes for using Lakera Guardrail on LiteLLM UI by @ishaan-jaff in #10906
- add constant to config settings.md by @jugaldb in #10908
- Remove extra test file by @jugaldb in #10889
- [UI] Allow editing guardrails by @ishaan-jaff in #10907
- Handle openai gpt file data + add openai 'supports_pdf_input' to all vision models + Support bedrock tool cache pointing by @krrishdholakia in #10897
New Contributors
- @imdigitalashish made their first contribution in #10617
- @LouisShark made their first contribution in #10688
- @OscarSavNS made their first contribution in #10764
- @arizedatngo made their first contribution in #10654
- @jugaldb made their first contribution in https://github.com/BerriAI/litellm/pull/...
Contributors
msabramo, shibaboy, and 25 other contributors
Assets 2
v1.70.0-nightly
What's Changed
- Capture Gemini reasoning tokens usage in streaming mode by @Dima-Mediator in #10789
- Anthropic unified web search + tool cost tracking support by @krrishdholakia in #10846
- avoid error
AttributeError: 'Message' object has no attribute 'audio'by @igtm in #10814 - fix: bedrock-sonnet-caching-cost by @shibaboy in #10752
- Fix/issue 10113 embeddings use non default tokenizer by @camfarineau in #10629
- Add
models importcommand by @msabramo in #10581 - Add documentation on in-memory and disk caching by @user202729 in #10522
- Bedrock pixtral fix by @ajac-zero in #10439
- added fix to resolve delete API by @jugaldb in #10845
- build(deps): bump undici from 6.21.1 to 6.21.3 in /docs/my-website by @dependabot in #10855
- [Refactor] Make Pagerduty a free feature by @ishaan-jaff in #10857
- [Refactor] Move enterprise_routes within litellm_enterprise by @ishaan-jaff in #10860
- refactor: make send_management_endpoint_alert a free feature by @ishaan-jaff in #10863
- [Feat - Guardrails] Expose /apply_guardrail endpoint for directly calling guardrail by @ishaan-jaff in #10867
- [CI/CD] Test: fix mapped tests by @ishaan-jaff in #10870
- [Feat] UI: Add support for Deleting Guardrail on UI by @ishaan-jaff in #10869
- [UI] Allow adding Bedrock, Presidio, Lakera, AIM guardrails on UI by @ishaan-jaff in #10874
- fix(proxy/_types.py): add missing comma for
/v2/rerankby @krrishdholakia in #10861 - fix(streaming_handler.py): patch vertex ai meta llama streaming error by @krrishdholakia in #10878
- build(dependencies)📦: Add numpydoc to dependencies in pyproject.toml by @ericmjl in #10736
- [Feat]: Configure LiteLLM to Parse User Headers from Open Web UI by @damgem in #9802
- Feat/support anonymize in aim guardrail by @hxdror in #10757
- Gemini Multimodal Live API support by @krrishdholakia in #10841
- Feat log retention cleanup by @jugaldb in #10872
- Model filter added on logs tab. (#10877) by @ishaan-jaff in #10891
- fix: fix novita ai docs by @jasonhp in #10882
- [Feat] Add Tracing for guardrails in StandardLoggingPayload, Langfuse by @ishaan-jaff in #10890
- [UI] - Add Guardrail Tracing to LiteLLM SpendLogs by @ishaan-jaff in #10893
- Fix example commands in reliability.md by @wwwillchen in #10894
- Feat: add guardrail tracing to OTEL, Arize phoenix by @ishaan-jaff in #10896
- Key hash filter added by @NANDINI-star in #10903
- [Feat] UI: Allow clicking into a guardrail by @ishaan-jaff in #10904
- [Feat] Add Lakera v2 Guardrail Support by @ishaan-jaff in #10880
- Key alias filter on logs by @NANDINI-star in #10905
- [Feat] handle multi-pod deployment for SpendLogs Retention by @jugaldb in #10895
- [UI] Fixes for using Lakera Guardrail on LiteLLM UI by @ishaan-jaff in #10906
- add constant to config settings.md by @jugaldb in #10908
- Remove extra test file by @jugaldb in #10889
- [UI] Allow editing guardrails by @ishaan-jaff in #10907
New Contributors
- @Dima-Mediator made their first contribution in #10789
- @igtm made their first contribution in #10814
- @shibaboy made their first contribution in #10752
- @camfarineau made their first contribution in #10629
- @ajac-zero made their first contribution in #10439
- @damgem made their first contribution in #9802
- @hxdror made their first contribution in #10757
- @wwwillchen made their first contribution in #10894
Full Changelog: v1.69.3-nightly...v1.70.0-nightly
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.70.0-nightly
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.70.0-nightly
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 190.0 | 206.64782881083894 | 6.289821899591554 | 0.0 | 1882 | 0 | 171.67403099995227 | 1154.6766310000294 |
| Aggregated | Passed ✅ | 190.0 | 206.64782881083894 | 6.289821899591554 | 0.0 | 1882 | 0 | 171.67403099995227 | 1154.6766310000294 |
Contributors
msabramo, shibaboy, and 15 other contributors