[Bug]: serverRootPath Configuration Not Working for LiteLLM UI Assets in restricted environment

[mateo-di]

What happened?

Summary

I'm experiencing issues with the serverRootPath configuration in LiteLLM. The LiteLLM application starts successfully and the API endpoints work correctly, but the UI assets (JavaScript files) are returning 404 errors because they're not being served from the correct path.

The tests are done with port forwarding

Environment Details

• Kubernetes Environment: Google Kubernetes Engine (GKE)
• LiteLLM Image: ghcr.io/berriai/litellm-non_root:main-latest
• Deployment Method: Helm chart deployment
• Base Path: /litellm (configured via serverRootPath)

Current Configuration

Environment Variables

data:
  SERVER_ROOT_PATH: "/litellm"
  PORT: "4000"
  # ... other configuration variables

Deployment Configuration

# Container Configuration
image: ghcr.io/berriai/litellm-non_root:main-latest
command: ["/bin/sh"]
args: ["-c", "exec litellm --host 0.0.0.0 --port 4000"]

# Health Checks
livenessProbe:
  httpGet:
    path: /health/liveliness
    port: http
readinessProbe:
  httpGet:
    path: /health/readiness
    port: http

Issue Details

Working Endpoints

The following endpoints work correctly with the serverRootPath configuration:

• GET /litellm/ - Main UI page loads successfully
• GET /litellm/openapi.json - OpenAPI specification
• GET /health/liveliness - Health check endpoint
• GET /health/readiness - Readiness check endpoint

Failing Asset Requests

The following UI asset requests are returning 404 errors:

GET /litellm-asset-prefix/_next/static/chunks/webpack-a426aae3231a8df1.js HTTP/1.1" 404 Not Found
GET /litellm-asset-prefix/_next/static/chunks/fd9d1056-205af899b895cbac.js HTTP/1.1" 404 Not Found
GET /litellm-asset-prefix/_next/static/chunks/117-c4922b1dd81b62ce.js HTTP/1.1" 404 Not Found
GET /litellm-asset-prefix/_next/static/chunks/main-app-475d6efe4080647d.js HTTP/1.1" 404 Not Found

Expected vs Actual Behavior

Expected: UI assets should be served from /litellm/_next/static/...
Actual: UI assets are being requested from /litellm-asset-prefix/_next/static/...

Application Logs

INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)
INFO: 127.0.0.1:38074 - "GET /litellm/ui/ HTTP/1.1" 200 OK
INFO: 127.0.0.1:38074 - "GET /litellm-asset-prefix/_next/static/chunks/webpack-a426aae3231a8df1.js HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:38074 - "GET /litellm-asset-prefix/_next/static/chunks/fd9d1056-205af899b895cbac.js HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:38074 - "GET /litellm-asset-prefix/_next/static/chunks/117-c4922b1dd81b62ce.js HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:38074 - "GET /litellm-asset-prefix/_next/static/chunks/main-app-475d6efe4080647d.js HTTP/1.1" 404 Not Found

Root Cause Analysis

The issue appears to be that:

1. API Endpoints: The serverRootPath configuration correctly affects API endpoints (/litellm/openapi.json, /litellm/ui/)
2. Static Assets: The static assets (JavaScript files) are being requested with a different path prefix (/litellm-asset-prefix/) instead of using the configured serverRootPath
3. Asset Path Mismatch: The frontend is generating asset URLs that don't match the configured base path

Testing and Validation

Working Requests

# These work correctly
curl http://localhost:4000/litellm/
curl http://localhost:4000/litellm/openapi.json
curl http://localhost:4000/health/liveliness

Failing Requests

# These return 404
curl http://localhost:4000/litellm-asset-prefix/_next/static/chunks/webpack-a426aae3231a8df1.js
curl http://localhost:4000/litellm-asset-prefix/_next/static/chunks/fd9d1056-205af899b895cbac.js

Potential Solutions

1. Frontend Asset Path Configuration: The frontend needs to be configured to use the correct asset path prefix
2. Static File Serving: Ensure static files are served from the correct path
3. Asset Prefix Environment Variable: There might be a separate environment variable for asset prefix configuration
4. Build-time Configuration: The asset paths might need to be configured at build time

Request for Assistance

I would appreciate guidance on:

1. Correct Configuration: What is the proper way to configure serverRootPath for both API endpoints and static assets?
2. Asset Prefix: Is there a separate configuration for static asset paths?
3. Environment Variables: Are there additional environment variables needed for proper asset serving?
4. Documentation: Is there documentation for deploying LiteLLM with custom base path?

Additional Context

• The application is deployed in a Kubernetes environment with ingress routing
• The API functionality works correctly with the current configuration
• The issue is specifically with the UI static assets
• The same configuration pattern works for other applications in the same environment

Environment Information

• Kubernetes Version: GKE 1.28+
• Base Path: /litellm
• Port: 4000 (internal)
• External Access: Through ingress at https://domain.com/litellm/

Expected Behavior

When accessing https://domain.com/litellm/, the UI should load completely with all assets (JavaScript, CSS) being served from the correct paths under /litellm/.

---

Note: This issue is separate from the previous Prisma connection issue and occurs after the application has successfully started and is running.

Full -f logs:

2025-06-18 18:12:26,460 - litellm_proxy_extras - INFO - Running prisma migrate deploy
2025-06-18 18:12:39,692 - litellm_proxy_extras - INFO - prisma migrate deploy stdout: Installing Prisma CLI
Prisma schema loaded from schema.prisma
Datasource "client": PostgreSQL database "litellm", schema "public" at "<hidden>:5432"

24 migrations found in prisma/migrations


No pending migrations to apply.

2025-06-18 18:12:39,692 - litellm_proxy_extras - INFO - prisma migrate deploy completed
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)

#------------------------------------------------------------#
#                                                            #
#              'I don't like how this works...'               #
#        https://github.com/BerriAI/litellm/issues/new        #
#                                                            #
#------------------------------------------------------------#

 Thank you for using LiteLLM! - Krrish & Ishaan



Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new


INFO:     10.2.0.49:41234 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:41236 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:55852 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:55854 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:46420 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:46430 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:54730 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:54722 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:45852 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:45844 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:50428 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:50430 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:56868 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:56854 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:41098 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:41092 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:45580 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:45596 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:60706 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:60712 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:36524 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:36522 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:32770 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:32778 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:33458 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:33460 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:53654 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:53666 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:51804 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:51802 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:52070 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:52068 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:38962 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:38960 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:53822 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:53830 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:40408 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:40410 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:48486 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:48484 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:41488 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:41498 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:44312 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:44302 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     127.0.0.1:56220 - "GET / HTTP/1.1" 200 OK
INFO:     127.0.0.1:56220 - "GET /litellm/openapi.json HTTP/1.1" 200 OK
INFO:     127.0.0.1:38074 - "GET /litellm/ui/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:38074 - "GET /litellm-asset-prefix/_next/static/chunks/webpack-a426aae3231a8df1.js HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:38084 - "GET /litellm-asset-prefix/_next/static/chunks/fd9d1056-205af899b895cbac.js HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:38108 - "GET /litellm-asset-prefix/_next/static/chunks/117-c4922b1dd81b62ce.js HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:38098 - "GET /litellm-asset-prefix/_next/static/chunks/main-app-475d6efe4080647d.js HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:38098 - "GET /favicon.ico HTTP/1.1" 404 Not Found
INFO:     10.2.0.49:52536 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:52546 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:57880 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:57872 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     127.0.0.1:45330 - "GET /.well-known/appspecific/com.chrome.devtools.json HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:45330 - "GET /litellm/ui/ HTTP/1.1" 304 Not Modified
INFO:     127.0.0.1:45330 - "GET /litellm-asset-prefix/_next/static/chunks/webpack-a426aae3231a8df1.js HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57422 - "GET /litellm-asset-prefix/_next/static/chunks/fd9d1056-205af899b895cbac.js HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57464 - "GET /.well-known/appspecific/com.chrome.devtools.json HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57454 - "GET /litellm-asset-prefix/_next/static/chunks/117-c4922b1dd81b62ce.js HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57438 - "GET /litellm-asset-prefix/_next/static/chunks/main-app-475d6efe4080647d.js HTTP/1.1" 404 Not Found
INFO:     10.2.0.49:55618 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:55630 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.49:41176 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.49:41172 - "GET /health/readiness HTTP/1.1" 200 OK

Relevant log output

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

main-latest

Twitter / LinkedIn details

@mdiloreto

[krrishdholakia]

This is a duplicate of an existing issue already filed.

[krrishdholakia]

The api is being served. The swagger docs don't load.

[mateo-di]

Hi @krrishdholakia i have tested this using litellm-non_root with tag 'main-latest' and 'main-rc' as of today and still getting an the same issue with the following message:

➜ k logs carto-notdelete-litellm-767b5bc6c5-29mx5 -f                                         
2025-06-23 14:40:33,654 - litellm_proxy_extras - INFO - Running prisma migrate deploy
2025-06-23 14:40:47,091 - litellm_proxy_extras - INFO - prisma migrate deploy stdout: Installing Prisma CLI
Prisma schema loaded from schema.prisma
Datasource "client": PostgreSQL database "litellm", schema "public" at "<hidden>:5432"

25 migrations found in prisma/migrations

Applying migration `20250618225828_add_health_check_table`

The following migration have been applied:

migrations/
  └─ 20250618225828_add_health_check_table/
    └─ migration.sql
      
All migrations have been successfully applied.

2025-06-23 14:40:47,092 - litellm_proxy_extras - INFO - prisma migrate deploy completed
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)

#------------------------------------------------------------#
#                                                            #
#               'A feature I really want is...'               #
#        https://github.com/BerriAI/litellm/issues/new        #
#                                                            #
#------------------------------------------------------------#

 Thank you for using LiteLLM! - Krrish & Ishaan



Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new


INFO:     10.2.0.13:59398 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.13:59392 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.13:60494 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.13:60504 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.13:37470 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.13:37480 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.13:60612 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.13:60614 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.2.0.13:50370 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.13:50376 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     127.0.0.1:55382 - "GET / HTTP/1.1" 200 OK
INFO:     127.0.0.1:55382 - "GET /litellm/openapi.json HTTP/1.1" 200 OK
INFO:     10.2.0.13:46584 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.2.0.13:46596 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     127.0.0.1:55392 - "GET /_next/static/chunks/fd9d1056-205af899b895cbac.js HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:55382 - "GET /_next/static/chunks/webpack-a426aae3231a8df1.js HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:37778 - "GET /_next/static/chunks/117-c4922b1dd81b62ce.js HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:37782 - "GET /_next/static/chunks/main-app-475d6efe4080647d.js HTTP/1.1" 404 Not Found

[krrishdholakia]

INFO: 127.0.0.1:55392 - "GET /_next/static/chunks/fd9d1056-205af899b895cbac.js HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:55382 - "GET /_next/static/chunks/webpack-a426aae3231a8df1.js HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:37778 - "GET /_next/static/chunks/117-c4922b1dd81b62ce.js HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:37782 - "GET /_next/static/chunks/main-app-475d6efe4080647d.js HTTP/1.1" 404 Not Found

These requests look wrong. I'm confused why the server root path isn't being appended here.

Will add some more debug logs - so we can understand what's happening @mateo-di

Are you able to test dev releases?

[mateo-di]

Hi @krrishdholakia, yes. Let me know which tag should i use for the troubleshooting and i can re-run and test. currently im using ghcr.io/berriai/litellm-non_root:main-latest

[mateo-di]

Hi @krrishdholakia let me know how i can help you move forward with this!

[WilsonSunBritten]

Also experiencing this still with the latest nightly build for docker.