Skip to content

Remove chunked prefill length divide for DP #5663

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from

Conversation

jokerwyt
Copy link
Contributor

Motivation

Modifications

After removing that division, my test on DeepSeek-R1 ran smoothly for about avg 15K, max 30K input benchmark, DP=4 and TP=16, chunksz 32768.
Call for more test for it.

Related #4390

Checklist

@merrymercy
Copy link
Contributor

merrymercy commented Apr 27, 2025

I think we still need this division. This is because we want to make the activation memory the same for TP=8 and DP=8. For your use case, can you manually adjust --chunked-prefill-size when you launch the server?

@merrymercy merrymercy closed this Apr 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants