Skip to content

[Fix] weights update for deepseek when enabling cuda graph #4961

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

hxdtest
Copy link

@hxdtest hxdtest commented Apr 1, 2025

Motivation

Fix weights update for deepseek when enabling cuda graph.
If model weights is updated using load_weightsafter launching engine,the address of self_attn.w_kc and self_attn.w_vc would change, which conflicts with cuda graph.

Modifications

Checklist

@hxdtest hxdtest changed the title Fix weights update for deepseek when enabling cuda graph [Fix] weights update for deepseek when enabling cuda graph Apr 1, 2025
@merrymercy
Copy link
Contributor

can @zhaochenyang20 @fzyzcjy take a look?

@zhaochenyang20
Copy link
Collaborator

@hxdtest could you rebase with main?

@fzyzcjy could you help to take a look?

Copy link
Collaborator

@fzyzcjy fzyzcjy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, though I wonder whether we can make such param same as other params, i.e. be not-none when creating the module (I guess not trivial though).

@merrymercy
Copy link
Contributor

Can you resolve the conflicts?

@merrymercy merrymercy added the ready-to-merge The PR is ready to merge after the CI is green. label Apr 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready-to-merge The PR is ready to merge after the CI is green.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants