[Feature] lora serving performance

[MichoChan]

lora reasoning speed is very slow, I ran a gemma's lora, found that qkv proj takes 0.0003s, but without lora only 0.0001s, so the result is a token decode time difference of 20ms+

however, vllm lora serving is faster

[Fridge003]

@MichoChan Hi, we updated Lora with a triton backend recently in #3161. It should improve the serving performance by setting --lora-backend argument to triton.

In the future we will support Cuda/Cutlass backend which can further improve the performance

[Qiaolin-Yu]

@Fridge003 Hi, I would like to work on this. Could you assign this to me 👀

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.