Is there any support for AMD GPU (ROCM)

[mdrokz]

Hi i was wondering if there is any support for using llama.cpp with AMD GPU is there a ROCM implementation ?

[MichaelDays]

There’s a ROCm branch that hasn’t been merged yet, but is being maintained by the author.

https://github.com/ggerganov/llama.cpp/pull/1087/commits

[aiaicode]

Have you tried this? : https://github.com/ggerganov/llama.cpp#clblast

To check if you have CUDA support via ROCm, do the following :

$ python
import torch
torch.cuda.is_available()
Output :
True or False

If it's True then you have the right ROCm and Pytorch installed and things should work. At least for Stable diffusion that's how you check and make it work.

If it's False then you need to check if your GPU has CUDA support or not. You can see the AMD OpenCL supported devices list here : https://en.wikipedia.org/wiki/OpenCL#Devices

[mdrokz]

There’s a ROCm branch that hasn’t been merged yet, but is being maintained by the author.

https://github.com/ggerganov/llama.cpp/pull/1087/commits

oh any idea why it hasnt been merged yet ?

[mdrokz]

Have you tried this? : https://github.com/ggerganov/llama.cpp#clblast

To check if you have CUDA support via ROCm, do the following :

$ python import torch torch.cuda.is_available() Output : True or False

If it's True then you have the right ROCm and Pytorch installed and things should work. At least for Stable diffusion that's how you check and make it work.

If it's False then you need to check if your GPU has CUDA support or not. You can see the AMD OpenCL supported devices list here : https://en.wikipedia.org/wiki/OpenCL#Devices

llama.cpp doesnt use torch as its a custom implementation so that wont work and stable diffusion uses torch by default and torch supports rocm.

[SlyEcho]

oh any idea why it hasnt been merged yet ?

Soon, I just haven't had time recently to work on it.

oh any idea why it hasnt been merged yet ?

Soon, I just haven't had time recently to work on it.

Which work needs to be done (except for Windows support, maybe)? Your version is working fine here with my 6650 XT and Linux.

[SlyEcho]

I want to add some CI checks to see if it compiles so if the CUDA code is updated it would not break (well, at least not break the build).

Then some small tweaks, like having the UI say "ROCm" or "HIP" instead of "CUDA".

[shibe2]

AMD GPUs are supported through CLBlast. In my experience, ROCm is much more problematic than OpenCL. I recommend going with CLBlast, unless you get better performance with another option or for some specific reason.

[mdrokz]

AMD GPUs are supported through CLBlast. In my experience, ROCm is much more problematic than OpenCL. I recommend going with CLBlast, unless you get better performance with another option or for some specific reason.

Oh wow i didnt know that i will try CL blast. do you know the performance difference between CLBlast and ROCM ?

For me, ROCm is much faster compared to CLBlast. And I don't see any reasons to not use ROCm (at least when we speak about Linux, ROCm for Windows is still really new). IF the hardware/os is supported, which is the only downside right now. There is a comparison between ROCm and CLBlast here, but I think it is a bit outdated:

https://github.com/YellowRoseCx/koboldcpp-rocm/

It's for koboldcpp, but this uses llama.cpp.

[SlyEcho]

The prompt evaluation is much faster with ROCm. If this could be optimized, I'm sure they could be similar. The only downside to OpenCL is that the memory management is not as advanced as it is in ROCm/CUDA.

I would keep an eye on the Vulkan version in #2059, it has a lot of promise to support much wider set of devices.

[mdrokz]

For me, ROCm is much faster compared to CLBlast. And I don't see any reasons to not use ROCm (at least when we speak about Linux, ROCm for Windows is still really new). IF the hardware/os is supported, which is the only downside right now. There is a comparison between ROCm and CLBlast here, but I think it is a bit outdated:

https://github.com/YellowRoseCx/koboldcpp-rocm/

It's for koboldcpp, but this uses llama.cpp.

which GPU did you use ? i cant get CLblast to run on my RX 6700 XT, also is opencl through rocm supported ?

[shibe2]

Although OpenCL and ROCm are different APIs, OpenCL driver for Radeon RX 6xxx is based on ROCm code (see AMD CLR). CLBlast supports Radeon RX 6700 XT out of the box with the default driver on Linux.

@mdrokz You need to make sure that OpenCL is working properly on your system. Try clinfo and other software that uses OpenCL. Also make sure that llama.cpp is compiled with CLBlast.