rocBLAS support

[daniandtheweb]

Since Nvidia's cuBLAS support has been added it is possible to implement AMD's rocBLAS support as well? It would make this the first llama project with official support for AMD gpus acceleration.

[SlyEcho]

I have a PR in draft #1087 for hipBLAS and HIP. If you can test, it would be great.

Probably not on Windows, though.

[daniandtheweb]

Is there any specific configuration to make it work? I built it with HIPBLAS and CUBLAS but it doesn't seem to work

[SlyEcho]

I have a Vega64 and I'm using Arch Linux. I have these packages:

[henri@taichi llama.cpp]$ pacman -Q | grep rocm
rocm-clang-ocl 5.4.3-1
rocm-cmake 5.4.3-1
rocm-core 5.4.3-4
rocm-device-libs 5.4.3-1
rocm-hip-libraries 5.4.3-2
rocm-hip-runtime 5.4.3-2
rocm-hip-sdk 5.4.3-2
rocm-language-runtime 5.4.3-2
rocm-llvm 5.4.3-1
rocm-opencl-runtime 5.4.3-1
rocm-opencl-sdk 5.4.3-2
rocm-smi-lib 5.4.3-1
rocminfo 5.4.3-1
[henri@taichi llama.cpp]$ pacman -Q | grep hip
hip-runtime-amd 5.4.3-1
hipblas 5.4.3-1
hipcub 5.4.3-1
hipfft 5.4.3-1
hipsolver 5.4.3-1
hipsparse 5.4.3-1
miopen-hip 5.4.3-1
rocm-hip-libraries 5.4.3-2
rocm-hip-runtime 5.4.3-2
rocm-hip-sdk 5.4.3-2
[henri@taichi llama.cpp]$ pacman -Q | grep amd
hip-runtime-amd 5.4.3-1
hsa-amd-aqlprofile-bin 5.4.3-2

Can you tell me what kind of issue do you have? Problem compiling, some kind of error? What device, OS?

[daniandtheweb]

I'm on Arch using a Radeon 5700XT. The problem is that the program doesn't use the GPU at all. The compilation works as expected and I have every hip and rocm package needed installed

[SlyEcho]

Does it say "BLAS = 1" when you start it?

[daniandtheweb]

Nope, 0

[SlyEcho]

That means it was not compiled with the right options. It will not configure any kind of acceleration automatically, it has to be enabled from the make/cmake command line.

How did you compile it?

[daniandtheweb]

Nevermind, I rebuilt it correctly with make -j8 LLAMA_HIPBLAS=1 and the program reports BLAS=1 however the gpu isn't used at all

[SlyEcho]

What size of batch did you use? The default is 8 which will make it skip any BLAS calls.

Try -b 512 when running ./main

[daniandtheweb]

Still 0 GPU usage

[daniandtheweb]

The only difference between the official master branch and yours is that load time on the official one is lower (490 ms vs 697 ms)

[SlyEcho]

Interesting, if it is loading longer, then it may be actually loading the hipBLAS libraries, I noticed it can take some time.

Can you try building and running it in Docker? I have some not-so-exact commands up there in the #1087 description.

[daniandtheweb]

I set up the docker and cloned your repo inside, recompiled and still the same 0% GPU usage.

[SlyEcho]

How are you checking the GPU usage? Because if tried to use ROCm and your card wasn't supported, the program would crash.

Can you also check the libraries linked to main with ldd main and see if there you see anything from /opt/rocm?