Feature request: AMD GPU support with oneDNN AMD support

[santhoshtr]

Hi, CTranslate2 uses oneDNN. oneDNN latest versions has support for AMD GPU. It require Intel oneAPI DPC++. The same approach can potentially enable NVIDIA GPU support too.

It would help running the MT models on AMD GPUs. With Rocm, this would be a full opensource way to run MT models in GPUs.

Thanks

[guillaumekln]

Hello,

Currently we only use oneDNN for specific operators such as matrix multiplications and convolutions, but a full MT models contains many other operators (softmax, layer norm, gather, concat, etc.). Even though some of them are available in oneDNN, it would require quite some work to specialize all operations for AMD GPUs.

At this time I don't plan to work on this feature, but it would indeed be a nice one to have!

[leuc]

I wanted to try faster whisper on a Intel A770 dGPU 16GB. A complete use of oneDNN could also enable that hardware support.

[towel]

Migrating a transcription component to faster-whisper, and using an AMD GPU, I'd also appreciate faster-whisper with ROCm support even more.

[phineas-pta]

@towel did you manage to get faster-whisper working on AMD ?

[CristianPi]

any way to run with an amd gpu?

[MidnightKittenCat]

Any update on this?

[guillaumekln]

I still don't plan to work on this at this time, and as far as I know no one else is working on this. I expect it would be quite some work to have a complete ROCm support.

[lenhone]

I had a go at converting the existing cuda stuff to rocm a few months ago but could never get it to build, not surprising as I have zero C++ or cmakelists skills.

curand, cublas, cudnn, cuda, cub appear to map to hip with minor adjustments, but I could never get the cmakelists to include thrust (the version supplied by rocm) and it always halted compiling due to producing too many errors.

[TheJKM]

I started trying to port CTranslate2 to ROCm last weekend and decided to share my (non-working) results here. The code is available in the rocm Branch of my fork.

Basically, hipify was able to convert most of the code automatically. I added a new CMake config option to enable compiling with ROCm, and so far calling the HIP compiler works, however it breaks the other options and requires a CMake version new enough to have HIP language support.

Current issues are some CUDA library dependencies I did not look at yet, and the use of bfloat16 data type. While latest ROCm has a (according to this GH issue -> ROCm/ROCm#2534) drop-in replacement for the CUDA bf16, it currently has some issues in missing operators. Therefore, I'm trying to completely disable bf16 for now, but without luck so far.

This work has right now just the goal of making it work, and not integrating HIP/ROCm into the (CMake) infrastructure.

In case someone wants to have a look at the code and help porting, feel free to look at my fork. Unfortunately, I don't expect to have much time in the near future for this project.

[BBC-Esq]

This is awesome dude. Wish I had programming experience to help with this, but alas I don't. I've been looking for ways to enable gpu acceleration for amd gpus using ctranslate2...Let me know if I can help in any way, whether it be by testing or what have you.

[BBC-Esq]

Have you gotten it to work at all yet?