whisper: use vulkan as gpu backend when available #2302
Conversation
mstephenson6
commented
Jul 14, 2024
mstephenson6
commented
- When built with Vulkan support, attempt to initialize the ggml vk backend as a gpu
Signed-off-by: Matt Stephenson <[email protected]>
Signed-off-by: Matt Stephenson <[email protected]>
mstephenson6
changed the title
|
Hi, have you tested that it works correctly when Vulkan is enabled? |
|
@ggerganov - yes, so far just on an Apple M1Pro running Fedora, with the "Honeykrisp" Vulkan driver. I'm excited about this working, it's the very first GPU-accelerated ML I've been able to do on Linux + M1. Happy to test other GPU/OS combos too. |
| VK_LOG_DEBUG("ggml_vk_init(" << ctx->name << ", " << idx << ")"); | ||
| ggml_vk_instance_init(); | ||
| GGML_ASSERT(idx < vk_instance.device_indices.size()); |
There was a problem hiding this comment.
Is this change necessary? The original code works in https://github.com/ggerganov/llama.cpp, not sure why we would need to change it here
There was a problem hiding this comment.
I changed it so ggml_vk_instance_init() would populate device_indices right before the assertion, and it would avoid another vk-specific call in whisper.cpp. Without this, the assertion failed every time, no other code had attempted to initialize devices.
I didn't look at how llama.cpp did it, but I can check it out.
There was a problem hiding this comment.
I see, seems the change is good. cc @0cc4m to confirm that it's OK to move the assert after the init call
There was a problem hiding this comment.
Yeah, that's fine. Makes more sense that way, too.
* ggml: use vulkan as gpu backend when available Signed-off-by: Matt Stephenson <[email protected]> * whisper: enable using vk as default buffer type Signed-off-by: Matt Stephenson <[email protected]> --------- Signed-off-by: Matt Stephenson <[email protected]>
* ggml: use vulkan as gpu backend when available Signed-off-by: Matt Stephenson <[email protected]> * whisper: enable using vk as default buffer type Signed-off-by: Matt Stephenson <[email protected]> --------- Signed-off-by: Matt Stephenson <[email protected]>
