Feature Request: Hexagon FP8 weight support and DiT optimizations for HTP v79+ #28904

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

[Hexagon] FP8 weight support and DiT-oriented optimizations for HTP v79+

Motivation

We have implemented and validated an FP8 inference path for diffusion transformers on Qualcomm Hexagon HTP v79+ devices, including Snapdragon 8 Elite / SM8750.

The goal of this issue is to discuss how to upstream the generic pieces into llama.cpp incrementally, particularly the FP8 type plumbing, Hexagon weight repacking, and HMX FP8 GEMM support.

Implementation

The current v79+ path includes:

• Loading standard F8_E4M3 weights from safetensors.
• One-time set_tensor repacking into the HMX 32x32 tiled weight layout.
• FP8 W8A16 GEMM using HMX.
• Fused Q/K RMSNorm and RoPE.
• A generic segmented-K matrix multiplication used by FLUX.2/Klein linear2 in single stream blocks.
• FP16 VAE activations.
• Implicit-GEMM convolution.
• Fused GroupNorm, affine transform, and SiLU.
• Fused upscale and convolution.

This enables complete on-device text encoding, DiT sampling, and VAE decoding at resolutions from 1024x1024 through 2048x2048.

FP8 HMX execution

The FP8 GEMM uses the v79 HMX:

• activation.hf
• weight.f8

instruction path with FP32 accumulation.

The HMX operation:

cvt.hf = acc(2)

applies the architectural 1/256 conversion when storing the accumulator as FP16.

The HVX output epilogue compensates by multiplying by 256. The same FP32 epilogue also folds in:

• The FP32 scale associated with the model's FP8 weight tensor.
• Any scalar multiplication represented by the surrounding GGML graph.

This avoids an additional tensor pass and adds negligible overhead to the GEMM store stage.

Results and integration

The stable-diffusion.cpp integration, benchmark comparison, commands, and image-quality examples are available here:

leejet/stable-diffusion.cpp#1970