Possible solution to allow K-quants on models with n_vocab!=32000#2148

Merged

LostRuins merged 3 commits intoggml-org:masterfrom

LostRuins:kquant_vocab_fix

Jul 11, 2023

Commits on Jul 8, 2023

This allows LLAMA models that were previously incompatible with K quants to function mostly as normal. This happens when a model has a vocab != 32000, e.g 32001 which means it's not divisible by 25…
LostRuins
committed

Commits on Jul 10, 2023

Fix indentation

LostRuins
and
ggerganov
authored
As an alternative, to avoid failing on Metal due to lack of Q8_0 support, instead quantize tok_embeddings.weight to Q4_0 and retain output.weight as F16. This results in a net gain of about 55mb fo…
LostRuins
committed