Inference multiple tokens

[PlayerSAL]

Thank you for your great work! I have a question about inference with multiple tokens. For instance, I have 10k tokens to inference and I want to process them in batches. Every time handle 10 tokens and process for 1000 times. Is it possible to make mamba work this way?

[tridao]

Probably yes.
How would you do it with Transformers?

[AnaRhisT94]

I already implemented decoding N tokens in a batch instead of one token every time.
@PlayerSAL #477
@tridao