[Bug] Batched greedy generation disagrees with one-at-a-time on Qwen3.5-2B and gemma-4-E2B-it (T4, left padding)

[danielhanchen]

Summary

After a short LoRA fine-tune, greedy generation at batch size 1 and at batch sizes 2, 4 and 8 produce different text for the same prompts, with left padding in place and the padding demonstrably happening.

Reproduces on two different models, on a Tesla T4 (sm_75), across two fresh processes each. This is the same family as #1066 and #3699, on a current stack.

Environment

torch          2.10.0+cu128      transformers  5.15.1
trl            1.10.0            peft          0.19.1
accelerate     1.13.0            bitsandbytes  0.50.1
datasets       5.0.1             triton        3.6.0
GPU            Tesla T4, sm_75   load_in_4bit  True, dtype float16

Reproduction

Eight prompts of deliberately different lengths, greedy (do_sample=False), generated one at a time and then in batches, decoded past the prompt width.

unsloth/gemma-4-E2B-it

prompt token lengths   [13, 13, 14, 15, 13, 12, 15, 16]   (5 distinct, so padding happened)
tokenizer.padding_side after generating: left

batch 1:  ['__UNSLOTH__!!!'] x 8

batch 2:  ['__UNSLOTH__!!!',
           '__UNSLOTH__!!!',
           ':\n### Answer:\n### Answer:\n__UNSLOTH__!!!',   <-- differs
           '__UNSLOTH__!!!',
           '__UNSLOTH__!!!',
           ':\n__UNSLOTH__UNSLOTH!!!',                      <-- differs
           '?\n### Answer:\n### Answer?\n__UNSLOTH__UN',    <-- differs
           '__UNSLOTH__!!!']

unsloth/Qwen3.5-2B

prompt token lengths   [13, 13, 14, 15, 13, 13, 15, 16]   (4 distinct)
tokenizer.padding_side after generating: left

batch 1:  ['__UNSLOTH__!!!'] x 8

batch 2:  ['__UNSLOTH__!!!',
           '__UNSLOTH__!!!',
           '?UNSLOTH__UNSLOTH__UNSLOTH__UNSLOTH',           <-- differs
           '__UNSLOTH__!!!',
           '__UNSLOTH__!!!',
           '__UNSLOTH__!!!',
           ' I__UNSLOTH__UNSLOTH__UNSLOTH__UNSLO',          <-- differs
           '__UNSLOTH__!!!']

Batch 4 and batch 8 disagree at least as much, and both repeats of each model agree with each other, so this is reproducible rather than noise.

What has already been ruled out

• Padding really happens. distinct_lengths is 5 and 4, so the batches are genuinely ragged. An earlier run of this harness reported every prompt as 1 token because of a bug on my side, and that run is not the evidence above.
• Padding side is left, read back off the tokenizer after generating rather than merely set beforehand (which is the shape of Newest Unsloth version silently FORCES Qwen2VL tokenizer padding side to right in inference, while training is left #2138).
• The models train fine. Losses fall, the adapter updates, and batch-1 output is the learned canary exactly.

The caveat, stated up front

Some divergence between batched and unbatched fp16 greedy decoding is possible from ordinary numerics: a near-tie in the argmax can flip when the batch changes the reduction order. That would normally show as an occasional single-token difference.

What is here is larger than that. The differing outputs gain a whole prefix (:\n### Answer:\n### Answer:\n, ?UNSLOTH__UNSLOTH__), which is what attending to padding looks like rather than a tie-break flip. The same positions differ in both repeats, and the same two models differ at the same prompt indices.

Where it was found

The Kaggle T4 CI legs, which assert the #3699 / #1066 / #1456 / #2138 contract on every run. Note that unsloth/Qwen3-0.6B passes the identical check on the same harness (batch 2, 4 and 8 all agreeing with batch 1), so this is not the harness being wrong about every model.