[Misc] Make cached tokenizer pickle-compatible

[DarkLight1337]

This is required to support running multi-modal processor in parallel using multi-processing (which is WIP).

I also changed all_special_tokens and similar methods to return a list instead of a set, since that's how it is in the HF-defined classes. Not sure why they were converted into sets in the first place in #2879.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[njhill]

I also changed all_special_tokens and similar methods to return a list instead of a set, since that's how it is in the HF-defined classes. Not sure why they were converted into sets in the first place in #2879.

@DarkLight1337 I think these are sets intentionally because we check whether they contain particular tokens and want that to be fast.

Do you have more context as to how/why this is needed for the MM processor multiprocessing? We are not going to be transferring tokenizers between procs at runtime right?

[DarkLight1337]

Do you have more context as to how/why this is needed for the MM processor multiprocessing? We are not going to be transferring tokenizers between procs at runtime right?

We need to transfer the tokenizers between processes because multi-modal processor calls the tokenizer during processing.

[DarkLight1337]

I think these are sets intentionally because we check whether they contain particular tokens and want that to be fast.

Based on a quick Ctrl-F, the only place where we actually check the attribute directly is sample_tokens inside benchmarks/benchmark_prefix_caching.py. Edit: I have updated that function to construct a new set before iterating through each token.

While _adapt_tokenizer in vllm/model_executor/guided_decoding/outlines_logits_processors.py and _convert_tokens_to_string_with_added_encoders in vllm/transformers_utils/detokenizer_utils.py both check whether a token is contained in tokenizer.all_special_tokens, they make a new set from tokenizer.all_special_tokens, so there is no need to create a new set inside the tokenizer itself.

[njhill]

@DarkLight1337 sorry for delay looking at this.

On the set thing, is there any harm in us continuing to return sets for these? It just seems "safer" to me in case lookups are done in future.

[njhill]

Why can't we modify in-place as was done before?

[DarkLight1337]

I need the original tokenizer to be passed to __reduce__. Otherwise there may be infinite recursion

[DarkLight1337]

On the set thing, is there any harm in us continuing to return sets for these? It just seems "safer" to me in case lookups are done in future.

@hmellor do you know why those are lists in HF?

IMO given that we type hint this attribute as being a list in TokenizerBase, our code should take this into account and assume that the attribute is a list.

[hmellor]

do you know why those are lists in HF?

I don't unfortunately. My best guesses would be some combination of:

• The tokenizer is stored as JSON which has no concept of sets
• We want them to be mutable (in vLLM we don't care if they're mutable though)

[njhill]

Thanks @DarkLight1337, looks much better to me!