[feat] Support generic remote tokenizer

[ae86zhizhi]

Pull Request Description

This PR implements a generic remote tokenizer that communicates directly with inference engines' tokenizer APIs, ensuring perfect alignment between client-side token calculations and server-side processing.

Related Issues

Resolves: #1306

Changes

Core Implementation

• Added RemoteTokenizer interface and implementation.
• Implemented adapter pattern for different engines:
  • VLLMAdapter: Full implementation completed with complete tokenize/detokenize support.
  • SGLangAdapter: Basic completion endpoint support (foundation for future enhancement).
• Created a robust HTTPClient with:
  • Connection pooling (100 max idle, 10 per host).
  • Exponential backoff retry logic.
  • Proper resource cleanup (fixed response body leak).

vLLM Integration (Fully Implemented)

• ✅ /tokenize endpoint support with all options.
• ✅ /detokenize endpoint support.
• ✅ Model-specific tokenizer selection.
• ✅ Special tokens handling (add_special_tokens parameter).
• ✅ Full compatibility with vLLM's tokenizer API.

Error Handling

• Comprehensive error types for different scenarios.
• Graceful handling of network failures.

Backward Compatibility

• Maintains existing Tokenizer interface.
• No breaking changes to current implementations.
• Opt-in via configuration.

Testing

• Unit tests for all adapters.
• Integration tests with vLLM mock server.
• Retry and timeout scenarios tested.
• Error handling edge cases covered.
• All tests passing.

Performance Considerations

• Connection reuse reduces latency.
• Configurable timeouts prevent hanging requests.

Usage

// vLLM tokenizer - fully supported
config := RemoteTokenizerConfig{
    Engine:   "vllm",
    Endpoint: "http://vllm-server:8000",
    Model:    "meta-llama/Llama-2-7b-chat-hf",
}

tokenizer, err := NewRemoteTokenizer(config)
if err != nil {
    return err
}

// Use standard tokenizer interface
tokens, err := tokenizer.TokenizeInputText("Hello, world!")

// Or use advanced options
result, err := tokenizer.TokenizeWithOptions(ctx, TokenizeInput{
    Type:             ChatInput,
    Text:             "User message",
    AddSpecialTokens: true,
})

Checklist

• Code follows project style guidelines.
• Tests added/updated.
• Documentation updated (README.md included).
• Linting passes.
• No breaking changes.
• vLLM integration fully implemented and tested.

Future Work

• Response caching for frequent strings.
• Complete SGLang implementation.
• Additional engine support (TensorRT-LLM, etc.).

[gemini-code-assist]

Summary of Changes

Hello @ae86zhizhi, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances our application's tokenization capabilities by introducing a flexible and robust framework for interacting with remote Large Language Model (LLM) inference engines. The primary goal is to ensure precise token count alignment between the client and the server, which is crucial for accurate model interaction and efficient resource management. This change lays the groundwork for seamless integration with diverse LLM backends while preserving existing functionality.

Highlights

• Generic Remote Tokenizer Framework: I've implemented a new, extensible framework for remote tokenization. This allows direct communication with inference engine tokenizer APIs (e.g., vLLM, SGLang), ensuring perfect alignment between client-side token calculations and server-side processing.
• Engine Adapter Pattern: A flexible adapter pattern has been introduced via the EngineAdapter interface. This provides specific implementations for vLLM (with full support for tokenization, detokenization, and chat message handling) and a foundational adapter for SGLang, ready for future expansion.
• Robust HTTP Client: A new HTTPClient has been developed to handle remote API interactions. It includes essential features like connection pooling (100 max idle, 10 per host), exponential backoff retry logic, and proper resource cleanup to ensure efficient and reliable communication.
• Unified Tokenizer Interface & Backward Compatibility: Existing local tokenizers (tiktoken, character-based) have been refactored and integrated into a new factory-based NewTokenizer function. This maintains backward compatibility for existing Tokenizer interfaces and legacy vLLM configurations, ensuring a smooth transition.
• Comprehensive Error Handling & Documentation: Custom error types have been added for various failure scenarios, improving the clarity of error messages. Additionally, a detailed README.md has been included to thoroughly document the new package's architecture, usage, and extensibility.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a well-designed generic remote tokenizer, using an adapter pattern to support multiple inference engines. The refactoring of existing tokenizers, addition of a robust HTTP client, and comprehensive documentation and testing are all excellent. My feedback focuses on minor improvements to further enhance the code, specifically around idiomatic error handling, replacing magic numbers with constants for better maintainability, and clarifying a deprecation comment. Overall, this is a very strong contribution.

[Jeffwan]

seems the change scope is very clear and extensible. Let's quickly polish this one and merge it

[Jeffwan]

let's remove it. this is kind of confusing since the pr is the first version

[autopear]

unable to resolve my own comments?

[ae86zhizhi]

This PR has been updated with a comprehensive refactoring of the tokenizer package to improve code quality, simplify the API, and enhance maintainability. All commits now include DCO sign-off.

Summary of Changes

1. API Simplification

• Made internal implementation types unexported (extendedTokenizer, remoteTokenizer, engineAdapter, etc.) to minimize the public API surface.
• Removed deprecated TokenizerV2 interface alias - users should use ExtendedTokenizer directly.
• Removed the VLLM compatibility layer (compat_vllm.go) - users should migrate to RemoteTokenizer with Engine="vllm".
• Simplified the factory pattern by removing unnecessary abstraction layers.

2. Code Quality Improvements

• Updated all files with 2025 copyright year.
• Fixed receiver naming conventions (e.g., VLLMAdapter from 'a' to 'va').
• Addressed all linting issues (removed extra blank lines, added proper error handling).
• Simplified exponential backoff calculation without changing behavior.

3. Enhanced Features

• Improved HTTP status code handling with precise retry logic and Retry-After header support.
• Optimized health checks to use empty strings, minimizing server processing overhead.
• Better error handling for JSON encoding operations in tests.

4. Testing and Documentation

• Increased test coverage from 51.8% to 66.7%.
• Removed deprecated test files for removed components.
• Completely rewrote README.md to reflect the new simplified API.
• Added migration guide for users upgrading from the old API.
• Clarified that tokenizer.go is the main entry point, not deprecated.

Breaking Changes

• TokenizerV2 type alias removed (use ExtendedTokenizer).
• VLLM compatibility layer removed (use RemoteTokenizer with Engine="vllm").
• Many previously exported types are now unexported (use type assertions for advanced features).

Benefits

• Cleaner API: Minimal public surface following Go best practices.
• Better Maintainability: Less code, clearer structure.
• Improved Performance: Optimized health checks and retry logic.
• Enhanced Documentation: Clear migration path and usage examples.

The refactoring maintains full backward compatibility for the core Tokenizer interface while significantly improving the internal implementation.

[autopear]

See my comments for the revision

[autopear]

LGTM