Skip to content

Commit 7f9d6b2

Browse files
SwordFaithzyzshishuiWANG-GHzhaochenyang20ocss884
authored and
huangjunyi.0
committed
[sglang] feat: Add SGLang async multi-turn rollout with tool support (volcengine#1037)
A redesigned version of volcengine#917 ## Current Status [Develop log & Tracker](zhaochenyang20/Awesome-ML-SYS-Tutorial#113) **What Has Been Done** - Async Rollout Refactoring: Integrate with the tool server to coordinate tool calls during generation, leveraging request IDs for state and progress tracking, support async multi-turn conversations in Agentic RL training (with Tool support). - Async Request Management: Encapsulate rollout requests into a unified structure, enabling efficient tracking and handling of concurrent multi-turn dialogues with chatml style messages. - Extensible Tools: A modular design for adapt tools in OpenAIFunctionTool format which is both support by SGLang and vLLM, with create separate instance, execute when tool call, calc score according to tool env state and release resource. - Multi-turn support has been implemented for the GSM8K task (new version working on). However, training has not yet converged, and we hope the community could join to investigate the issue. **What Is WIP** - [x] Merge loss mask to training process from last version - [x] Add more user friendly tool config and e2e tests for gsm8k with tool training - [ ] We are going to validate our multiturn feature in open-source sandbox environments. ## Key Features will be introduced in future version - Integrate a Ray-based agent trainer to enable explicit separation of the rollout and training pipeline. Provide support for partial rollout handling and fine-grained request state management. - Extend the framework to support simulated user interactions (e.g., roleplay, interactive feedback) and more complex environment-in-the-loop RL tasks. **Future Plan** [Discussion Thread](zhaochenyang20/Awesome-ML-SYS-Tutorial#74 (comment)) [RFC doc](https://github.com/SwordFaith/verl-sglang-dev-log/blob/main/rlhf/verl/multi-turn/veRL-multiturn-rollout-RFC.md) will be updated soon. ## Contributors & Acknowledgement - Xiang Long [[email protected]](mailto:[email protected]) @SwordFaith (Design RFC & core-dev of refactor part) - Yuzhen Zhou [[email protected]](mailto:[email protected]) @zyzshishui (Core-dev) - Chenyang Zhao [[email protected]](mailto:[email protected]) @zhaochenyang20 (PM) - Guanhua Wang @WANG-GH - Junrong Lin @ocss884 (verl-sglang support) - Hanchen Zhang [[email protected]](mailto:[email protected]) - Haoran Wang [[email protected]](mailto:[email protected]) - Rui Lu [[email protected]](mailto:[email protected]) - Yujiang Li [[email protected]](mailto:[email protected]) - Jiajun Li [[email protected]](mailto:[email protected]) - Jin Pan [[email protected]](mailto:[email protected]) - Zhi Zheng [[email protected]](mailto:[email protected]) @zh-zheng --------- Co-authored-by: zyzshishui <[email protected]> Co-authored-by: guanhua <[email protected]> Co-authored-by: zhaochenyang20 <[email protected]> Co-authored-by: ocss884 <[email protected]> Co-authored-by: Shawn/Yuxuan Tong <[email protected]> Co-authored-by: HL <[email protected]>
1 parent 7020eef commit 7f9d6b2

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+2432
-99
lines changed

‎.github/workflows/dataset.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,8 @@ jobs:
3333
env:
3434
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
3535
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
36-
NO_PROXY: "localhost,127.0.0.1"
36+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
37+
HF_ENDPOINT: "https://hf-mirror.com"
3738
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
3839
container:
3940
image: whatcanyousee/verl:ngc-cu124-vllm0.8.3-sglang0.4.5-mcore0.12.0-te2.2

‎.github/workflows/e2e_dapo.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,8 @@ jobs:
3636
env:
3737
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
3838
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
39-
NO_PROXY: "localhost,127.0.0.1"
39+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
40+
HF_ENDPOINT: "https://hf-mirror.com"
4041
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
4142
container:
4243
image: whatcanyousee/verl:ngc-cu124-vllm0.8.3-sglang0.4.5-mcore0.12.0-te2.2

‎.github/workflows/e2e_eval_aime24.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,8 @@ jobs:
3737
env:
3838
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
3939
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
40-
NO_PROXY: "localhost,127.0.0.1"
40+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
41+
HF_ENDPOINT: "https://hf-mirror.com"
4142
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
4243
container:
4344
image: whatcanyousee/verl:ngc-cu124-vllm0.8.3-sglang0.4.5-mcore0.12.0-te2.2

‎.github/workflows/e2e_ppo_trainer.yml‎

Lines changed: 66 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,8 @@ jobs:
6363
env:
6464
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
6565
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
66-
NO_PROXY: "localhost,127.0.0.1"
66+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
67+
HF_ENDPOINT: "https://hf-mirror.com"
6768
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
6869
container:
6970
image: whatcanyousee/verl:ngc-cu124-vllm0.8.3-sglang0.4.5-mcore0.12.0-te2.2
@@ -141,7 +142,8 @@ jobs:
141142
env:
142143
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
143144
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
144-
NO_PROXY: "localhost,127.0.0.1"
145+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
146+
HF_ENDPOINT: "https://hf-mirror.com"
145147
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
146148
container:
147149
image: hiyouga/verl:ngc-th2.6.0-cu126-vllm0.8.3-flashinfer0.2.2-cxx11abi0
@@ -174,7 +176,8 @@ jobs:
174176
env:
175177
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
176178
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
177-
NO_PROXY: "localhost,127.0.0.1"
179+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
180+
HF_ENDPOINT: "https://hf-mirror.com"
178181
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
179182
container:
180183
image: ocss884/verl-sglang:ngc-th2.6.0-cu126-sglang0.4.5.post3
@@ -195,14 +198,73 @@ jobs:
195198
ray stop --force
196199
ENGINE=sglang bash tests/e2e/ppo_trainer/run_function_reward.sh
197200
201+
e2e_ppo_trainer_sglang_async:
202+
runs-on: [L20x8]
203+
needs: pre_commit_for_ppo
204+
timeout-minutes: 40 # Increase this timeout value as needed
205+
env:
206+
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
207+
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
208+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
209+
HF_ENDPOINT: "https://hf-mirror.com"
210+
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
211+
container:
212+
image: ocss884/verl-sglang:ngc-th2.6.0-cu126-sglang0.4.5.post3
213+
options: --gpus all --shm-size=10g
214+
steps:
215+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
216+
with:
217+
fetch-depth: 0
218+
- name: Install the current repository
219+
run: |
220+
pip3 install -e .[test,gpu,sglang] --no-deps
221+
- name: Prepare gsm8k dataset
222+
run: |
223+
ray stop --force
224+
python3 examples/data_preprocess/gsm8k.py
225+
- name: Running GSM8K E2E training tests on 8 L20 GPUs with rmpad using function rm and save ckpt with sglang async
226+
run: |
227+
ray stop --force
228+
ENGINE=sglang_async bash tests/e2e/ppo_trainer/run_function_reward.sh
229+
230+
e2e_ppo_trainer_sglang_async_with_tool:
231+
runs-on: [L20x8]
232+
needs: pre_commit_for_ppo
233+
timeout-minutes: 40 # Increase this timeout value as needed
234+
env:
235+
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
236+
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
237+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
238+
HF_ENDPOINT: "https://hf-mirror.com"
239+
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
240+
container:
241+
image: ocss884/verl-sglang:ngc-th2.6.0-cu126-sglang0.4.5.post3
242+
options: --gpus all --shm-size=10g
243+
steps:
244+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
245+
with:
246+
fetch-depth: 0
247+
- name: Install the current repository
248+
run: |
249+
pip3 install -e .[test,gpu,sglang] --no-deps
250+
- name: Prepare gsm8k dataset with tool
251+
run: |
252+
ray stop --force
253+
python3 examples/data_preprocess/gsm8k_multiturn_w_tool.py --local_dir $HOME/data/gsm8k_verl_sgl_multi_turn_preprocessed
254+
- name: Running GSM8K with tool E2E training tests on 8 L20 GPUs with rmpad using function rm and save ckpt with sglang async
255+
run: |
256+
ray stop --force
257+
bash tests/e2e/run_gsm8k_fsdp_sgl_multiturn_w_tool.sh
258+
198259
e2e_ppo_trainer_sglang_vlm:
199260
runs-on: [L20x8]
200261
needs: pre_commit_for_ppo
201262
timeout-minutes: 40 # Increase this timeout value as needed
202263
env:
203264
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
204265
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
205-
NO_PROXY: "localhost,127.0.0.1"
266+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
267+
HF_ENDPOINT: "https://hf-mirror.com"
206268
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
207269
container:
208270
image: ocss884/verl-sglang:ngc-th2.6.0-cu126-sglang0.4.5.post3

‎.github/workflows/e2e_ppo_trainer_megatron.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,8 @@ jobs:
4444
env:
4545
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
4646
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
47-
NO_PROXY: "localhost,127.0.0.1"
47+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
48+
HF_ENDPOINT: "https://hf-mirror.com"
4849
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
4950
container:
5051
image: whatcanyousee/verl:ngc-cu124-vllm0.8.3-sglang0.4.5-mcore0.12.0-te2.2
@@ -82,7 +83,8 @@ jobs:
8283
env:
8384
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
8485
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
85-
NO_PROXY: "localhost,127.0.0.1"
86+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
87+
HF_ENDPOINT: "https://hf-mirror.com"
8688
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
8789
container:
8890
image: whatcanyousee/verl:ngc-cu124-vllm0.8.3-sglang0.4.5-mcore0.12.0-te2.2

‎.github/workflows/e2e_prime.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,8 @@ jobs:
3636
env:
3737
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
3838
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
39-
NO_PROXY: "localhost,127.0.0.1"
39+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
40+
HF_ENDPOINT: "https://hf-mirror.com"
4041
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
4142
container:
4243
image: whatcanyousee/verl:ngc-cu124-vllm0.8.3-sglang0.4.5-mcore0.12.0-te2.2

‎.github/workflows/e2e_sft.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,8 @@ jobs:
4242
env:
4343
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
4444
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
45-
NO_PROXY: "localhost,127.0.0.1"
45+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
46+
HF_ENDPOINT: "https://hf-mirror.com"
4647
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
4748
container:
4849
image: whatcanyousee/verl:ngc-cu124-vllm0.8.3-sglang0.4.5-mcore0.12.0-te2.2

‎.github/workflows/model.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,8 @@ jobs:
2929
env:
3030
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
3131
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
32-
NO_PROXY: "localhost,127.0.0.1"
32+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
33+
HF_ENDPOINT: "https://hf-mirror.com"
3334
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
3435
container:
3536
image: whatcanyousee/verl:ngc-cu124-vllm0.8.3-sglang0.4.5-mcore0.12.0-te2.2

‎.github/workflows/ray_test.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,8 @@ jobs:
3232
env:
3333
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
3434
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
35-
NO_PROXY: "localhost,127.0.0.1"
35+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
36+
HF_ENDPOINT: "https://hf-mirror.com"
3637
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
3738
container:
3839
image: whatcanyousee/verl:ngc-cu124-vllm0.8.3-sglang0.4.5-mcore0.12.0-te2.2

‎.github/workflows/sandbox.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,8 @@ jobs:
3131
env:
3232
HTTP_PROXY: ${{ secrets.PROXY_HTTP }}
3333
HTTPS_PROXY: ${{ secrets.PROXY_HTTPS }}
34-
NO_PROXY: "localhost,127.0.0.1"
34+
NO_PROXY: "localhost,127.0.0.1,hf-mirror.com"
35+
HF_ENDPOINT: "https://hf-mirror.com"
3536
HF_HUB_ENABLE_HF_TRANSFER: "0" # This is more stable
3637
container:
3738
image: whatcanyousee/verl:ngc-cu124-vllm0.8.3-sglang0.4.5-mcore0.12.0-te2.2

0 commit comments

Comments
 (0)