YOLO-WORLD-S在coco上finetune无法复现，且validation map呈现下降趋势

[shupinghu]

【复现步骤】
你好，我下载了yolo_world_s_clip_base_dual_vlpan_2e-3adamw_32xb16_100e_o365_goldg_train_pretrained-18bea4d2.pth，加载它进行预训练，config文件是从configs/finetune_coco/yolo_world_l_dual_vlpan_2e-4_80e_8gpus_finetune_coco.py拷贝后修改的，即将_base_里面的yolov8_l_syncbn_fast_8xb16-500e_coco.py直接修改为yolov8_s_syncbn_fast_8xb16-500e_coco.py，然后在yolov8_s_syncbn_fast_8xb16-500e_coco.py里增加了“mixup_prob = 0.1”以解决mmengine报错的问题。

此外，我只用了单卡V100进行训练，担心单卡比8卡batch size差距太大，我将单卡的batch size从16修改成了32。

【实验现象】
其他没有做修改，开始训练后发现validation map从第15个epoch开始，就呈现下降趋势（从epoch 10的0.414、0.576下降到epoch 15的0.410、0.574，当前训练到epoch55，已经下降到了0.377、0.541）。

【问题咨询及分析】
想请问是否是我哪里复现有问题？（check了一下貌似直接改成s之后mixup好像没有开，有可能是这个原因，但是我觉得不至于会让validation map呈下降趋势）

[wondervictor]

Hi @shupinghu, we have also met the same problem: fine-tuning YOLO-World on COCO without mask-refine leads to performance degradation. We're checking it. However, you can enable mask-refine=True for better results currently.

[shupinghu]

Hi @shupinghu, we have also met the same problem: fine-tuning YOLO-World on COCO without mask-refine leads to performance degradation. We're checking it. However, you can enable mask-refine=True for better results currently.

Does the validation map in your experiment gradually decrease during fine-tuning as well?

In my experiment, not only was map unable to reproduce the values in the paper, but the bigger problem was that map was getting worse and worse during fine-tuning.

[wondervictor]

@shupinghu, w/ mask-refine, the fine-tuning results are normal and consistent with the results from the paper. However, removing mask-refine will produce abnormal results.

[shupinghu]

@shupinghu, w/ mask-refine, the fine-tuning results are normal and consistent with the results from the paper. However, removing mask-refine will produce abnormal results.

OK, I will try this config file and feedback the experiment result. Does use "mask-refine" mean that we use the segmentation annotation to reproduce the bbox annoration?

[wondervictor]

mask-refine provides box refinements and supports copypaste during training.

[taofuyu]

1. I'm confused that in transform YOLOv5RandomAffine, use_mask_refine is deprecated in your version of mmyolo. So, it should not influence the result ?

2. And, custom dataset usually dosen't have segmentation annotations, dose it mean that fine-tuning on custom dataset never yield out good result ?

[wondervictor]

@taofuyu

1. Compared to w/o mask-refine, w/ mask-refine contains another copypaste augmentation.
2. It should work well on datasets without segmentation annotations and we need to find out what's wrong under this setting.

[taofuyu]

@wondervictor
Thanks. For me, the problem is the decline of open-vocabulary ability after fine-tuning on custom dataset.

[wondervictor]

@taofuyu I'll add it in TODO and fix it soon.

[wondervictor]

Hi @shupinghu and @taofuyu, I've uploaded the fine-tuned weights and logs for models with mask-refine=True in configs/finetune_coco.

[shupinghu]

@shupinghu, w/ mask-refine, the fine-tuning results are normal and consistent with the results from the paper. However, removing mask-refine will produce abnormal results.

OK, I will try this config file and feedback the experiment result. Does use "mask-refine" mean that we use the segmentation annotation to reproduce the bbox annoration?

using "mask-refine" is OK.