Cannot run parallel inference with DDP

[thomassajot]

Search before asking

• I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

Validation, Detection

Bug

I am trying to make predictions in parallel using multiple GPUs in order to speed up inference on large datasets.
From what I gathered, the best way to go about it with Pytorch is to use torch.nn.DataParallel.
However, the model first gets created in cuda:0 then is copied over to the desired gpus. This overloads cuda:0 and if not (when the batch size is small) then the same model is present over multiple gpus. I then get the following exception:
RuntimeError: CUDA error: invalid device ordinal CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

See full error:

YOLOv5 🚀 v6.2-145-gf8b7463 Python-3.9.13 torch-1.12.1+cu102 CUDA:4 (NVIDIA GeForce RTX 2080 Ti, 11019MiB)

Fusing layers...
Model summary: 416 layers, 140038156 parameters, 0 gradients, 208.0 GFLOPs
Adding AutoShape...
Traceback (most recent call last):
  File "/mnt/remote/data/users/thomasssajot/yolov5/notebooks/generate_classification_results.py", line 152, in <module>
    main(device=2)
  File "/mnt/remote/data/users/thomasssajot/yolov5/notebooks/generate_classification_results.py", line 136, in main
    model = get_model(model_path).to(f'cuda:{device}')
  File "/home/thomassajot/miniconda3/envs/yolov5/lib/python3.9/site-packages/torch/nn/modules/module.py", line 927, in to
    return self._apply(convert)
  File "/mnt/remote/data/users/thomasssajot/yolov5/models/common.py", line 621, in _apply
    self = super()._apply(fn)
  File "/home/thomassajot/miniconda3/envs/yolov5/lib/python3.9/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/thomassajot/miniconda3/envs/yolov5/lib/python3.9/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/mnt/remote/data/users/thomasssajot/yolov5/models/yolo.py", line 155, in _apply
    self = super()._apply(fn)
  File "/home/thomassajot/miniconda3/envs/yolov5/lib/python3.9/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/thomassajot/miniconda3/envs/yolov5/lib/python3.9/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/thomassajot/miniconda3/envs/yolov5/lib/python3.9/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/thomassajot/miniconda3/envs/yolov5/lib/python3.9/site-packages/torch/nn/modules/module.py", line 602, in _apply
    param_applied = fn(param)
  File "/home/thomassajot/miniconda3/envs/yolov5/lib/python3.9/site-packages/torch/nn/modules/module.py", line 925, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Environment

PyTorch version: 1.12.1+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu118.04) 9.4.0
Clang version: 13.0.1-++20220120110844+75e33f71c2da-1exp1~20220120230854.66
CMake version: version 3.10.2
Libc version: glibc-2.27

Minimal Reproducible Example

import torch 
from torch.utils.data import DataLoader
from tqdm import tqdm

def get_model(path):
    model = torch.hub.load('ultralytics/yolov5', 'yolov5s')
    model.eval()
    return model

def get_image_files():
    images= 'path/to/image.jpeg'
    return [image] * 64

def main():
    images = get_image_files()
    model = get_model()
    net = torch.nn.DataParallel(model, device_ids=[0, 1])

    loader = DataLoader(dataset=images[:64 * 4], batch_size=4, shuffle=False, num_workers=8) 

    with torch.no_grad():
        for batch in tqdm(loader, ncols=140, desc=f'Predictions'):
            res = net(batch, size=1280)


if __name__ == "__main__":
    main()

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[thomassajot]

I also tried to use multiple devices directly through torch.hub.load

model = torch.hub.load(
        ...
        device='4,5,6,7',
    ) 

Without success. The script runs on a single GPU.

[glenn-jocher]

@thomassajot a single YOLOv5 model never actually exists across multiple GPUs. What happens in DDP is that multiple copies of the same model are loaded, one per GPU, and then each RANK receives a part of the batch, which is passed through the local RANK model, with losses combined afterwards in RANK 0 to compensate correctly for the distributed losses.

If you want to run multiple models in parallel you can simply direct models to different devices and dispatch them in daemon threads, i.e.

model0 = torch.hub.load(..., device=0)
model1 = torch.hub.load(..., device=1)

[thomassajot]

Thank you for your fast reply.

I agree with your comment and I was hopping to leverage DP / DDP to still make predictions in parallel since one batch is split across GPUs. But this does not work due to some code in Yolov5 (I believe).

I have been trying to work this out for a few days now. I could not run multiple models in different threads because Pytorch complains that I cannot spin up a model in a child process.
Could you share more concrete way of how to proceed ?

[glenn-jocher]

@thomassajot seems pretty simple

[thomassajot]

@glenn-jocher Thank you for this example. Unfortunately, when instantiating the models on 2 devices (device 4 and 5 out of 8 gpus). both models seem to be on the same device.
Using the gpustat command, I only see a single instance of the model instead of 2.

(I am user thomassajot only)

During instantiation, I see the logs mentioning GPU 4 and 5, but when looking into model.model.device I get the value cuda:0.

Code:

def get_model(path, device: int):
    model = torch.hub.load(
        repo_or_dir='/mnt/remote/data/users/thomasssajot/yolov5/', 
        model='custom', 
        device=device,
        path=path, 
        source='local'
    ) 
    model.eval()
    return model

def run_inference(model, dataset, image_size, batch_size: int = 32):
    print(f'Processing {len(dataset)} images on  device {model.model.device}.')
    loader = DataLoader(dataset=dataset, batch_size=batch_size, shuffle=False, num_workers=8) 

    with torch.no_grad():
        for batch in tqdm(loader, ncols=140, desc=f'Predictions on device {model.model.device}'):
            model(batch, size=image_size)

def main():
    image_size = 1280
    batch_size = 4
    model_path = '/mnt/remote/data/users/thomasssajot/yolov5/runs/traffic_light_2020_undistorted/yolov5x6_1280_multi_label/weights/best.pt'
    models = [get_model(model_path, device=4), get_model(model_path, device=5)]
    n = len(models)
    print([m.model.device for m in models])
    images = get_image_files()[:64 * 4]
    threads = [threading.Thread(target=run_inference, args=[m, [im for im in images if hash(im) % n == i], image_size, batch_size], daemon=True) for i, m in enumerate(models)]
    for thread in threads:
        thread.start()

    for thread in threads:
        thread.join()

So one one side, it seems that I can run models in parallel, but this is not going to improve the inference speed as they are sharing the same GPU.

[glenn-jocher]

@thomassajot hi, thank you for your feature suggestion on how to improve YOLOv5 🚀!

The fastest and easiest way to incorporate your ideas into the official codebase is to submit a Pull Request (PR) implementing your idea or fix, and if applicable providing before and after profiling/inference/training results to help us understand the improvement your feature provides. This allows us to directly see the changes in the code and to understand how they affect workflows and performance.

Please see our ✅ Contributing Guide to get started.

[thomassajot]

I believe the issue comes from select_device:

yolov5/utils/torch_utils.py

Lines 115 to 129 in 7f097dd

	elif device: # non-cpu device requested
	os.environ['CUDA_VISIBLE_DEVICES'] = device # set environment variable - must be before assert is_available()
	assert torch.cuda.is_available() and torch.cuda.device_count() >= len(device.replace(',', '')), \
	f"Invalid CUDA '--device {device}' requested, use '--device cpu' or pass valid CUDA device(s)"
	if not cpu and not mps and torch.cuda.is_available(): # prefer GPU if available
	devices = device.split(',') if device else '0' # range(torch.cuda.device_count()) # i.e. 0,1,6,7
	n = len(devices) # device count
	if n > 1 and batch_size > 0: # check batch_size is divisible by device_count
	assert batch_size % n == 0, f'batch-size {batch_size} not multiple of GPU count {n}'
	space = ' ' * (len(s) + 1)
	for i, d in enumerate(devices):
	p = torch.cuda.get_device_properties(i)
	s += f"{'' if i == 0 else space}CUDA:{d} ({p.name}, {p.total_memory / (1 << 20):.0f}MiB)\n" # bytes to MB
	arg = 'cuda:0'

For each torch.load, select_device is called in which we set os.environ['CUDA_VISIBLE_DEVICES'] = device. In this case first the env var is set to 4. Pytorch loads this cuda information. During the second load, we set the env var to 5 but I believe that pytorch's knowledge of the available gpus stay the same. In these cases the function returns cuda:0 as the device to put the weights on and that points to GPU 4.

I am not familiar with the code base yet. I do not understand the purpose of this function select_device. Are there any tests in the codebase?

I would suggest removing setting the env variable CUDA_VISIBLE_DEVICES and leaving it to the user. And then this function should return the actual device number desired by the user instead of returning cuda:0

[glenn-jocher]

@thomassajot got it. I'll add a TODO to run some tests here. CUDA_VISIBLE_DEVICES is really intended for CLI ops like:
CUDA_VISIBLE_DEVICES 1,2 python detect.py

[d710055071]

detection_device = torch.device("cuda:1")

det_model                   = get_detection_model("yolov5x_1280_c15_v1.pt",detection_device)
the mode run in cuda:0 ,not cuda:1
this is bug? 

yolov5/utils/torch_utils.py

line:129 modify
arg = 'cuda:' + devices[0]

the mode run cuda:1 is ok

[saumitrabg]

@thomassajot @glenn-jocher we are running to challenges here and wanted to see if you are able to resolve this issue to completion. Would love to compare notes and find a way forward.

[glenn-jocher]

@saumitrabg hello! As YOLOv5 is a community-driven open-source project, we appreciate any comments, feedback, and suggestions from our users to make improvements to our codebase. However, unfortunately, without a more specific and detailed description of the issue you are encountering, it's hard to provide immediate support. Could you please provide more specific details on the challenges you are facing, any error messages or stack traces you are receiving, as well as any steps you have taken to resolve the issue so far? This information would help us better understand the problem and suggest potential solutions going forward. We look forward to hearing back from you!

[deJQK]

Hi @glenn-jocher , maybe a little late, but I wonder, if I have one model yolo say with the type of ultralytics.engine.model.Model, and I would like to apply it on a large dataset, say dataloader, in the following manner:

for batch in dataloader:
  res = yolo(batch, ...)

where batch is a tensor with shape [N, C, H, W], what is the best practice to do so? Naively using the above code will report error like

TypeError: type Tensor is not a supported Ultralytics prediction source type.
See https://docs.ultralytics.com/modes/predict for supported source types.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (
pid: xxx) of binary: /yyy/bin/python3

I am using Yolov8 but the error looks the same.
Thanks!

[glenn-jocher]

Hi @deJQK, to apply the ultralytics.engine.model.Model on a large dataset using a dataloader, you can use the forward method of the model to make predictions. Here's a way to do it:

for batch in dataloader:
    res = yolo.model(batch)  # Assuming yolo is your ultralytics.engine.model.Model object
    # Process res here

This will pass the batch through the model and return the predictions. Let me know if this resolves the issue for you or if you need further assistance!