Clarification on CAM-Based Heatmap Generation for Each YOLO Version ?

[nhatnam2609]

Hi,

I’ve been reading your paper and I’m intrigued by the CAM-based method you used to generate heatmaps, especially how it is applied to different YOLO versions. I have a few questions to better understand the implementation details:

Heatmap Generation Across YOLO Versions:
How does your CAM-based approach adapt to the different architectures or versions of YOLO? Are there any specific modifications to the standard CAM/Grad-CAM pipeline when applying it to, say, YOLOv10 versus YOLOv11 or YOLOv12?

Intermediate Feature Map Selection:
Which layers or feature maps are used for the heatmap extraction in each YOLO version? Is the process identical for all versions, or are there version-specific choices to better capture the network’s focus?

Normalization and Upsampling Details:
Could you provide more details on how the normalization and upsampling are handled for the heatmaps? Specifically, are there any differences in these post-processing steps for different YOLO versions to account for their varied feature map resolutions?

Implementation Nuances:
Are there any particular challenges or nuances you encountered while implementing the CAM-based method across different YOLO versions? Any code snippets or pointers to where these details might be elaborated in the repository would be extremely helpful.

I’m looking forward to understanding these aspects in more depth to help integrate your method into my work. Thank you for your time and assistance!

Best regards,
Nhat-Nam Nguyen

[sunsmarterjie]

I think all YOLO versions can generate the heat maps. The last layer of the third stage is used for heatmap extraction for all versions without special designs. I will consider open-source the code to generate the maps.

[nhatnam2609]

Great, thank you for clarifying. Could you please publicize the code soon? Alternatively, would you mind providing more detailed instructions on extracting the heatmaps using the last layer of the third stage as described? Thanks again for your support!

[sunsmarterjie]

Hi. You should first upsample the images by:

import os
from PIL import Image

imgs_dir_8x = './imgs_dir_8x'
os.makedirs(imgs_dir_8x, exist_ok=True)
up_size = 8

for img in os.listdir('imgs_dir'):
    image = Image.open(os.path.join('imgs_dir', img))
    new_size = (image.width * up_size, image.height * up_size)
    upscaled_image = image.resize(new_size, Image.BILINEAR)
    upscaled_image.save(os.path.join(imgs_dir_8x, img))

Then, you follow the code below:

from PIL import Image
from torchvision import transforms
import torch
import matplotlib.pyplot as plt
from ultralytics import YOLO
import os
import types


def _predict_once(self, x, profile=False, visualize=False, embed=None):
        y, dt, embeddings = [], [], [] 
        tmp = 0
        for m in self.model:
            if m.f != -1:  
                x = y[m.f] if isinstance(m.f, int) else [x if j == -1 else y[j] for j in m.f] 
            if profile:
                self._profile_one_layer(m, x, dt)
            
            tmp += 1
            if tmp ==7:  # you can decide the output layer here
                return x

            x = m(x) 
            y.append(x if m.i in self.save else None) 
        return x


def heatmap(model, img_path, save_file='./outputs'):

    save_img_path = os.path.join(save_file, img_path.split('/')[-1])
    img = Image.open(img_path)
    w, h = img.size

    transform = transforms.Compose([
        transforms.Resize((h // 3 * 2 // 32 * 32 , w // 3 * 2 // 32 * 32)),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ])

    img = transform(img).unsqueeze(0)
    feature = model.model._predict_once(img)

    outputs = feature.squeeze(0)
    outputs = outputs.mean(dim=0)

    plt.imshow(outputs.detach().cpu().numpy(), cmap="viridis")
    plt.axis("off")
    plt.subplots_adjust(left=0, right=1, top=1, bottom=0)
    plt.savefig(save_img_path, bbox_inches="tight")


if __name__ == "__main__":
    model = YOLO('yolov12x.pt')
    setattr(model.model, "_predict_once", types.MethodType(_predict_once, model.model))

    heatmap_dir = './outputs'
    os.makedirs(heatmap_dir, exist_ok=True)

    for path in os.listdir('./imgs_dir_8x'):
        img_path = os.path.join('./imgs_dir_8x', path)
        heatmap(model, img_path, save_file=heatmap_dir)

Please use V1.0 to reproduce the heatmaps and keep this issue open.

[TRYOKETHEPEN]

Thanks a lot!
I try to change "if tmp ==7: # you can decide the output layer here" to look at other layers, it's ok for yolo11, but raise error for yolov12:

The error raised at "x = m(x)" when "tmp" is 7.

[nhatnam2609]

Hi TRYOKETHEPEN,
Which version did you try on yolov12? V1.0?

Best regards,
Nam