Skip to content

Why not scale values when attribution values smaller than 1e-5? #393

Closed
@andreimargeloiu

Description

@andreimargeloiu

When displaying the attribution, you normalise and scale the values.

However, do you skip normalising if the scaling factor (which is the max value after the outliers) is below 1e-5?

def _normalize_scale(attr: ndarray, scale_factor: float):
    if abs(scale_factor) < 1e-5:
        warnings.warn(
            "Attempting to normalize by value approximately 0, skipping normalization."
            "This likely means that attribution values are all close to 0."
        )
     ....

Activity

andreimargeloiu

andreimargeloiu commented on Jun 3, 2020

@andreimargeloiu
Author

Is there any update on this?

vivekmig

vivekmig commented on Jun 3, 2020

@vivekmig
Contributor

Hi @margiki , we have this check to try to catch instances where attribution values are all approximately 0 and avoid cases where the user may be misled by visual artifacts in the attribution maps which might be magnifying small magnitude differences when normalizing (e.g. noise or floating point error). By not normalizing in these cases, the visualization would indicate that values are all approximately 0, and if outliers exist, they would be particularly salient.

Do you have a use-case where normalization below this magnitude is meaningful? You can also alternatively normalize attributions prior to calling visualize_image_attr and set the outlier_perc argument to 0.

andreimargeloiu

andreimargeloiu commented on Jun 8, 2020

@andreimargeloiu
Author

Use-case
My use-case is interpreting robust model (they are trained using adversarial training [1]). Such models are trained on adversarial inputs.

On robust models, the gradients with respect to the input are very small (see picture below), where the s axis represents the attributions before rescaling. Notice that the range is around 1e-3. Using SmoothGrad, the gradients are around 1e-5, 1e-6 -> which creates issues with Captum.

image

Issue with current warning
For people investigation interpretability on robust models, it's essential to be able to plot them, despite potential errors associated with floating-point arithmetic.

In Jupyter this warning wasn't printed, which took me hours to dig into Captum and understand why the saliency map was essentially white (because the inputs weren't scaled)

Potential solution:
It would be good to allow power-user to bypass this warning (either through a parameter), or simply disable the check.

[1] https://arxiv.org/pdf/1706.06083.pdf

bilalsal

bilalsal commented on Jun 9, 2020

@bilalsal
Contributor

Thank you very much @margiki for the useful insights.
Indeed we need to give user the choice instead of a silent warning.
We will plan this for the next release.

andreimargeloiu

andreimargeloiu commented on Jun 9, 2020

@andreimargeloiu
Author

Awesome! Maybe the best way for users is to do the scaling anyways, and get a warning if the values were small.

What do you think? I'm happy to make a pull request/

vivekmig

vivekmig commented on Jun 9, 2020

@vivekmig
Contributor

@margiki Thanks for the details on your use case, makes sense! I agree, the cleanest solution is probably just to do the scaling regardless and update the warning message accordingly. If you want to make the pull request with the change, that would be great, thanks!

NarineK

NarineK commented on Aug 22, 2020

@NarineK
Contributor

@margiki, @vivekmig , do you still want to work on the PR ? Can we close this issue ?

andreimargeloiu

andreimargeloiu commented on Aug 24, 2020

@andreimargeloiu
Author

Thank you for the heads up! @vivekmig, please go ahead as you initially proposed and plan this change for a future release :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @bilalsal@vivekmig@NarineK@andreimargeloiu

      Issue actions

        Why not scale values when attribution values smaller than 1e-5? · Issue #393 · pytorch/captum