Field permissions are not respected in CRM Lead Activity

I have given a different permission for specific fields in CRM Lead, and when I change those values, the CRM Lead Activity captures it. Those activity is visible for everyone. 

Is it possible to prevent users not able to see non permitted fields in the CRM activity section?