I always go bare back when fishing, unless it's in the hardwater season, and it's -30°.
However, there is no science to that decision. I can't really answer your question about damaging the fish.
What kind of gloves do you wear? They must provide some sort of grip on the palms and fingers?