You're not the only one -- I did it for many years as well. The ones that I've ever heard ancedotal evidence as to being major issues, are perfumes and anti-persperant deodorants... The anti-persperant variants use aluminum oxide as well as a couple other choice metals, if I recall correctly, to prevent perspiration.. These metals are where the concern comes in, AFAIK..

As far as soap goes -- I used to worry that one a lot, but it became very unfeasable for me (since my hands are constantly in contact with my wheels -- which are in turn constantly in contact with the ground.) For 4+ years now, I wash my hands regularly with soap, and have done so on many occasions just before going elbow-deep into the tank. I've yet to have a deleterious effect from the soap.

I think that a perfumed soap would probably be one you want to avoid, depending on what the components of the scent are. I tend to use the basic scented SoftSoap for my hands, but have also used dishwashing detergent -- never a negative that I've witnessed.

-Justin