Scrape it off instead by pulling and scraping at the base of the xenia. The part that is attached to the rock is actually fairly tough. If there is any remaining reside use a clean cheap toothbrush or something like it to scrub the rest off.