The tubes themselves aren't desirable to crabs, but the stuff that grows on them is. He's probably just picking off some algae. If he gets too rough, the worm will hide again.