Hermits have the potential to disturb anything.

Look around for the crown. Sometimes if they're stressed they'll drop the feathery bits. It can regrow after a few weeks, so don't toss the worms tube.

It could be just spooked and hiding out. Don't worry too much yet. If you don't see him after a while, you can lift the tube to see if you feel any weight in it. But picking it up can stress it and cause it to crawl out of the tube.