I've never noticed a problem with not fishing them out. I don't think there is much nutrients in the shell that would significantly pollute a tank.