I would say just the filter depending on the media. Most media won't leach chemicals back in the water, but it's probably better practice to just replace the media if you plan on putting it back in the display.