No it is not harmful to leave it in longer than 4 days and it will not release the phosphate it has adsorbed. However, if it is exhausted then it won't be removing anymore phosphate. How long it takes to become exhausted depends on how much phosphate is in your water so that is why you test. I usually use a smaller amount and just change weekly since I don't do alot of testing for phosphates. I just let the algae tell me if there is excess phosphates or not.

Most important in using the aluminum based phosphate removers is to rinse them very well. The dust from them can irritate corals and fish. It is best to rinse with ro/di water since tap water usually has phosphate in it and you will be exhausting the media some by rinsing with tap.