Always err on the side of caution. As noted above, with sufficient other forms of filtration available, removal of the bio balls didn't disturb enough to cause a problem.

As well, the problem typically associated with bio balls is that they trap detritus, and when you pick them up to remove them, it shakes years of gunk loose.

Where is the live rock you are going to replace them with coming from? A LFS, an established tank, or from elsewhere in your tank? That will make a difference. You don't want to remove part of your bio balls AND put uncured/incompletely cured live rock into your sump at the same time.