And it didn't overflow like that when you tested it after setting it up? Or was it never tested?

The problem of having too strong a return pump will be an issue even if you had your tank drilled. Overflows, whether hang-on or drilled, can only handle so much water based on their size.