I don't rinse my sand either. I just put two 40lb bags into the new system with no problems but I was filling it slowly using my RO/DI so not a ton of water pressure.

Those small particles will eventually either get skimmed out or the bacteria will grow on them and they will settle to the bottom which is a good thing. The more bacteria the better.