Why?
Because according to them the physical is all there is. Imagine if it were the case that in some way mathematics
becomes material and has an independent existence
not dependent on it but the physical very much beholden to mathematics for it's existence.
There could not also be the same reservations expressed against other ontologies becoming physical.
If maths, whatever it is incarnates in physical form for instance.