Maths is a language for describing abstract objects. It's not "stuff" in the sense that the Universe is stuff. It's as nonsensical to say the Universe is made of mathematics as to say that this cup of coffee is made of the English language.
Another restatement of your position. The problem is that we have no idea what 'stuff' is and, more specifically, we don't know why there seems to be a close relationship between 'stuff' and mathematics, which have led may people, over the course of millennia, to speculate about some necessary connection or even identity (
Tegmark). Many mathematician feel that mathematics is discovered rather than invented, so we would then have two 'systems' we are discovering with a very close correspondence.
I still have no idea where you get the confidence to dismiss all that and call the relationship between the physical world and mathematics 'lucky'.