It might be, one day, that you will need a license of some sort and zones where you can fly them without getting in the way of plane flight paths and a height restriction depending on where it is.
You probably would have to abide by no fly zones around power stations and the like.
You could have some restrictions, but not a total ban, to make it safer and so people can enjoy flying them.
Some people like shooting rockets, and they go quite high.
IMO it's all about where you do it.