Well there are women only spaces that are there for privacy and women only spaces that are there for safety. There are also women only spaces for fairness. In the last case, I am thinking of women's sports mainly.
The problem is that excluding trans women from women only spaces denies them their identity but having people who are physiologically male in those spaces may cause distress to some natal women. Whose rights do we consider the more important when we have conflicts of this kind? I don't think it's helpful to try to shut down the debate in the way that some trans-activists do.
In fact, I suspect the reason why they try to do this is because they know that, in the end, they'll have to accept that having a biologically male body does make a difference and they are going to have to make some compromises.
Multi-gender communal changing areas at swimming pools existed back in the 90's for families to use. Communal changing rooms in shop are a thing of the past, these days there are single cubicles. My local authority has done away with gendered toilets and has a row of identical cubicles accessed through an archway, each of which can be used by any gender, and several of which are equipped for disabilities and baby changing. One of the most popular cafes in town has one set of toilets with two cubicles and one area for hand washing etc.
I agree that closing down the debate is unhelpful and, being honest, to me as a woman it feels threatening. But seeing a penis in a changing room isn't that big a deal - in many parts of Europe mixed gender nudity isn't cause for concern - in Austria I stayed in a place with a spa where it was considered normal for both genders to partake together, and without bathing costumes. There has to be compromises, of course, and safety has to be paramount. Where the lines will end up being drawn, who knows.