I think you need to reflect on what "case by case" means.
There is no case where a man is a woman.
Once you've accepted the dogma that men can be women, you have no basis on which to exclude any man who says he is a woman from women's single-sex facilities.
"You can come in, but only if you cut off your testicles and invert your penis" isn't a solution for any problem.
That the so-called "left" have bought into this self-serving, individualistic, narcissistic, reality denying nonsense, that benefits mainly pharmaceutical corporations and unethical surgeons, is disappointing. You can't identify into oppression. Material reality matters.
"Gender affirming care" should be called what it is: stereotype enforcing harm.