No, they are sex based spaces, just as women's sports are sex based.
Women's sport is sex-based because there are biological checks (some of questionable value) which are done to ensure biological sex of the people taking part. Toilets are assigned based on a gender expression in the general culture - there is no blood test, no DNA check, no sex-organ check... how is it a 'sex based' place? Historically, given that there wasn't a cultural acceptance of the difference between sex and gender, it could be treated as though the two concepts were interchangable, but that's no longer the case.
O.