I don't disagree with much of what you say, and they are solid add-ons to what I wrote. But the “who policies it” goes both ways — demands coming from both sides that are incompatible with rights and identities. To condemn “bathroom bans” without also speaking against “bathroom entitlements’ is hypocritical.
I mentioned that same argument that Wittgenstein made several times, and he is hardly the only one who has made it. William of Ockham highlighted that back in the 14th century. It's a fact that every philosophy accepts outside of psychoanalytic theory.