Tragically, I'm not a gold star gay and have had the displeasure of sex with a cisgender male penis.
There is a wide spectrum of transgender experience. Binary trans people, non-binary, genderfluid, agender, and many more. But they all have a lot of negative externalities because, like you said, people can't mind their own business.
A current law that conservative shitheads are trying to implement (I forget if it's Texas or not, but really doesn't matter) essentially outlaws trans people performing in any regard as themselves in public with the way the statute is written. Because if little Johnny (who knows deep down inside she is actually little Joanna) sees a trans person who isn't the butt of the joke or who isn't a sexual object for hypocritical straight dudes, she might actually think a fulfilling life is possible and start to transition when she is able to. The horror of a person actualizing themselves in a way "they" don't understand and fear.