There's a lot of pain in the way trans people talk and write about ourselves, prompted by callous requests by cis people that tear us in opposite directions. We are expected both to conform to our lived genders--and only "man" or "woman"--110%, and simultaneously to use gender neutral facilities and labels.
About ten years ago I met a notable trans woman who gave a transgender 101 presentation on campus and told her story. At the time, since I was pre-T and lacked confidence, I favored gender neutral restrooms and beamed about how I'd found one in the building. Her response was to quip "Good, if somebody has a problem with me using the ladies' room they'll have somewhere else to go." That stuck with me, and she's not exactly wrong. It's a common invalidation tactic to tell a trans man or trans woman that we are, in fact, something other than men or women and that we must not only use "neutral" amenities but be very enthusiastic at the opportunity. To use a topical example, a school in my district has been corralling trans students off in gender neutral restrooms. These restrooms can be a huge relief for a lot of trans people, including trans people who don't identify with either "male" or "female" as well as those who do but don't pass well yet or have the confidence to start using what would be their proper bathroom... but many of us aren't in that situation. We know we're men, or we know we're women, we know we belong in those spaces, we're much more comfortable and safe in them, and cis people try to degender us by forcing us into a third category, insisting that this is totally progressive and we should be grateful.
Because of this there are trans folks out there who really, really hold on to the label "binary." A month ago or so I read a rant by a trans guy on a trans male forum where he complained about people who would dare take away his right to call himself "binary," because that was "misgendering" and, after all, he was a veritable Ron Swanson of manliness!
Being fully male or fully female doesn't make you "binary," though... because there is no binary. Not just for trans people. There is no gender binary, period. Full stop. So fully male and fully female trans people exist, but not "binary" trans people, nor "binary" cis people, for that matter.
Back when I still identified as fully male rather than in-between-worlds as I feel today, I was always very torn about this, because I wanted language to express that I was fully male without reinforcing the idea that a binary existed. Often I would settle on "binary privileged," because it was usually in the context of talking about the privilege trans men and women may wield over those who don't fully feel that way, but this never really described it for me outside of that context. That said, I would not necessarily heckle a trans person who called him or herself "binary," because we have very inefficient language to describe our feelings and sometimes it's important to make that distinction. That doesn't make it correct, though.
That perception didn't just stop when my gender identity morphed again and I stopped considering myself fully male. If I hated calling myself "binary," why would I call myself "nonbinary?" And to complicate things, when I came out, we didn't really use the term nonbinary. I never developed an attachment to the concept of "nonbinary" because I was critical of the binary anyway, and so instead I use the term that we were using locally much earlier: Genderqueer. I love this word. It's not sanitized for cishet consumption. It doesn't reinforce a false dichotomy between "binary," a concept that doesn't exist, and "nonbinary,' which by extension either also doesn't exist or alternatively encompasses everyone.
I'm not going to tell people not to call themselves "nonbinary" for the same reasons I don't hassle people who call themselves "binary." This is a concept for which there isn't much accurate vocabulary, not everybody likes the word "genderqueer" (either because they object to the word "queer" or because they don't like the history or want something that isn't going to scare away the cishets), and so forth. And I'll accept people including me under the "nonbinary" umbrella for simplicity's sake, as solidarity is more important to me than purity. But on a personal level? I really don't like it and rarely use it for myself and prefer if people are talking about just me they stick to "genderqueer" instead.