I do believe you're missing the point. Any man currently has the protected right to marry any untaken and consenting woman. Any woman has the right to marry any untaken and consenting man.
However, men are discriminated against on the basis of gender, in that they cannot marry other untaken and willing men, as can women. And women are discriminated against, likewise.
There's no possibility of a compelling public interest on behalf of either the state or federal governments to justify the gender discrimination (which demands heightened scrutiny) and, as such, it doesn't pass the necessary test for gender-based discrimination.
It's really a simple question.
Full disclosure: I am not a lawyer.