One might be tempted to answer by asserting that arbitrary discriminatory association choices in marriage don't have the important socioeconomic effects that other discriminatory choices have. That's dead wrong. Race and income are highly correlated. Whites have higher income than blacks. Only about 5 percent of all marriages are interracial. That means whites marrying other whites makes the income and education distribution more skewed than it would be if there were more interracial marriages. I imagine that most of us would be horrified by the suggestion of mandated marriage diversity.
If an activity is publicly financed, then arbitrary discriminatory association should be prohibited. That would apply to, among other things, public libraries, schools and universities. Private libraries, schools and universities should have complete freedom of association, whether it's discrimination for or against a particular race, sex, religion or any other trait upon which it chooses to associate. Interestingly, Americans who support racial preferences should be the strongest supporters of privatization, but they're not.
The bottom line is that the true test of one's commitment to freedom of association doesn't come when he allows people to associate in ways he deems acceptable. The true test comes when he's willing to permit others to associate in ways he deems grossly offensive.