As with freedom of expression (yelling "fire" in a crowded theatre), there are limits to freedom of association. I believe those limits are breached when talking about employment, as this is central to being able to enjoy the pursuit of happiness.
As for forming clubs, I believe that social and cultural pressure would prevent the formation of a club nowadays that would exclude people on the basis of race. (Perhaps I am Polyanna, but I think we have grown much in the past 50 or so years.) On the other hand, men's clubs and women's clubs would likely be accepted to a large degree as people understand and accept the differences between men and women. Banning such associations would only have the members "dig in," to resist change.
Great column Professor Williams. Thank you for writing it.