Most Americans probably agree there should be freedom of association in the cases of marriage and tennis, but what about freedom of association as a general principle? Suppose white men formed a club, a professional association or any other private association and blacks and women wanted to be members. Is there any case for forcing them to admit blacks and women? What if it were women or blacks who formed an association? Should they be forced to admit men or whites? Wouldn't forced membership in either case violate freedom of association?
What if you wanted to deal with me but I didn't want to deal with you? To be more concrete, suppose I own a private company and I'm looking to hire an employee. You want to deal with me, but I don't want to deal with you. My reasons might be that you're white or a Catholic or ugly or a woman or anything else that I find objectionable. Should I be forced to hire you? You say, "Williams, that's illegal employment discrimination." You're absolutely right, but it still violates peaceable freedom of association.
Much of the racial discrimination in our history was a result of legal or extralegal measures to prevent freedom of association. That was the essence of Jim Crow laws, which often prevented blacks from being served in restaurants, admitted into theaters, allowed on public conveyances and given certain employment. Whenever one sees laws or other measures taken to prevent economic transactions, you have to guess that the reason there's a law is that if there were no law, not everyone would behave according to the specifications of the law.