If places of public accommodation were free to racially discriminate, how much racial discrimination would there be? In answering that question, we should acknowledge that just because a person is free to do something, it doesn't follow that he will find it in his interest to do so. An interesting example is found in an article by Dr. Jennifer Roback titled "The Political Economy of Segregation: The Case of Segregated Streetcars," in Journal of Economic History (1986). During the late 1800s, private streetcar companies in Augusta, Houston, Jacksonville, Mobile, Montgomery and Memphis were not segregated, but by the early 1900s, they were. Why? City ordinances forced them to segregate black and white passengers. Numerous Jim Crow laws ruled the day throughout the South mandating segregation in public accommodations.
When one sees a law on the books, he should suspect that the law is there because not everyone would voluntarily comply with the law's specifications. Extra-legal measures, that included violence, backed up Jim Crow laws. When white solidarity is confronted by the specter of higher profits by serving blacks, it's likely that profits will win. Thus, Title II of the 1964 Civil Rights represented government countering government-backed Jim Crow laws.
One does not have to be a racist to recognize that the federal government has no constitutional authority to prohibit racial or any other kind of discrimination by private parties. Moreover, the true test of one's commitment to freedom of association doesn't come when he permits people to associate in ways he deems appropriate. It comes when he permits people to voluntarily associate in ways he deems offensive.