Because people love to see things in black and white, and not understand that in almost all things there are shades of Gray.
It is far easier to paint you as a racist, than to touch your ideology, or your believes.
The reality was that both South and Union were wrong, although South had a better claim, their support for Slavery did in FACT ruined their argument. You can not speak of freedom while holding people slaves. You can not speak of Liberty while denying it. Most importantly you can not have liberty while Government ensures Slavery, which means by definition Government must be large and powerful. (Slavery requires Government, a private person may hold another illegally, but never legally without Government monopoly of force).