Voting for gay marriage does not remove a restriction. It adds another member to a socially privileged catagory. By your definition, allowing a man to marry his dog or his two sisters and brother would be a blow for freedom. Meh.
Recognizing that a social environment exists is not collectivist, just sensible. Society has existed since man came down from the trees or left the garden, depending on your point of view. Increasing the role of government with society is collectivism -- which gay marriage does.
Marriage has nothing to do with freedom of association. Go to bed with whomver you want. Marriage is about special treatment from government and burdens members of society to likewise confer special treatment and benefits.