I don't see any polygamists lobbying for the right to legal marriage, so that's an irrelevant argument. They practice their own definition of marriage with only one legal wife and multiple spiritual wives. There's no crime and no prosecution because they're entitled to live with and bed with as many woman as they want.
So it comes down to what we want to LEGALLY allow. Do you want 3 wives claiming a Social Security survivor benefit when one man dies? Do want an Army officer marching three wives into the military infirmary to get free medical care? As a society we have decided that two people -- no more -- constitute a marriage, and for good reason. But there are no good reasons to exclude gay couples.