Not every man will want to marry a second woman. These days, it seems as if a lot of men don't want to marry at all. I find your scenario very unlikely. However, I appreciate the fact that you came up with a reason why polygamy is wrong (and I agree with that, but for different reasons).
I still think that any argument for allowing gay marriage can be used in favor of polygamous marriage or incestuous marriage.