It's a specious argument, to be sure. How is it "intolerant" to ask that a towering symbol of Islam be moved a few blocks from the place where 3000 murders were committed in the name of radical Islam? And, to use a favored left-wing word, isn't it, in fact, quite "insensitive" of the mosque's supporters to insist on precisely this location?
To use another First Amendment analogy, no one is trying to "censor" the practice of Islam. Rather, given the unique damage inflicted on the U.S. by the 9/11 attacks, many Americans are just asking for the analogue to a "time/place/manner restriction": Not at this time (less than a decade after 9/11), in this place (the shadow of Ground Zero) and in this manner (if the mosque's supporters really wanted to symbolize interfaith unity, as they claim, why not make the building an interfaith chapel?).