The only debate among scholars isn't whether these patterns exist but whether they reflect acculturation or genetics. A substantial body of work suggests genetics. The fact of business is that we do differ genetically by race and sex, not only in intelligence and aptitude, but in physical ways as well.
Why in the world would we deny these differences, and deny their effects on observed outcomes, particularly in an academic setting where there's supposed to be open inquiry? I think we do so for a couple of foolish reasons. First, most of us share the value of equality before the law. We falsely believe that equality before the law requires that we must in fact be equal. In my book, being a human being is the only condition for equality before the law. The second reason has to do with human arrogance. If a particular outcome is deemed undesirable and it's genetically determined, our hands are tied and we just have to accept it.
Dr. Summers has responded to the criticism created by his NBER remarks with serial mea culpas, groveling and apologies. He's in deep trouble. Faculty members don't differ that much from chickens in a barnyard. The sight of the boss chicken bleeding is all that's needed for the vicious pecking to commence.
If there's a legitimate criticism that can be made about Dr. Summers' NBER comments, it's that he didn't exercise discretion. There are certain things best left unsaid in front of children. Children have little understanding and can be easily offended by unvarnished truths.