The whole concept is profoundly silly. Without some reference to a Supreme Being who encompasses certain qualities and calls us to particular standards of behavior, the concept of "goodness" is empty and ultimately entirely subjective.
In a world where human desires and interests inevitably conflict, eliminating religion's yardstick for human behavior does nothing more than give license to the strong to exploit the weak. After all, it might be "good" for A (and A's family) to force B into servitude. We know, however, that it wouldn't be right. How? Through religious tenets like the Golden Rule, and the moral truth that all men are created equal before God (and without a creator, it's easy to adopt a theory of "natural inferiority" that rationalizes exploitation).
Atheists can try to popularize concepts like "goodness" without religion, but either they're simply putting a secularist gloss on inherently religous concepts, or they're creating the conditions for the "war of all against all" where the only constraint on human behavior is what the strong can get away with.