Your argument is a fundamental logical error--a fundamental case of irrelevancy--called "Tu quoque." That is, "And you also." The Latin names for the classic forensic fallacies go back to the Middle Ages.
What you are saying is that because Jefferson was not perfect, he had no right to defend liberty. But if only people who were perfect could defend liberty, there would be no one to defend liberty because we are all sinners--we are all imperfect. Jefferson himself was painfully aware of the contradictions between his ideals and his practice in his own life.