"Let us agree that he can be very funny. "
No, let us agree that he reinforces his admirers' preconceptions. His admirers are dolts, their preconceptions are idiotic, and thus Colbert's act is dreck.
Colbert's act consists of repeating his tired, TIRED old shtick, of adopting the persona of a rock-ribbed conservative whom Colbert portrays as a complete donkey.
But I think Colbert plays the part of a complete dolt, clod, moron and donky ALL TOO CONVINCINGLY, if you know what I mean.