He who pays the piper calls the tune. Once you accept government money, you accept government control/regulation/approval/guidelines/criteria. Voucher advocates are essentially holding the position that allows the tentacles of a failed entity (the government education monopoly) to have involvement with a successful one (private education.) You wouldn't want a doctor disciplined for malpractice to be in charge or in any way involved with the practice of a good doctor. Why do that with education?
Government should no more be involved in education than it should be involved with healthcare or religion; for mostly the same reasons.