This is a subject VERY dear to me. Both my sons were accepted into the IB program here. Neither went. They both decided to go to a magnet school (also public) instead.
The IB program was most assuredly 'one size fits all'. My youngest is a math wiz. He was ready to take precalc as a freshman in hs. The IB school said he couldn't do that - precalc was a 3d year course, so he would either have to take dipsh*t math as a time killer or no math for 2 years. Neither was acceptable.
We put him in a STEM magnet program (Middleton, in Tampa). He is now going into his 3d year. As he got a 5 on the calc BC AP test, he will be taking calc 3 and 4 (vector calc and diff eq), and physics at a local community college.