The only things that can tear Texas down in the current CFP setup are internal poor performance, a drum beat of consistent negative press, and letting has-beens who wish things had stayed they way they used to be influence decisions. Mack Brown is absolutely a threat to create issues with the latter two.
Texas already faces the most hostile local media and national media from their own program alumni in the country. For fuck's sake Desmond Howard spent the Gameday broadcast in Ann Arbor pimping his school and even using his own kid to do so. Meanwhile Sam "The Less Terrible" Acho fucking picked Clemson to beat Texas and win it all probably so the national media thinks he's super objective. You know, Kirk Bohls disease. Bohls, by the way, being an old Mack Brown chum still covering Texas in Austin but now going so for the Houston rag.
Mack Brown can't take Texas down on his own but he can absolutely hurt Texas through his relationships. Inviting on a potential saboteur is idiotic no matter what, and it's the ones you think can't be too bad who are the most dangerous kind.