I credit an interview with government studies professor Harvey Mansfield for showing me two inconsistent narratives on diversity in higher education faculty.
If there is a lack of minority faculty members, including women and racial minority members, it is the result of discrimination, prejudice and other efforts to keep them out.
However, when asked to explain why there are so few conservative faculty members, those same people will say they lack the competence to become faculty members, or there is a lack of interest for academic jobs among conservatives and they self-select into other fields.
That is to say, in the first scenario the problem is prejudice, while in the second scenario the university functions as a meritocracy and the problem is with the conservatives themselves.
I will add that people like me tend to see it in reverse, where the university is a place of discrimination for conservatives, but when it comes to minority hires it is a perfect meritocracy.
In a recent blog post, Bryan Caplan gives the political diversity question a one-two-three combo beatdown, with a jab showing just how deep the imbalance is in faculty political positions, a right and a left hook showing the value of a politically diverse faculty and for the haymaker? He showed a 2012 study of social psychologist academics where 82 percent of surveyed liberals admitted they would be prejudice against a conservative candidates.
The usual defense of this is that conservatives are just plain wrong, well, paper authors Yoel Inbar and Joris Lammers addressed that argument in the paper:
Is it a problem that conservative political opinion is not tolerated? If one believes that conservatives are simply wrong, perhaps not. After all, geologists are not obliged to accept colleagues who believe the earth is flat. But political or moral beliefs often do not have a truth value. A belief that the earth is flat is factually false; a belief that abortion should be prohibited is not. Neither is a belief that cultural traditions should be respected or that economic inequality is acceptable. It may also be that many aspects of conservative thinking can serve as inspiration for interesting research questions that would otherwise be missed. Finally, as offensive as it may seem to many (liberal) social psychologists, believing that abortion is murder does not mean that one cannot do excellent research.
I can't speak for how much of demographic diversity is discrimination, but it looks compelling that political diversity is a result of discrimination.