The Guardian's conclusions are too strong
The Guardian has an article today in which it reports allegations of "institutional bias" against ethnic minority candidates for admission to Oxford University. The paper has obtained data on admissions, and found that there is a statistically significant difference between the proportions of applicants of ethnic minority students and white students admitted to two subjects - Economics & Management and Medicine - after controlling for an applicant's (eventual) A-Level Grades. To quote from the article:The gap has often been explained as being due predominantly to the fact that students from ethnic minorities are more likely to apply for the most competitive courses, such as medicine. But the latest figures, which for the first time break down success rates by both ethnicity and grades for some of Oxford's most competitive subjects, cast significant doubt on these long-running explanations.Their data blog draws an even stronger conclusion:
[Subject] mix cannot explain the discrepencies within some of those subjects themselvesThey miss a rejoinder to this claim. Any candidate of a given ability, as measured by A-Level grades, is still going to be more or less likely to be admitted depending on which course they apply to. I achieved high grades at A-level, but given my specific skills and interests I knew I was more likely to be admitted to university economics courses than, say, history programmes.1
If ethnic minority candidates are pushed by cultural factors towards courses such as Economics & Management and Medicine (The Guardian only looked at these two and Law), then this will obviously reduce the likelihood of them making the choice which maximises their chance of admission. If white applicants face no such handicap, then you would expect white applicants to be more likely to be admitted, as they would be more likely to select courses to which they know they are best suited.
Thus, the university's defence - that ethnic minority candidates are more likely to apply to competitive courses, and thus have a lower chance of being admitted - remains intact, even looking at within subject data.
1. Of course, what I am saying here is that ability for a given subject varies in a way that is not captured by A-Level grades, and the paper does acknowledge this weakness ("university spokeswomen were keen to stress A-levels are just one measureof ability, which they say is also ascertained through additional tests and interview"). Nonetheless, they missed this highly important case of such a variation caused by the selection effect which has previously been pointed out by the university.
It is also possible that the opposite of what you claim is true: that minority students tend to be pushed by their parents to choose A-levels that will help them in a particular career, which would mean that Oxford admissions are discriminating even more than they appear by the stats. This seems like an equally "obvious" counter explanation.
ReplyDeleteAnyone can come up with a hypothesis that could plausibly explain any data. Doing so is hardly a "debunking", it is just speculation. So why don't you go collect some new data to test your theory?
I am *not* claiming that there is positively evidence that discrimination is *not* going on. We just don't know - there is not enough evidence. What I am claiming to debunk is the Guardian's specific assault on Oxford that their explanation for a difference-in-averages doesn't make sense. That is captured in the assertion:
ReplyDelete"[Subject] mix cannot explain the discrepencies within some of those subjects themselves "
from their data blog, and comments such as this from the author on Twitter:
"James Ball @jamesrbuk
@Widerife81 @benjywoolgar @WillardFoxton Ox gave one explanation for gap since 2010. That explanation isn't consistent with the data"
Clearly, the explanation is still consistent with the data due to the selection effect I am talking about. The claims above are obviously false. They are the ones asserting something, not me, and I am contesting their assertion.
"I can think up a specific scenario where the data can be explained despite the hypothesis being false" is very different from "there is not enough evidence to say the hypothesis is true". You have come up with a scenario that could potentially create a bias, under certain circumstances (specifically, that minority students apply preferentially to certain degree courses without preferentially being suitable to that subject). But to say that this undermines the evidence sufficiently to make it uninformative, you have to come up with an estimate of how large the effect would have to be to explain the data without the hypothesis being true, and give evidence that it is plausible that the effect is that large.
ReplyDeleteLook at the data at the Guardian - among applicants to Medicine with an AAA grade (with no A*s), Asian students have 2.59 times lower odds of being made an offer (95% CI 1.8-3.9), and black students are 6.7 times less likely (95% CI 1.7-57.5). For economics, these figures are 2.5x (1.6-4.2) and 6.3x (0.97-263.8). The racial bias happens across two different subjects and across two different racial groups. It is stronger for black students than Asian students (and that difference is itself statistically significant).
Your hypothesis is that this large, consistent gap is explained by minority students applying for economics and medicine despite being less well suited to the subject. But that also runs counter to the evidence: minority students are more likely to take appropriate A level grades for medicine and economics - such as maths, biology, chemistry, economics, accounting and business studies (data).
So you would need the bias of minority students being pressured into taking E+M or medicine to be so strong that it overwhelms the fact that these students are, a priori, better suited to the subject, and still have effect size left over to explain why these students are more than half as likely to be offered a position conditional on having three As. And even that doesn't explain why the effect would be twice as strong in black students than in Asian students. At face value, your hypothesis just doesn't seem likely to explain this data.
This is not the only reason why the simple comparison of averages is flawed; my post was only addressing the consistency of the "Oxford defence" with the data and was not a comprehensive discussion of its shortcomings. There are several other reasons why we have not identified the causal effect of race, and which mean there is insufficient evidence that the hypothesis is true. For example:
ReplyDelete(1) I suspect minority students are more likely to be from poorer schools and therefore poorly advised on their applications
(2) If we suspect that the "minority disadvantage" caused by discrimination / inequality in life before you apply to Oxford shifts the distribution of 'admissibility)' of minorities even slightly to the left, you would significantly distort the ratio of white-minority in the high tail of the distribution. When it comes to both A*A*A* A-level results, and to Oxford admissions, you are dealing with draws right from the top of the distribution. If we assume ability is normally distributed, this means that even a slight shift in the distribution for one subpopulation will distort the ratio above any cutoff greatly, because the probability mass deteriorates rapidly in the tail.
To respond to the data you posted: when I was talking about subjects to which you are most suited I was not simply referring to your A-level subjects. The entire point of the Oxford admissions process is to distinguish between people with very high A level grades and appropriate subjects. They care about talent and enthusiasm for the subject on dimensions not measured by A-level choice and success - that's the entire justification for the interview process rather than going on the paper application. I would venture that most Oxbridge undergraduates could achieve top grades in almost any set of A-levels. That would not have rendered them equally likely to get in for another subject. It is not surprising that the selection into subjects perceived as traditional and rigorous applies at A-level as well as degree level.
One should not draw conclusions from data where there are so many effects which are not controlled for. A comparison might be accusations that are often made against Oxford about discriminating against applicants from poor backgrounds. Again, this is usually made using simple comparisons of averages. But the only sophisticated study into this has found no evidence of discrimination (and some evidence of positive discrimination): http://www.economics.ox.ac.uk/materials/papers/12041/paper608.pdf .
Yes yes data is complicated there are lots of explanations we need to dig deeper (though the thresholded skew effect you are describing is exactly why I, and the Guardian, restricted our analysis to AAA students with no A*s - this is a much finer band and is not subject to this problem). The only point I was making is that you threw out a potential confounder without actually looking into how much ability it had to explain the data, which wasn't terribly helpful.
ReplyDeleteAlso, describing that one study as "the only sophisticated study" done on Oxford admissions and socioeconomic background implies that you have missed a few or that you are using a non-standard definition of "sophisticated". See here and here, for example. There are more studies again done in Cambridge that also inform the debate, and we also often run our own internal analyses on the admission datasets within colleges or departments to aid in access and tuning our admission procedures. The one study you cite takes one approach (looking at outcomes for marginal candidates) that is helpful but far from the only approach, and is very prone to confounding from biases in teaching and resource access during the course itself (another area under active research).
Hmm sorry if that last comment (or any, really) came across as overly aggressive or dismissive. It has been good exchanging comments about this, and I don't mean to be disrespectful.
ReplyDeleteI recently came across your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have enjoyed reading. Nice blog.The post is written in very a good manner and it entails many useful information for me. I am happy to find your distinguished way of writing the post. Now you make it easy for me to understand and implement the concept.
ReplyDeleteThank you for the post. . Accountants Oxford