After seeing my previous post, my old and dear friend Rajeeva Karandikar called me. “I just saw your post about multiple choice tests. While everything you say is valid and correct, your story is in many ways incomplete …”
“Are you telling me picture abhi baaki hai?”, I asked.
“Well, yes … if you want to put it that way …”, Rajeeva agreed.
What follows is a reconstruction of our chat on multiple choice tests.
Why do I get the impression that you don’t like multiple choice tests?
To be very honest, I don’t. I still think that traditional question-answer tests, where the candidate has to write down the solution, are the best.
But they are hard to manage …
Yes, traditional tests take too long, require good examiners, are less computer- and network-friendly, and are now very expensive. I accept that multiple choice tests offer several compelling advantages, but there are also serious problems …
Such as …?
Incorrect or ambiguous questions are a worry — often requiring us to award full marks to all candidates! A bigger problem is to come up with four equally credible choices. I have been on several committees that make up these multiple choice questions and I can tell you that we really struggle to do this! Usually one of the four choices tends to be quite absurd … so candidates have to randomly guess one out of three, not one out of four. That increases the probability of making a right guess.
Yes … I see what you mean. If there are effectively only three choices then there is a greater urge to gamble and guess. And a negative mark of -1/3 for a wrong answer may not be a sufficient deterrent to guessing. Should the negative mark for a wrong answer then be -1/2?
Exactly! In general, if there are n choices, the negative mark to eliminate the advantage of random guessing is 1/(n-1).
How you mathematicians love notation! But let us talk of the ‘big question’ bothering you.
I have several big questions bothering me! Here’s the first: how many candidates who should not be selected actually get selected because of random guessing?
You are asking how many imposters are there on that selected list?
Yes, ‘imposters’, ‘gatecrashers’ call them what you wish.
How do you estimate this number?
I formulate a reasonable probability model and then I run simulation trials. For this example I assumed that our test has 200 questions, that 200,000 candidates take the test, and that we select the top 1000 candidates (these numbers are reasonable; 215,000 candidates reportedly took the IIM entrance test this year).
In my simulation I considered three marking choices for a wrong answer: 0, -1/4 and -1/3. Then I considered nine guessing options: that 10%, 20% … 90% respectively of the candidates resort to random guessing. And for each of these 27 cases I estimated how many ‘imposters’ sneaked in (see the chart below).
I am looking at your chart, and I am alarmed! If there are no negative marks (blue line), and if half the candidates resort to random guessing, then that list of 1000 successful candidates has 216 ‘imposters’! Correct?
Correct! Notice however that the ‘imposter’ count is much lower with -1/3 mark for a wrong answer (green line). In this case if half the candidates guess then we will only have 72 ‘imposters’.
There is also another way of looking at this picture … which is even more worrying! If there are 72 or 216 ‘imposters’ in the list, it means that 72 or 216 deserving candidates failed to get selected!
That’s terrible! Can’t we do something about this?
Not much. We are talking of two variables here: the magnitude of the negative mark, and the percentage of candidates guessing. While choosing the negative mark is in our control (we could, e.g., make it -1/2), we have no control over the percentage of candidates guessing! The only way to reduce guessing might be to reduce the time of the test … but I don’t like that option too much either.
So things don’t look good.
No. And here’s my second big concern.This is a little tricky, so listen carefully. Imagine that we have a completely fair test with no random guessing, and the true cut-off mark for passing this test is 86/100. That means the last ‘true’ guy to board the bus must have 86/100; even with 85/100 he can’t get in!
Now return to the real world with random guessing. Check out the last guy who actually got into the bus. What would have been his ‘true’ mark? If this true mark is 83/100 or even 80/100, it wouldn’t be too alarming. But our simulation revealed that it could have been as low as 70/100. That’s a huge gap of 16! (see the table below)
Yes, looking at the table, I see that with no negative marks for a wrong guess, and with only 10% of the candidates guessing, the gap is indeed 16. In fact the gap is at least 10 in most cases; even with -1/4 or -1/3 for a wrong answer the gap isn’t coming down by much. This is really worrying!
I would worry too. A lot of poor candidates are getting on to that bus, and a lot of good guys are left stranded outside!
I just wonder … could I make it to the top 1000 list with a ‘true’ rank as low as 5000?
You could. You really could! And that’s my third big concern. Look at the chart below. With no negative marks for a wrong answer, and with 10% of the candidates resorting to random guessing, someone with a ‘true’ rank as low as 7392 could just make it to the bus. In fact, look at the other numbers in the chart. Almost every bus that leaves this bus stop is likely to have someone with a ‘true’ rank above 3000.
So a very lucky random guesser could jump 2000 places up the queue to board to bus?
Yes, he could. And this makes me really angry!
But is there an alternative? Surely the conventional question-answer tests that you advocate too have flaws?
They do. Haven’t we heard all those horror stories of how a school topper failed a Board exam, or got a lowly pass class (as, e.g., in Jana Aranya)? I am personally convinced — and others have proposed this too — that we must have questions that may have one or more correct answers. Think of a question like: Which of the following are prime numbers … with the alternatives being 63, 37, 91 and 83. To get your mark you must tick off both 37 and 83.
How does that help?
It practically eliminates guesswork! Notice that the candidate no longer has only four options; he has 15 options (he can tick one out of four in 4 ways; two out of four in 6 ways; three out of four in 4 ways; and, of course, also tick four out of four. So that’s 4+6+4+1 = 15 options). Getting one out of four right by random guessing has a probability of 1/4; getting one out of 15 right has a probability of just 1/15.
I’d say that’s very neat. Random guessing is the scourge of multiple choice tests, and this is being reduced in very significant measure. Studies have also shown that candidates who pass such ‘modified’ multiple choice tests go on to do much better in the subsequent interview.
Rajeeva and I go back a long way. We were classmates at the Indian Statistical Institute in the mid-1970s. He was always the topper; I was always the average performer. Rajeeva has had a glittering professional career; many readers must have seen him calling out his election prediction numbers on CNN-IBN with Rajdeep Sardesai and Yogendra Yadav.