Designing an exam that can survive a leak

On scale: 22.76 lakh registered and 22.09 lakh appeared in 2025, across 5,468 centres, and 2024 ran larger still at 24.06 lakh registered.¹ About 12.4 lakh qualified in 2025, nobody scored a perfect 720, the topper finished at 686, and the general-category qualifying floor was 144.¹ The 2024 list initially showed 67 candidates tied at a perfect 720, inflated by grace marks and a disputed answer key; the Supreme Court eventually cut the perfect scores to 17.⁴ The same mark buys very different amounts of rank at different points of the distribution:

The value is computed to seven decimal places so ties stay rare. The topper of every shift lands at exactly 100 whatever the raw marks were, and the merit list is built from these percentiles.⁸ Raw marks from different shifts are never compared to each other.⁷

A worked example with two shifts of 1,00,000 candidates each. Shift 1 draws a hard paper; a candidate scores 512, and 97,000 people in that shift scored 512 or less. Shift 2 draws an easy paper; a candidate scores 540, and 97,000 people in that shift scored 540 or less. Both candidates receive 97.0000000 and rank identically. The 28-mark gap between them was paper difficulty, and dividing by the shift removed it.

A sampling algorithm assembles one paper per session against a fixed blueprint: the subject counts above, topic coverage quotas so no chapter is skipped, a target difficulty distribution, and a total estimated solve time that fits the 3-hour budget. Two papers from the same blueprint should feel interchangeable to a well-prepared candidate, and normalization mops up whatever difference remains.

For the difficulty distribution I compared three approaches. Sampling uniformly across the whole 0-to-1 range spends too many questions at the ends, where they separate nobody. Fixed band quotas, say 30% easy, 50% medium, 20% hard, are easy to explain but put a cliff at every band edge; a 0.49 question and a 0.51 question get treated as different kinds of thing. I settled on drawing each slot’s target difficulty from a bell curve and picking the closest unused bank question, which is what this platform already does for its practice papers with mean 0.5 and spread 0.18. For the real exam I would move the mean to about 0.6. The 2024 result had 67 candidates at a perfect 720 before revision; a paper with more genuinely hard questions stretches out the top of the score distribution so the rank list leans less on tie-breaking. The exact mean is a choice to simulate, and section 7 is the plan for choosing it.

Exam day runs backward from each session’s start time. The paper is assembled by the algorithm four hours out. Between four and two hours out, one prestigious medical college is picked at random and a few of its senior faculty are walked into a room with no internet, where they read the paper and flag anything defective, an ambiguous stem or a wrong answer key being the usual suspects. A flagged question goes back to the algorithm, which draws a replacement from the same topic and difficulty slot. At two hours the paper freezes and goes out to centres encrypted. At one hour the original setter panel does a final read of the frozen paper; they can no longer edit anything, and if they catch something genuinely broken the session switches to a standby paper assembled the same way.

Human review is the bottleneck at national scale, and splitting it up also fixes a security problem. Divide the community pool into chunks of about 500 questions and hand each chunk to a different large institution for vetting, so a lakh-scale pool spreads across a couple hundred vetting bodies. No committee ever holds the full set, so a compromised committee leaks at most its own slice, and auditing a random sample from each chunk keeps the vetting honest.

Then the part I like most: one month before the exam, the entire community pool goes public for practice, every question shown exactly as it was submitted rather than in its rewritten operational form. Every candidate gets the same material at the same time, which collapses the resale value of any single question. Operational papers do not lift questions verbatim from the public pool; the expert-rewrite step means live items are unpublished variants, so the pool teaches the syllabus without handing anyone the paper. The experiment in the next section makes the deeper case: once the pool is large, no feasible amount of rote memorization clears the seat cutoff, so publishing the whole thing costs the exam nothing and quietly rewards understanding over cramming.

The cap that keeps this workable sits on submissions, not on reading: 100 questions per mobile number per year. Reading the pool is unlimited. I want the cap on the input because it bounds how large the public pool can grow. Without a limit, a coaching operation could dump 10 lakh questions into the pool and force every student to grind through all of them just to be safe. A per-person yearly submit cap keeps the pool at a size a student can actually practise against.

Here is how he handles three questions from the first sampled paper under today’s +4/−1 marking. He answers what he knows, takes a shot where he feels confident enough to risk it, and some of those shots land:

Two students are a story, not evidence, so we run two thousand of them, resampled ten times over so no single unlucky draw decides anything. Writing down the exact set of questions each of tens of lakhs of students has learned would never scale, so instead we model it with probability: every student carries a chance of knowing any given question, and we roll against it question by question. Someone who understands more of the subject simply has better odds on more questions, and when they do not know one, they still answer if the marks make the gamble worth it.

That leaves one modelling choice that carries the rest of this section. A student’s chance of knowing a question is really the slice of the public pool they could realistically learn before the exam. Give the average student a fixed study budget of a few thousand questions and their odds on any one question are that budget divided by the pool size. Then hand out seats to the top 2.5%, roughly NEET’s fifty thousand among twenty lakh, and ask what it took to win one.

At today’s practice-bank scale of about ten thousand questions, the least-prepared student who still took a seat had memorized around 60% of the pool, and the score tracked true recall almost perfectly, a rank correlation near 0.95. Rote works here: grind the bank and you can place. Now grow the pool and hold the cram budget fixed. At fifty thousand questions that budget is 8% recall and the score already tracks knowledge only weakly, the correlation falling to about 0.4. By a quarter of a million questions recall is near 2%, the correlation is close to zero, and seats scatter to whoever guessed luckiest rather than whoever knew most.

To stay as competitive as today’s seat-winners you would have to memorize that same 60% of the pool, which at a quarter of a million questions is 150,000 items in a few months. Memory research makes that a fantasy: massed cramming decays fast and is far weaker than spaced, understanding-based study, the largest synthesis of the effect covering 317 experiments.¹¹ So a sufficiently large public pool simply cannot be gamed by rote, which is the case for publishing the whole thing: the only way to answer the questions you could never have memorized is to genuinely understand the material. The live experiment lets you drag the pool size and watch rote collapse yourself.

The same two thousand students also settle the marking question, and the answer surprised me. Rather than argue about +4/−1 versus +5/−2, I swept the whole marking space on the same papers and cohort: every correct mark from +1 to +5 against every wrong-answer penalty from 0 to −4. The first thing I checked was ranking quality, by correlating each student’s score with the recall they actually walked in with. It sits around 0.95 in every single cell of the grid. However I set the marks, the exam sorts people by ability about equally well, and the marking scheme moves that correlation by a hundredth at most. That is reassuring and a little deflating: rank quality is not where the marking scheme earns its keep.

Where the scheme does bite is guessing, and the grid below shows it plainly. Each cell is the share of not-known questions a student still bothers to answer, and the rule the students follow is simple: guess only when the expected marks are positive. Under today’s +4/−1 that is always true, even for a blind one-in-four shot, so everyone answers everything and nobody blanks. Push the penalty to −2 and a quarter to a half of the guessing stops; push it further and it collapses.

Read down the −1 column: at +4/−1 the reward so outweighs the penalty that guessing stays at 100%, but at +1/−1 the same penalty wipes it out. It is the ratio of reward to penalty that matters, not either number alone, and +5/−2 lands where roughly a third of blind guessing is deterred while a student with real partial knowledge still answers. That is the behaviour I actually want: reward what you know, discourage pure luck, without punishing a candidate who can half-reason their way to an answer.

The positive mark, by contrast, mostly just rescales. Raising it from +1 to +5 grows the score spread from single digits into the low thirties because the unit got bigger, not because the order got sharper. Its one real effect is headroom, and it only shows up when the cohort is strong. At a 40% mean nothing approaches the ceiling, but at a 55% mean the +4 ceiling of 128 starts saturating, dozens of strong students pile onto the maximum and the top of the list turns to mush, while a +5 ceiling of 160 still has room to separate them.

So the honest case for +5/−2 is narrower than I first wrote it. It ranks no better than +4/−1, because almost nothing ranks better than +4/−1; what −2 buys is less rewarding of blind luck, and what +5 buys is ceiling headroom for a strong year. If neither of those worries you, today’s +4/−1 is simpler and just as accurate. The argument for moving is about behaviour and headroom, not measurement.

Then the part that ties it all together. I swapped the synthetic students for real ones and had nine language models of clearly different strength sit the exact five papers this experiment saved, marked +4/−1 out of 640, with no access to an answer key. The strongest cleared a perfect 640; the weakest, Claude Haiku 4.5, still managed 325, about 61%. Here is the ladder:

None of these scores move with the public pool size, because a model answers from understanding rather than from having memorized the specific question. The rote cohort’s scores do move, and hard: at a quarter-million-question pool the seat-winners fall to roughly a fifth of the paper, below every model on this ladder, Haiku included. That is the thesis in one line. Make the pool large enough and even the weakest model that actually understands the subject out-ranks a diligent crammer, because there is nothing left to cram. You can run the same comparison on the live experiment page.