Reflecting on CS Graduate Admissions
I was just indulging in my favorite bad habit (reading hacker news), when I came across a discussion about prestige in faculty hiring - namely, the fact that the majority of CS professors across the US come from the top 18 or so universities, and that even within that, there's bias. The HN discussion turned to Ph.D. admissions, with a commenter noting:
"I'm not sure, but I'd bet that the best institutions actually don't care where their applicants come from. They can afford to just choose the best. It's the so-so places that have to signal their quality by 'hiring the best'. I've seen this in graduate school applications - students with poor marks often stand a better chance of admittance in a top program than a so-so one."
I just finished chairing the CMU CS Ph.D. admissions committee, so I thought it might be worth writing down my experience in doing so. With two very important notes:
What does category 1 look like? They're one of the top students at a top school; they have a near-4.0 or equivalent GPA; they've done research, have publications, and have strong letters of recommendation from professors known to the faculty on the admissions committee. They have an good statement of purpose that suggests both focus and drive.
Yawn. OK - so now that we're done with those 10 candidates, let's get on to where it's interesting! :) But we can do that by breaking down the inputs for that top student.
Why "at a top school"? Because it's a very easy way of being calibrated. We admit a lot of students from CMU, MIT, Berkeley, and Stanford, and a lot of our faculty went to these institutions. We know the courses, because we also know and collaborate with the faculty who teach them. We know that an A in 15-410 at CMU is a pretty impressive indicator of systems hacking ability; that for an undergrad to get an A in 6.856 at MIT is a "wow" indicator for undergrad theory ability. We know what the hard courses are at these schools, and how to read through that to get an accurate estimate of what a student is good at. But this is also conflated with other things, because students at these schools have easy access to engaging in research.
Why publications: (And by publications, I mean "at good, known conferences, journals, or workshops" -- not at pay-to-publish paper mills. In fact, we had a rousing discussion about whether to count those as a negative.) They're a sign that the student was involved in research to a significant degree. We often hoped to see a cogent discussion of this research and their role in it in the student's SoP.
Why a near 4.0? Pretty obvious - it's a sign they've mastered the fundamentals of computer science and won't have any basic sticking points when they enter a Ph.D. program.
Why letters from known faculty: First of all, calibration. When someone I know who does great research writes a letter, I generally know that we're operating on the same scale, and that their experience working with a student is likely to translate to something similar to my own estimation. Second, trust: After enough time handling admissions, you build up institutional knowledge that "Professor X at university Y writes very understated letters", or "Professor Z at university Q sends us a best-in-20-years student every year, and we don't believe any of it."
Why a good SoP and what is a good SoP: It's one that conveys purpose and drive, first and foremost. Focus: During admissions at CMU, we bin students broadly into areas of interest, and try to mostly rank within sub-areas. In the latest phases of admissions, we incorporate feedback from the entire faculty in ways that are specific to them, as a way to try to balance yield and the number of faculty working in an area who want students. Drive: There are a lot of people who're smart. But succeeding in a top Ph.D. program requires also a high degree of self motivation, and dogged determination to get you through the inevitable setbacks. The best SoPs elucidated a compelling vision.
Why did I make everyone watch de-biasing videos: Because our goal is to find the best students to admit, period. If we find amazing students that our competitors don't, that's an automatic win. :)
It's easy for the very few candidates who meet all of those bins. The real job of admissions is to find the students who don't. This latter point is exactly why the HN post author is right: we try very hard to look for signals of research success, which are somewhat different from the signals of academic success that are present in applications.
And sometimes, the results can be surprising. The author of this blog post, for example, not-so-proudly received a "D" in organic chemistry lab from my undergrad days, and I came out of a good state school (the University of Utah) that's ranked #42 -- not #4 or #2. So - if you're one of the vast majority of students who didn't go to blah blah university, how do you maximize your chances? (hint: You don't do it in your senior year... :-)
The answer to this varies by area. I asked each "area" (somewhat self-defined) at CMU to write up a few paragraphs explaining what they value in an applicant. Note that almost all areas valued the things noted above, but the interesting questions arise when you ask them to weigh one vs the other. Here are some examples of possibly unexpected things:
Systems faculty valued examples of students being able to manage large systems and complexity. Large programming experience / projects outside of class, whether that be on their own, through a startup, internship, or through research. Many faculty noted that for native speakers, the verbal GRE was a better predictor of success than math. (For non-native speakers, the GRE fails in myriad ways. But that's another story. I tended to ignore it unless it was off-scale in either direction.) One faculty noted specifically that he looks for students who can impose structure on uncertain or poorly-structured things -- again, an example that the applicant can handle the complexity and sometimes ill-defined tradeoffs and goals of constructing large systems.
Theory faculty valued publications as a great, but not mandatory, indicator. Participation in things like math olympiad, ACM programming contest (with wins), taking grad-level math or theory classes, and strong grades at top schools were all ranked as important. Letters that talk about the intellectual capability of the student.
So: Grades in your area of interest matter. We had some rousing arguments about, e.g., systems students with amazing programming experience but poor grades in some non-systems CS courses. Sometimes the experience won over.
For international students, grades mattered more: We have less calibration for letters from the faculty, particularly for large schools in India and China where there's less of a research focus. But many of those students have figured out the trick of doing an internship at a place we'll recognize, such as Microsoft Research. A top student in Yao's class at Tsinghua, or the top few from the top IITs, etc., with letters from MSR, IBM, or other research labs local to the area, is also something easy to calibrate for.
But publications and research experience typically mattered more. Particularly when it was good research, from someone we trust. I had a colleague in undergrad who was the first author of several SOSP and OSDI papers during his undergrad. Needless to say, nobody cared that he came from Utah when it came time for Ph.D. admissions -- he was an easier admit than a Stanford graduate with a 4.0 and a triple major but with no compelling research (or equivalent) experience.
Once we got out of those bins, though, the Statement of Purpose started to carry even more weight, because it gave people a chance to demonstrate the other factors that lead to academic success. In fact, the programming language faculty specifically called out the SoP as important for demonstrating that the student really knew what they were asking for when it came to Carnegie Mellon's very formal approach to PL. But the SoP is not just about writing -- it's also about having interesting things to write about, and relevant experience to tell us about and synthesize a vision from. In systems, for example, the SoP was a chance for applicants to talk about the projects they'd done outside of class or other obvious research venues. It's also a chance to explain why your grades in computational basket weaving sucked, but not to hold them against you. Sometimes this doesn't work - but I've seen a few letters of rec + SoP combinations that very successfully explained away crappy years or semesters in a way that was convincing. It's particularly helpful if what's in the SoP is backed up by a letter writer. If you've had a non-standard path getting to the point of grad school admissions, by all means tell your story (but not in an overdone way). Particularly if it shows off that you're the most persistent critter to cross the mississippi in the last 5 years, or something similar, and that you're going to apply that same persistence (or other awesome trait) to your grad school effort.
We do admit students who haven't yet figured out what area they really want to do, but they have to be very strong in a broad academic context (and still have rec letters attesting to their potential). We view this as a category with potentially high yield and return, but with higher risk (what if they show up and decide they hate research? :). We're more selective in admitting from this pool than from the students whose SOP and research experience convinces us they know what they're doing. The easy path is to have gained enough experience in undergrad to figure out broadly what you're interested in, but don't freak out and try to fake it if you haven't.
Perhaps the most important signal was the letters of recommendation. One (or two!) strong, very specific letter(s) from a faculty member who'd worked directly with the student for a longer period of time was the most compelling possibility. In fact, we have a term for rec letters from professors who taught a class to the student: "DWIC" ("Did Well In Class"), upgradeable to DWIC++ if the student was amazing in class. But being amazing in class isn't a demonstration of independent, creative thought; of drive to succeed absent the structure of coursework and assignments. DWIC is a good supporting letter that helps us trust the other recommendations, but pure-DWIC is almost fatal. On the other hand, a letter from a trustworthy source that says "this student is brilliant and amazing! if you don't admit them, you're insane!", while it's not a guarantee of admission, is a guarantee that the application will receive very serious consideration, even if other things are missing (poor or spotty grades, etc.).
"I'm not sure, but I'd bet that the best institutions actually don't care where their applicants come from. They can afford to just choose the best. It's the so-so places that have to signal their quality by 'hiring the best'. I've seen this in graduate school applications - students with poor marks often stand a better chance of admittance in a top program than a so-so one."
I just finished chairing the CMU CS Ph.D. admissions committee, so I thought it might be worth writing down my experience in doing so. With two very important notes:
- I'm not doing it next year, so please don't use this an excuse to drop me an email saying "CAN I GET IN PLEASE" - I'm going to be on sabbatical, in fact, so I'll be about as far removed from the process as possible.
- No, this was not the program that sent out the accidental admissions letters. We dodged that one narrowly (we use the same codebase). phew.
- Don't spend a lot of time reviewing the top 2% of applicants. They're easy to identify, they get admitted everywhere, and the error rate is low.
- Don't spend a lot of time reviewing the bottom 50% or differentiating among them.
- Please focus your attention on finding the students who have the potential to be in the top 10% percent, but are likely to be overlooked by other top programs.
- Go watch two videos about dodging hidden bias:
What does category 1 look like? They're one of the top students at a top school; they have a near-4.0 or equivalent GPA; they've done research, have publications, and have strong letters of recommendation from professors known to the faculty on the admissions committee. They have an good statement of purpose that suggests both focus and drive.
Yawn. OK - so now that we're done with those 10 candidates, let's get on to where it's interesting! :) But we can do that by breaking down the inputs for that top student.
Why "at a top school"? Because it's a very easy way of being calibrated. We admit a lot of students from CMU, MIT, Berkeley, and Stanford, and a lot of our faculty went to these institutions. We know the courses, because we also know and collaborate with the faculty who teach them. We know that an A in 15-410 at CMU is a pretty impressive indicator of systems hacking ability; that for an undergrad to get an A in 6.856 at MIT is a "wow" indicator for undergrad theory ability. We know what the hard courses are at these schools, and how to read through that to get an accurate estimate of what a student is good at. But this is also conflated with other things, because students at these schools have easy access to engaging in research.
Why publications: (And by publications, I mean "at good, known conferences, journals, or workshops" -- not at pay-to-publish paper mills. In fact, we had a rousing discussion about whether to count those as a negative.) They're a sign that the student was involved in research to a significant degree. We often hoped to see a cogent discussion of this research and their role in it in the student's SoP.
Why a near 4.0? Pretty obvious - it's a sign they've mastered the fundamentals of computer science and won't have any basic sticking points when they enter a Ph.D. program.
Why letters from known faculty: First of all, calibration. When someone I know who does great research writes a letter, I generally know that we're operating on the same scale, and that their experience working with a student is likely to translate to something similar to my own estimation. Second, trust: After enough time handling admissions, you build up institutional knowledge that "Professor X at university Y writes very understated letters", or "Professor Z at university Q sends us a best-in-20-years student every year, and we don't believe any of it."
Why a good SoP and what is a good SoP: It's one that conveys purpose and drive, first and foremost. Focus: During admissions at CMU, we bin students broadly into areas of interest, and try to mostly rank within sub-areas. In the latest phases of admissions, we incorporate feedback from the entire faculty in ways that are specific to them, as a way to try to balance yield and the number of faculty working in an area who want students. Drive: There are a lot of people who're smart. But succeeding in a top Ph.D. program requires also a high degree of self motivation, and dogged determination to get you through the inevitable setbacks. The best SoPs elucidated a compelling vision.
Why did I make everyone watch de-biasing videos: Because our goal is to find the best students to admit, period. If we find amazing students that our competitors don't, that's an automatic win. :)
It's easy for the very few candidates who meet all of those bins. The real job of admissions is to find the students who don't. This latter point is exactly why the HN post author is right: we try very hard to look for signals of research success, which are somewhat different from the signals of academic success that are present in applications.
And sometimes, the results can be surprising. The author of this blog post, for example, not-so-proudly received a "D" in organic chemistry lab from my undergrad days, and I came out of a good state school (the University of Utah) that's ranked #42 -- not #4 or #2. So - if you're one of the vast majority of students who didn't go to blah blah university, how do you maximize your chances? (hint: You don't do it in your senior year... :-)
The answer to this varies by area. I asked each "area" (somewhat self-defined) at CMU to write up a few paragraphs explaining what they value in an applicant. Note that almost all areas valued the things noted above, but the interesting questions arise when you ask them to weigh one vs the other. Here are some examples of possibly unexpected things:
Systems faculty valued examples of students being able to manage large systems and complexity. Large programming experience / projects outside of class, whether that be on their own, through a startup, internship, or through research. Many faculty noted that for native speakers, the verbal GRE was a better predictor of success than math. (For non-native speakers, the GRE fails in myriad ways. But that's another story. I tended to ignore it unless it was off-scale in either direction.) One faculty noted specifically that he looks for students who can impose structure on uncertain or poorly-structured things -- again, an example that the applicant can handle the complexity and sometimes ill-defined tradeoffs and goals of constructing large systems.
Theory faculty valued publications as a great, but not mandatory, indicator. Participation in things like math olympiad, ACM programming contest (with wins), taking grad-level math or theory classes, and strong grades at top schools were all ranked as important. Letters that talk about the intellectual capability of the student.
So: Grades in your area of interest matter. We had some rousing arguments about, e.g., systems students with amazing programming experience but poor grades in some non-systems CS courses. Sometimes the experience won over.
For international students, grades mattered more: We have less calibration for letters from the faculty, particularly for large schools in India and China where there's less of a research focus. But many of those students have figured out the trick of doing an internship at a place we'll recognize, such as Microsoft Research. A top student in Yao's class at Tsinghua, or the top few from the top IITs, etc., with letters from MSR, IBM, or other research labs local to the area, is also something easy to calibrate for.
But publications and research experience typically mattered more. Particularly when it was good research, from someone we trust. I had a colleague in undergrad who was the first author of several SOSP and OSDI papers during his undergrad. Needless to say, nobody cared that he came from Utah when it came time for Ph.D. admissions -- he was an easier admit than a Stanford graduate with a 4.0 and a triple major but with no compelling research (or equivalent) experience.
Once we got out of those bins, though, the Statement of Purpose started to carry even more weight, because it gave people a chance to demonstrate the other factors that lead to academic success. In fact, the programming language faculty specifically called out the SoP as important for demonstrating that the student really knew what they were asking for when it came to Carnegie Mellon's very formal approach to PL. But the SoP is not just about writing -- it's also about having interesting things to write about, and relevant experience to tell us about and synthesize a vision from. In systems, for example, the SoP was a chance for applicants to talk about the projects they'd done outside of class or other obvious research venues. It's also a chance to explain why your grades in computational basket weaving sucked, but not to hold them against you. Sometimes this doesn't work - but I've seen a few letters of rec + SoP combinations that very successfully explained away crappy years or semesters in a way that was convincing. It's particularly helpful if what's in the SoP is backed up by a letter writer. If you've had a non-standard path getting to the point of grad school admissions, by all means tell your story (but not in an overdone way). Particularly if it shows off that you're the most persistent critter to cross the mississippi in the last 5 years, or something similar, and that you're going to apply that same persistence (or other awesome trait) to your grad school effort.
We do admit students who haven't yet figured out what area they really want to do, but they have to be very strong in a broad academic context (and still have rec letters attesting to their potential). We view this as a category with potentially high yield and return, but with higher risk (what if they show up and decide they hate research? :). We're more selective in admitting from this pool than from the students whose SOP and research experience convinces us they know what they're doing. The easy path is to have gained enough experience in undergrad to figure out broadly what you're interested in, but don't freak out and try to fake it if you haven't.
Perhaps the most important signal was the letters of recommendation. One (or two!) strong, very specific letter(s) from a faculty member who'd worked directly with the student for a longer period of time was the most compelling possibility. In fact, we have a term for rec letters from professors who taught a class to the student: "DWIC" ("Did Well In Class"), upgradeable to DWIC++ if the student was amazing in class. But being amazing in class isn't a demonstration of independent, creative thought; of drive to succeed absent the structure of coursework and assignments. DWIC is a good supporting letter that helps us trust the other recommendations, but pure-DWIC is almost fatal. On the other hand, a letter from a trustworthy source that says "this student is brilliant and amazing! if you don't admit them, you're insane!", while it's not a guarantee of admission, is a guarantee that the application will receive very serious consideration, even if other things are missing (poor or spotty grades, etc.).
- I'd take a passionate letter of rec from someone I don't know over a "DWIC" from MIT any day. Particularly when it says "went above and beyond", "did X on their own", and other phrases like that that indicate initiative, not just an ability to follow instructions.
- If you're coming from a school with few recognizable researchers, find a way to get hooked up with the recognizable ones. Internships are wildly successful for this for international students, for example. A summer internship is a much lower-risk commitment on the part of someone to advise you, but gives you a way to bootstrap into a letter of recommendation.
- If you're applying for an internship, be specific, persistent, and polite. By specific, I mean not to blast out 100 letters saying "Dear Sir/Madam" - I and everyone I know deletes those upon receipt. Take the time to research the people in your field who are doing things you really like, and write a personalized email to them. Get the greeting right, spell their name right, etc. Put effort into your introduction letter, because people respond well to that. Before you do that, look at their web page and see if they have guidelines for doing so.
- Hints: Find some of their recent work that resonates with you, not the stuff they did 10 years ago. Don't pin your hopes on just one internship, because randomness happens. But be persistent - if you don't get a reply, do send a very polite followup, mentioning that you were specifically hoping to work in their lab (by name, etc.).
This comment has been removed by the author.
ReplyDeleteWhat kind of signal does a non-CS degree (e.g. Math, Statistics, or Electrical Engineering) from a top CS school such as CMU, UIUC, or UW send?
ReplyDeleteThe fact that you're basing a 4.0 GPA to command over CS fundamentals is pretty flawed. It just shows that the student is good at getting As and nothing else. Actually, more often than not, the perfect 4.0 students are pretty suspicious - having learnt to game the system but with shaky fundamentals.
ReplyDeleteInterestingly, I got an A+ in 6.856 at MIT, but was rejected from 14/15 CS PhD programs I applied to.
ReplyDeleteCongrats for getting into #15!
Delete