16

Universities and companies conduct, administer, or provide programmer aptitude tests. A number of people have expressed both skepticism as well as agreement that they may be useful with the hiring process as claimed by one company,

From a practical point of view, the PAAT definitely identifies those people with a high aptitude for programming and for systems analysis. It also identifies those individuals who are high-risk hires (low scores). This is the true value of the test. It minimizes the costs associated with a poor employment decision.

Is there any evidence that this is a reliable mechanism for selection?

  • 7
    Welcome to Skeptics! We want to focus our attention on doubtful claims that are widely held or are made by notable people. Please provide some references to places where this claim is being made - including a specific test. – Oddthinking Jun 11 '13 at 11:27
  • @LarryOBrien I don't think you can retract close votes. – user5582 Jun 11 '13 at 20:08
  • I could say, as a non-reliable source programmer, that I'm highly sceptical aptitude tests for hiring people who has supposedly an education. But separating out those will will do well in a programming course is certainly easier. If you have made it through a university It education, you have at least basic programming aptitude. – Lennart Regebro Jun 12 '13 at 4:15
  • IBM does have a PAT www-05.ibm.com/employment/uk/graduates/apply_selection.html it is now called IPAT for initial selection. It consists of two parts. – user14144 Jun 12 '13 at 12:51
  • 1
    @rob: Why is it even relevant if they are related or not? That makes no difference. They are both aptitude tests, one for programming, another apparently more generic for logic and math tasks. Hence in some way they are clearly related. IBM not endorsing PAT seems to then, in context, mean that they don't endorse their PAT. The one used before. As they have replaced it with the IPAT. – Lennart Regebro Jun 12 '13 at 22:03
12

This is actually a very interesting question since we need to look at this from at least two standpoints: the ability for an aptitude test accurately predict if the test taker is suited to leaning the skill set and the ability for such tests to accurately predict employee performance.

With regards to the use of such tests to predict academic success (i.e. leaning the skill set), it appears that IBM lead the way in this department with their development of the IBM Programmer Aptitude Test (PAT)1, 2, 3 in the 1960's. However, in 1980 a study was published that followed 2,000 students, 1,350 of whom finished and concluded that,

The results indicate that future programming skill is not predictable by the most commonly used written test (IBM's Programming Aptitude Test). The suggestion is that this type of test should not be administered to college level people as the results are unreliable.

In 1984 another study was published that examined the more traditional means of student section to programs (e.g. SAT scores, class rank, high school courses, etc.) which found that there was a positive correlation between these variables and academic success. However, it is important to note that in this situation a aptitude test specific to programming was not administered though. In fact, a 2010 survey paper concluded that, with regards to aptitude tests predicting academic success,

Aptitude tests are not reliable predictors of success, and have not identified any cognitive capacity which is especially significant for learning to program. Evidence relating to predictors such as educational or demographic background, cognitive development or style, or a range of other factors, is also ambiguous or weak. On balance some factors (such as mathematical background/ability, or self-ratings of confidence/comfort as the course progresses) emerge as moderately correlated with success.

The apparent working draft Dehnadi and Bornat paper that was referred to by the articles (1, 2) in the question had it's hypothesis rejected by another paper and in fact the Dehnadi and Bornat later presented (direct link) revised findings where the concluded that their orginal working paper was incorrect, concluding,

We cannot claim to be separating the programming goats from the non-programming sheep: experiment 3 demolishes the notion that consistent subjects will for the most part learn well, and others for the most part won't. And even in the most encouraging of our results, we find a 50% success rate in those who don't score C0 or CM2.
None the less, some of our results indicate that there may be something going on with consistency. There is a case for continuing our investigations.

So with regards to aptitude tests indicating academic success, there doesn't appear to be strong evidence that programming aptitude tests are useful.

This brings us to the question of the use of aptitude tests as a prediction of employee success. For the purposes of this answer, I'm going to assume that you are not referring to tests such as the Oracle Certification Program which tests candidates on a specific topic, but rather more generic ones such as the Walden Programmer Analyst Aptitude Test which attempts to

... simulate what the job applicant is required to do on the job, as well as to determine who is likely to be successful in technical training courses.

In this situation I couldn't find much in the way of recent research that definitively showed a correlation between aptitude tests and employee performance. Much of the literature specific to programmer aptitude tests is either focused on predicting academic success which has already been discussed.


  1. Some early examples of which can be found with great difficultly.
  2. Incidentally at least one journal article claims that that IBM no longer endorses the use of PAT and I was unable to find any mention of it being used any later than about the 1980's. A new version of the PAT could not be found when searching the IBM website.
  3. It appears that IBM is using a new test in the United Kingdom for recruitment that goes by the name IPAT or IPATO which is short for the Information Processing Aptitude Test On-line. There doesn't appear to be any relation between the IPAT and the original PAT nor does it appear to test a candidates ability to acquire programming skills.
  • 1
    I don't see how your '2' claim for IBM comes out of the link you use to support it. – Lennart Regebro Jun 12 '13 at 4:12
  • 1
    the "tests" don't measure learning ability as they're not intended to be applied to people who still need to learn, they're intended to be applied to people who already have the skills, and supposedly show whether those skills indeed exist. – jwenting Jun 12 '13 at 13:42
  • 1
    @jwenting - The question is referring to aptitude tests which generally mean tests that are used to access if someone can learn a given set of skills. Proficiency tests are what you would use when testing to see if someone already has the given skills. – rjzii Jun 12 '13 at 13:56
  • @rob as those tests are only applied to those who have already passed the resume screening to determine whether they have the skills needed to do the job (including learning, if they couldn't learn they wouldn't have the skills, if they didn't have the skills they'd never be hired) they don't test for learning ability. If you'd ever done any of them, you'd realise that. SUPPOSEDLY they test "logical thinking ability" and "whether you fit in the team", but they don't do even that. They test whether you can do tests like that and nothing else. – jwenting Jun 13 '13 at 5:23
  • @jwenting: "as those tests are only applied to those who have already passed the resume screening" The references show that this isn't true. "If you'd ever done any of them" - special pleading. The rest begs the question and needs references. – Oddthinking Feb 4 at 13:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .