Take the 2-minute tour ×
Skeptics Stack Exchange is a question and answer site for scientific skepticism. It's 100% free, no registration required.

John Carmack is a renowned developer and CTO. His Twitter account has over 100,000 followers.

On the occasion of Stack Exchange's 5th anniversary, he quipped on Twitter:

[... Stack Overflow] has probably added billions of dollars of value to the world in increased programmer productivity.

The tweet was widely read and retweeted over 100 times.

Is this claim supported by evidence?


What have I tried?

I've made a quick calculation to see if he was completely off the mark, but the numbers seem to add up to a layman like me.

  • Our visit counter recently overflowed Int32: 2,147,483,647.
    The above is a reference to my own tweet but I can also provide supporting evidence: the site is 5 years old and currently does around 6m visits/day
  • Average pay for a programmer in the UK is 45,000£/annum (source) or 22.5£/h. Let's assume a low pay of 20US$/h. I assume an hour is saved every time an answer is provided.
  • Each visit will potentially save some money, but only 77% of visits land on answered questions
  • There is a cost relative to answering a question, however answers are useful to hundreds of people.
  • There are many other factors I am not calculating: for example, does answering a question make you more knowledgeable? Does finding an answer make you less knowledgeable than you would be otherwise? I am ignoring these as this is a ballpark estimate, certainly not a valid answer to my own question.

To sum it up:

  • value provided = 2 billion page views * 77% answer ratio * 20US$ ≈ 30 billion US$

What kind of answer do I expect?

I expect answers to provide evidence to the value (or lack thereof, or cost!) of the increased (or decreased!) programmer productivity as to ascertain whether it is true that they are "probably billions". I do not really want to see calculations like the one I provided.

Disclosure

I work for Stack Exchange.


A Rather Unprecedented Moderator Note from @Oddthinking

This question has caused a startling amount of interest and discussion. A big welcome to the users from Hacker News and Reddit.

Everyone seems to have an opinion. Unlike those sites though, opinions generally aren't welcome here - as answers or in the comments. We are looking for definitive answers based on empirical evidence. Comments should be directed at improving the question, not at discussion.

This is a tough question. It isn't obvious how economists might measure value to an economy. It isn't obvious how computer scientists might measure productivity. It isn't obvious how much time it takes to answer or how much time is saved for each page view. There may be unintentional side-effects of making answers easy to find. etc. Arguments from incredulity are not a valid response to such difficulty.

Sklivvz has given openly naive answers to those questions, as a back-of-the-envelope calculation, and is asking if anyone has done the calculation professionally. If you can find such calculations in the literature, please answer the question. If all you want to do is point out the naive calculations are naive, let's take that as stipulated; it doesn't help answer the question.

In the meantime, the question has been "protected", so only users with a modicum of rep on the site may answer, and the comments have been cleaned up several times. If you want to comment on this mod note, please take it to meta.

share|improve this question
3  
As a side note: there are many scientific studies regarding specificaly stack overflow, so it's not impossible that this value has been studied. –  Sklivvz Nov 30 '13 at 18:17
2  
What is the total number of upvotes of answers on SO? –  Paul Dec 8 '13 at 20:39
2  
@rob it all depends on how many assumptions we have to make. The more assumptions, the more uncertainty, and the more potential for pointless debate. As a sanity check, if you need to ask, you probably need to make a stronger case before posting. :-) –  Sklivvz Dec 11 '13 at 20:39
12  
80K views and not a single answer!! –  Songo Mar 27 '14 at 14:30
6  
This question claims exactly the opposite : workplace.stackexchange.com/questions/27555/… –  isJustMe Sep 29 '14 at 21:50

1 Answer 1

One of the few published papers on the relationship between Stack Overflow and productivity finds that active GitHub users "ask fewer questions and provide more answers than others" on the site.1 The authors write that users who ask questions "distribute their work in less uniform way," than those who do not (indicating interruption). But, they also note that "despite interruptions incurred" Stack Overflow activities correlate positively with participation and commit rates on GitHub.

In other words, there's evidence that Stack Overflow tends to cause interruptions (as a stand in for loss of productivity),2 but people who code a lot (on GitHub) also tend to contribute to SO, and the other way around. The relationship between GitHub and SO participation is in this way used to approximate impact on productivity.3

Neither this study nor the studies it cites provide direct evidence for or against Carmack's claims. Fundamentally, the authors conclude that:

Despite the popularity of Stack Overflow, its role in the work cycle of open-source developers is yet to be understood: on the one hand, participation in it has the potential to increase the knowledge of individual developers thus improving and speeding up the development process. On the other hand, participation in Stack Overflow may interrupt the regular working rhythm of the developer, hence also possibly slow down the development process.4

Because the original claim argues for value through productivity, under conditions where the impact of SO on productivity "is yet to be understood" in systematic study, it (the original claim) remains speculative and not supported by evidence beyond anecdote.

  1. Vasilescu, B., V. Filkov, and A. Serebrenik. “StackOverflow and GitHub: Associations between Software Development and Crowdsourced Knowledge.” In 2013 International Conference on Social Computing (SocialCom), 188–95, 2013.
  2. "Goal: Is participation in Stack Overflow related to productivity of GitHub developers?" (p.188)
  3. "Here, following Adams et.al., we look at only one, but representative, facet of productivity."
  4. Emphasis mine.
share|improve this answer
1  
Thanks. The thing is, it is not useful to compare productivity and stack overflow behavior, because even if someone is less productive because they are on stack overflow all day, their contributions make many other devs productive. –  Sklivvz May 12 at 1:27
1  
I don't see how this answers the question. I don't even see how this is even related to the question. –  George Chalhoub May 12 at 8:28
1  
@denten the paper you cite is about Stack Overflow data in 2012, ancient... even in 2013, that would have covered much less than 75% of the data available (Stack Overflow has been growing exponentially since 2009) –  Sklivvz May 14 at 8:20
1  
Unless someone can show different results that rise to the same standards of evidence, I consider this question resolved. As per site rules I respectfully suggest all further personal opinion and original calculations be expressed in chat as per mods note. –  denten May 14 at 12:39
3  
The original claim understands value explicitly as "increased programmer productivity." It is therefore enough to show results on productivity. I updated the answer to make that connection more explicit. –  denten May 14 at 17:38

protected by Sklivvz Dec 4 '13 at 23:30

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.