28

I read on https://stackoverflow.blog/2020/01/21/scripting-the-future-of-stack-2020-plans-vision (mirror) (emphasis is mine, to match the question):

In the fourth quarter of 2019, we created a large task force made up of passionate community advocates from across the company to propose solutions to this core problem. The team’s mission is to improve our feedback loop and working relationship with our community. We added a member of our Community team to our leadership team and restructured the organization to invest in Product leadership to build Community-centric features. We are also forming a moderator council, which will include a group of users with diverse experience levels and backgrounds who can help guide our processes. We’re making hard choices and treating no assumptions as sacred in considering ways to evolve the community.

[...]

Community engagement and inclusion is a top priority for Stack Overflow in 2020. Already, the team has established and released a plan for improving communication and empowering our users internally. We are experimenting to improve in areas we know need work: encouraging more question asking, reduce user to user friction, expanding inclusion, and creating a more integrated experience between Stack Overflow and other technical sites on Stack Exchange. The results of all of this work is being shared publicly through our new Loop series on the Stack Overflow blog and by subscribing to our Stack Overflow newsletter and podcast. We have seen some encouraging results: more people are asking questions, we cut the number of negative comments nearly in half, and December was our best month ever for new user signups!

Did Stack Overflow cut the number of negative comments nearly in half between the beginning of the fourth quarter of 2019 and January 21, 2020 (day when the blog post was published)? I don't see any source in the linked webpage supporting that claim.

  • 16
    How would you measure objectively whether a comment is negative or positive? Is this here a negative comment? – Philipp Jan 22 at 12:27
  • 1
    I think this fits the category of "recent events" or whatever it's called and hence is off-topic. – Daniel R Hicks Jan 22 at 14:16
  • @Philipp e.g. search for ""sentiment analysis". – Franck Dernoncourt Jan 22 at 18:09
  • 1
    Would this be better suited to Meta.SE? – The Forest And The Trees Jan 23 at 10:44
  • 1
    I posted an answer on meta.stackoverflow to highlight some of the possible errors. meta.stackoverflow.com/a/393705/1536976 It's mostly the false negatives rate of the algorithm that would influence the validity of the statement. – Trilarion Feb 6 at 16:36
26

I think it's a little ambiguous if this is on topic for Skeptics, since it relies on knowledge only Stack Overflow employees have.

So I posted a long answer on Meta Stack Overflow (we're only measuring on Stack Overflow) that goes into some detail about how we measure negative comments and the recent trends we have seen that lay behind Prashanth's comment in the blog post.

The TL;DR is:

  • There is an algorithm that we trained on user flags of negative comments, and then refined until it had accuracy comparable (actually slightly better, as of the latest revision) to actual people based on moderator feedback.
  • We plotted that algorithm's classifications of new comments negativity over time, and have used it to investigates trends. It is that plot that suggests a 50% reduction in negative comments.

The graph from the MSO Post gives a 30,000 ft view:

enter image description here

Because it is not possible construct a split test for this, what we strictly have is a lot of suggestive correlations. In particular, I wouldn't pin reductions on any single action we have taken nor would I say every action we have taken was successful. However, in aggregate and for practical purposes I'm professionally comfortable with "negative comments, as a proportion of all new comments, have been about cut in 1/2 over the end of 2019".

| improve this answer | |
  • 8
    Since you're only measuring on Stack Overflow, should the blog post be updated to specify SO instead of SE network-wide? – TylerH Feb 5 at 22:30
  • 4
    This is very interesting, and thanks for the details and rigor. What are the specific two numbers you're comparing to come up with a 50% reduction? Both the numbers (e.g., 1% vs. 0.5% or whatever) and what they represent ("negative" comments in 2019Q3 vs. 2019Q4 or whatever). (Yes, I've read the MSO post, and it still wasn't clear to me what specific figures were being used for that claim.) – T.J. Crowder Feb 6 at 8:05
  • @TylerH the post is on a blog called the Stack Overflow Blog and the content is quite clearly Stack Overflow-centric. Stack Exchange gets specific mentions at point as a related matter, but by implication isn't the main subject. It's the wording in this question that mistakenly presumes the whole of the Stack Exchange network, for which I've made a proposed edit. – Will Feb 7 at 10:28
  • 3
    @Will The blog is for the company Stack Overflow. That's not the same thing as the SE site called Stack Overflow. Where the comment claim is made, there is no attempt to distinguish the site from the broader network owned by the company. – TylerH Feb 7 at 14:31
  • 1
    @TylerH That's a common misconception. The blog is [currently] specifically about Stack Overflow the site, not about Stack Overflow the company. That happened without warning/announcement but is true regardless, and has been since last year's paradigm shift. The best example is everything about The Loop. – Asteroids With Wings Feb 9 at 2:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .