8

Bill Gates has recently said:

The ultimate is computers that learn. So called deep learning which started at Microsoft and is now being used by many researchers looks like a real advance that may finally learn. It has already made a big difference in video and audio recognition - more progress in the last 3 years than ever before.

The statement was quoted in many news articles.

How true is it that Deep Learning started at Microsoft?

Improve this question

1 Answer 1

Reset to default
9

That is incorrect. It might be true that Microsoft pioneered commercial applications of Deep Learning though, which is what I suspect Bill was getting at.

Li Deng, who works for Microsoft Research, took the Deep Belief Nets (DBNs), devised by Hinton and his team at the University of Toronto, and applied them (successfully) to the TIMIST dataset for speech recognition. This got Deep Learning a lot of interest, in the commercial sector.

DBNs were the start of a resurgence of interest in deep learning. Originally when neural nets were created in 1986, they were often deep, in the 90's something called the Universal Approximation theorem was proven, which roughly says "A neural net with one hidden layer (that is sufficiently large) can approximate (given sufficient training data) any continuous function." This, combined with the difficulty in training deeper nets, basically ended Deep Learning. Hinton's paper in 2006, and Bengio's monograph in 2007, sparked a resurgence of interest, because Hinton showed a novel new technique that made training deep nets feasible (this technique being Deep Belief Networks), and Bengio argued that deep nets were important, and that we could get many advantages over shallow nets.

Li Deng at Microsoft research took this onboard, and with his work on TIMIST, showed the world that Deep Learning was feasible and great.

--

DBN's in academia

(Still trying to date Li Deng's work. citations needed.)

Improve this answer
8
  • Hinton's early work was published in 1986 in mitpress.mit.edu/books/parallel-distributed-processing. That's one of the earliest examples of multi-layer non-linear perceptron training.
    – user5582
    Feb 12, 2014 at 0:22
  • See also: nature.com/nature/journal/v323/n6088/abs/323533a0.html.
    – user5582
    Feb 12, 2014 at 0:27
  • @Articuno: Deep Learning Typically refers to 3 or more hidden layers. The Universal Approximation Theorem, was not proven til much later. (For reference my goto paper for MLP's is Hinton, Rumelhart et al's paper "Learninging Represtentiosn by back-properagating errors" in nature 1986)
    – Frames Catherine White
    Feb 12, 2014 at 0:29
  • That's what I linked to :) Also, the models introduced in 1986 were not limited to single hidden layers. My point in those comments was giving your references that you could use to establish that deep learning has been happening for at least as long as your answer claims.
    – user5582
    Feb 12, 2014 at 0:32
  • Yeah, you ninjaed me. Posted will i was still trying to find the link. Problem with calling the stuff that was happening in the late 80's deep learning is that it was only deep becuase the Universal approximation theorem hadn't been proven. (oh bother now I will have to add that ot my answer)
    – Frames Catherine White
    Feb 12, 2014 at 0:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .