10

MIT claim to have invented a technology that, by using special video cameras, can detect vibrations and recover the sounds being played that cause those vibrations:

YouTube video: The Visual Microphone: Passive Recovery of Sound from Video Reddit discussion

This seemed really sketchy to me. The biggest worry was the line about the vibrations only registering as changes in a "hundredth of a pixel".

the vibrations caused by the music are so subtle that they move the plants leaves by less than a hundredth of a pixel.

If the vibrations were indeed that small, you would see no change in the image at all, because that's how pixels work- they're the smallest measurement recorded.

In addition, the YouTube channel (Abe Davis's Research) has only one video uploaded to it, it's not on MIT's channel or anything.

Any other supporting evidence that this is real?

Improve this question
10
  • people.csail.mit.edu/mrub/VisualMic - They also link to their related (prior) publications, which is evidence that these techniques have been progressively developed over the past few years.
    – John Lyon
    Commented Aug 5, 2014 at 1:23
  • I don't see any reason to disbelieve MIT researchers. It is on MIT page: newsoffice.mit.edu/2014/… This hasn't been peer-reviewed yet, but before it happens it is hard to say anything else...
    – sashkello
    Commented Aug 5, 2014 at 1:33
  • The claim about vibrations might be an exaggeration or a mistake by the presenter. I also find it strange, as they claim to use only 700x700 pixel camera resolution.
    – sashkello
    Commented Aug 5, 2014 at 1:35
  • 1
    @sashkello And it was accepted via peer review for presentation at next week's SIGGRAPH, a very highly respected computer vision conference. The papers at that conference are then published in a journal, the ACM Transactions on Graphics, without additional peer review. SIGGRAPH is the highest venue of publication for this field of research.
    – user5582
    Commented Aug 5, 2014 at 3:37
  • 1
    @sashkello. Yeah, it's definitely not the norm for science in general. But it is the case for many fields in computer science.
    – user5582
    Commented Aug 5, 2014 at 4:01

2 Answers 2

Reset to default
17

This is real work.

Provenance

The authors are Abe Davis (MIT), Michael Rubinstein (MIT, Microsoft), Neal Wadhwa (MIT), Gautham J. Mysore (Adobe), Fredo Durand (MIT), and William T. Freeman (MIT). (Project website)

This work didn't come out of nowhere. Three of the authors overlap with this paper on Eulerian Video Magnification. That previous work was the focus of a previous question about detecting heart rate from small changes in video.

The work was peer reviewed and accepted to SIGGRAPH 2014, the highest ranked computer graphics conference in the world.

Plausibility

Regarding subpixel motion, a displacement of an object less than one pixel does affect the signal being received by all pixels in the image and most noticeably along the boundary. The effect of a point source of light in the world on the pixels of a camera depends on the optical transfer function that describes how the light from that point is spread across the pixels of the camera. Movement of that point source of light will affect the light received at all pixels of the camera.

They used a high frame rate camera (2200-20000 Hz) to allow reconstruction of the relevant frequencies. When they used a lower frame-rate camera (60 Hz), they took advantage of the rolling shutter in the camera to gather signal at a higher effective rate (61920Hz, but with missing samples) than the base frame rate.

Improve this answer
1
7

As a Video DSP Engineer, I can attest that it is definitely possible and practical to register motion of a fraction of a pixel. I've used such a technique myself (and this general notion is far from new, though its use to extract sound may indeed be).

Consider an image of a coin, say, 80 pixels wide; if it moves as a whole by, say, 1/10 pixel, there will be tiny but well coordinated changes in most of its hundreds of pixels, which can be mathematically analyzed; in fact, given a single frame from the video, you can analyze it and predict that if the coin moves, say 0.04 pixel to the left, then a particular pixel will get brighter (or dimmer) by a certain amount. Any individual prediction may not be accurate when compared to actual motion, but averaged over hundreds of pixels, you can get very good results, especially in ideal conditions (still camera, uniform lighting; good focus...). So you then mathematically work back by comparing the predictions to the actual changes, to estimate the motion.

This technique http://en.wikipedia.org/wiki/Optical_flow is one which can be used, and is similar in its approach to what I've described; here is another: http://en.wikipedia.org/wiki/Phase_correlation

A more difficult limitation is that you have onlr whatever number of audio 'samples' per second, depending on the frame rate. For conventional video, this number is typically in range 24 ... 60, which is too low to resolve interesting audio. The music samples on the web page http://people.csail.mit.edu/mrub/VisualMic/ are all very low notes; however a sample rate of 2200 Hz is given. For applications where you want to understand speech, you'd want to have at least 1000-2000 samples per second, and up to 4000 or 8000 if you want to be able to identify the speaker by the sound of their voice. Refer to this http://en.wikipedia.org/wiki/Voice_frequency -- but be aware that standard telephone processing uses 8kHz sampling, and is clearly more than adequate for clear speech of even higher-pitched voices with ability to recognize the speaker; lower sample rates can still yield intelligible speech.

The paper also discusses taking advantage of the 'rolling shutter' effect in a video camera to get more continuous audio information.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .