Can an image flashed in the middle of a video for a single frame (as in this hoax by Vicary), induce a specific behavior in people? An example would be the supposed use of this to get people in theaters to buy snacks. Is there any research demonstrating this to be effective or ineffective?
Reading this article might be a good one.
In short, it tells of the start of all this "single frame" theory, namely that research that made you drink cola / eat popcorn. This was not good research, if there was any research at all.
There is however a lot of media coverage and 'urban myth' stuff going on.
That last part triggers me to check Snopes for the subliminals for the pocorn story :)
To go beyond the popcorn-myth, still in the same article (emphasis mine, do also check the background of the title, the 'cargo-cult' story is too great to miss):
The article states a lot of references in the bottom, and the conclusion is clearly that there have been attempts to proof, but no evidence found, that subliminal messaging works.
In short: Sometimes they do something.
Showing someone who has a strong phobia of spider for a split second can produce a rise in blood pressure without the person being consciously aware of having seen a spider.
There was even a study that used subliminal images of spiders to combat the phobia of spiders.
You don't need to be consciously aware of a stimulus to be effected by the stimulus. In most cases a stimulus that raises conscious attention is however much stronger than a stimulus that doesn't.
According to the Straight Dope, subliminal advertising is generally ineffective. Subliminal tapes, however, might be able to increase the effectiveness of training, especially when the message flashed is: "Mommy and I are one". The effectiveness of this message, however, is disputed.