I have done a reasonable amount of searching and was only able to find one experiment that provided solid data on what you were asking.
Foreign-language experience in infancy: Effects of
short-term exposure and social interaction on
phonetic learning (Kuhl et al. 2003)
The results indicate limits on phonetic learning; 9-mo-old infants
exposed to foreign-language material from DVDs did not reverse the
decline in phonetic perception. The AV and A exposure sessions
duplicated Experiment 1’s live sessions, yet no learning occurred. The
results suggest that phonetic learning from complex language input
relies on more than raw auditory sensory information. At this age,
learning is influenced by the presence of a live person.
The current results are consistent with a
variety of studies on older children (preschool age) exposed to
language material, both native and foreign, from children’s TV shows.
The results indicate that, although there is evidence that specific
vocabulary items can be learned through exposure to television
programs, the more complex aspects of language, such as phonetics and
grammar, are not acquired from TV exposure.
I have not been able to track down information specifically on adults learning a second language and the effect of passive listening.
Note that passive listening is not putting a language tape in and repeating the words, its sitting in a room full of native speakers and just listening to what they are saying or having a foreign language television on in the background while you go about some other task.
I could sit in a room full of nuclear physicists and listen to them talk all day, but even though I know rudimentary mathematics what I can learn from them is going to be very little if nothing.
I think research would suggest as per the child study that having a live adult engaging with you versus passive listening provides significantly greater benefit.