I haven't watched TV since 1969. I worked in it for decades, and knew enough not to watch it. It makes you stupid, and my colleagues in the networks agree! Some of them would point out what a verbal beating they'd get from friends who actually saw what got broadcast. When you work in most aspects of TV, you rarely have time to watch what's going out.
Watching television does not negatively affect intelligence.
Since your brain is recognizing visual input, it is in fact creating neural associations that were not already in place. No matter what you watch, the brain is busy either creating new associations (learning) or recalling previous associations (remembering) [On Intelligence by Jeff Hawkins, Cognitive Evolution by Alice Travis, The Singularity is Near by Ray Kurzweil].
In this way, you cannot become "more stupid" just by receiving visual input even if it is from a television (assuming the person is not susceptible to seizures or has some other input processing disorder)
However, there are several studies that suggest that prolonged exposure to television can negatively affect attention span (especially in children) ["Early Television Exposure and Subsequent Attentional Problems in Children" by Dimitri A. Christakis, et. al]. Also it is possible for various parts of the eye and its connections to the brain to be damaged from staring at a digital screen for long periods of time [Visual Ergonomics, Jeffrey Anshel]. Likewise, it has been known for quite some time that long periods of sitting (which watching tv tends to encourage) have negative effects on a person's health ["Television Watching, Energy Intake, and Obesity in US Children" by Carlos J. Crespo, et. al].
There are many more articles on the subject that can be found at scholar.google.com.
Bottom line: The act of receiving visual input from a television does not negatively affect your intelligence. However, prolonged viewing can have negative effects on physical and mental health so go play outside every once in a while :)
Edit - Apologies for the lack of references. They are now included in the answer. Also, thank you @Reuben for pointing out the issue with seizures.