Ideomotor action simply refers to small, unconscious movements. Since they are so subtle, those tiny movements aren't normally studied directly, but they can be "magnified" into a larger, more measurable effects. One of the easiest to study is the hand-held pendulum:
The hand-held pendulum first came under scientific scrutiny during the
early 1800s in studies by the highly regarded French chemist
Michel-Eugene Chevreul. His investigation of the mysterious pendulum
was prompted by an 1808 book published by a group of physicists who
claimed "such phenomena were attributed to a new force which
inaugurated a new chapter in physics." Chevreul found this theory
scientifically unacceptable. (snip) The conclusions of Chevreul's study were reduced to
a single principle: The swing pattern of the pendulum was caused by
"nonconscious (involuntary) muscle movements initiated by
autosuggestion."
http://www.magician.org/portal/node/769
Sometimes called "Chevreul pendulum illusion", this effect has been widely studied (1975, 1996, 1998). A 1976 study by Easton and Shor, An experimental analysis of the Chevreul pendulum illusion, found that the effect was enhanced when the subject was concentrating harder, or when the subject was under more physical strain.
The pendulum test is sometimes used as a measure of a subject's susceptibility to suggestion when studying other phenomenon, like facilitated communication:
Next, ideomotor responsiveness was assessed via the Chevreul pendulum
illusion (Easton & Shor, 1976). The stimulus was a pendulum consisting
of a metal washer fastened to a 30-cm string. Participants were asked
to grasp the end of the string and to hold the pendulum directly above
a line on a page that was placed on the desk in front of them.
Ideomotor responding was indicated by movement in the direction of the
line. The Chevreul pendulum illusion can be produced either by asking
people to imagine the pendulum moving in the indicated direction
(Easton & Shor, 1976) or by asking them to prevent the pendulum from
moving in the desired direction (Ansfield & Wegner, 1996).
Facilitated Communication As An Ideomotor Response, Burgess et al, 1997 (PDF)
This study found a significant correlation between the subject's response to suggestion using the pendulum and their success at the facilitated communication tasks. (Facilitated communication involves a helper guiding the hand of an impaired patient to type messages using a keyboard; controlled studies such as this show the message is really coming from the facilitator, not the patient.)
With this background, then, research into other mysterious phenomenon typically involves identifying a plausible mechanism to magnify these small involuntary movements, and tests carefully designed to rule them out. One further example:
Dowsing typically involves a forked stick held at high tension, or one or two freely-pivoting wires held at their balance point. Small involuntary movements are magnified into dramatic dips or swings:
With one rod in each hand, I first demonstrated how dowsing works by
holding the rods in front of me, aimed straight ahead and with their
horizontal arms parallel to each other and to the floor. I then slowly
walked about the room until the rods suddenly crossed one another. I
walked away from that spot and showed how the rods uncrossed and
became parallel again. I suggested that the place where the rods had
crossed must be near a source of flowing water, perhaps a water pipe
under the floor. I then requested that each of the students try the
rods. To their amazement, the rods crossed when they walked over the
spot I had indicated. ...
I did similar demonstrations for the second group of students.
However, this time I let them see my dowsing rods crossing at a
different arbitrarily chosen location in the room. Sure enough, for
these students, too, the rods crossed just at the spot where mine had.
(How People Are Fooled by Ideomotor Action by Ray Hyman, originally printed in The Scientific Review of Alternative Medicine, 1999)
This seems to me to be a clear demonstration of how much subject expectation contributes to the effect, since it is unlikely these students had a stake in the outcome, like a professional dowser does.
Other related phenomenon include the "Clever Hans effect", named after a horse who appeared to be able to perform arithmetic, but was eventually shown to be responding to the (involuntary) cues of his trainer.
Further reading: