Snopes has an article on the "Charlie Charlie Challenge" and they point out that, at the least, the origin is a bit silly:
Prior to May 2015, few mentions appeared on the Internet of any practice fitting the description of the Charlie Charlie Challenge. If Mexican folklore featured it at all, it was a well-hidden secret; the BBC quoted a regional correspondent about the dubious ancient origins of the game (noting that "Charlie" is not a Mexican name, nor does any known folkloric entity of that culture resemble the one summoned):
Some have claimed that it has been revealed as an attempt at viral marketing for a film named The Gallows. It's also possible that the marketing team for The Gallows appropriated the meme.
As to how it works, it's the ideomotor phenomenon at work, the same thing behind automatic writing, Ouija boards, and pendulum divination:
The ideomotor response (or "ideomotor reflex"), often abbreviated to IMR, is a concept in hypnosis and psychological research. It is derived from the terms "ideo" (idea, or mental representation) and "motor" (muscular action). The phrase is most commonly used in reference to the process whereby a thought or mental image brings about a seemingly "reflexive" or automatic muscular reaction, often of minuscule degree, and potentially outside of the awareness of the subject.
Basically, no matter how still you try to keep your hands, they twitch. The more you notice the twitch and try to still it, the greater the twitches get. From there, you get into various debates as to whether people subconsciously pick out answers in their motions, or if all meaning comes from interpretation. In versions where there's ostensibly no forces acting on it (such as setting it on the table), it's due to the fact that everything is moving and vibrating unless you do very specific controls on the environment, and once an object starts moving in a direction, random motion is likely to keep it moving that way until it is stopped, such as by friction. And, of course, that's assuming that no one at the table is doing a seance trick of intentionally influencing the motion by jiggling the table or blowing surreptitiously.