Mostly, it's the cost of the car itself, not any externalities. High-density batteries cost a lot of money for the amount of energy they store - especially in comparison to gas - and cheap batteries have such low energy density that you may as well not even bother. Compare a Tesla Roadster at to a gasoline car of similar performance, which you can typically get at half that cost from premium brands like Porsche. Likewise, a Nissan Leaf costs around $32,780 (link includes original press release), while a comparatively equipped Toyota Matrix costs around $22,900.
Right now, that cost is almost at a point where it can be considered feasible. For example, you can reasonably expect to spend $10,000 in gas[1] (and much, much less on electricity for the Leaf) for that Matrix inside of about 8 years, but you will probably have to upgrade the battery in the Leaf after about 10 or 12 years (Nissan says they plan to produce a Leaf with twice the battery capacity by 2015). That said, you don't have to look far to find other examples of similarly-sized, gas powered hatchbacks at the $32,000 price point, either (the Audi A3 or Mercedes Benz B-class for example).
The fact that we are at this point is a testament to recent improvements in battery density and cost, and in the future we will very likely see electric cars that cost the same or only slightly more than gas-powered ones. In the past (say, in 1998), the idea was completely infeasible from an economic standpoint[2], which is why so few automakers are making electric cars today. Next year, that will change as several automakers that aren't already making electrics are bringing them to market, and several more also have plans in place to do so in the next two or three years.
So to say that there's a conspiracy among automakers to not make electric cars[3] is not true since at least 5 - GM, Nissan, Ford, Volvo, and VW are either now making them, or explicitly plan to in the near future. That oil companies are actively trying to suppress the technology may be true, but it doesn't seem to be in their hands anymore. Chevron may own the patent for NIMH battery chemistry, but it doesn't for the Lithium Ion batteries that are now favoured for electric cars.
FOOTNOTES:
[1] Link shows my own analysis, showing the math to demonstrate how I got there. I adjusted downwards from $16,640 to $10,000 to reflect that my pricing is in Canadian dollars, with local gas prices being about 40% higher than most of the US, plus the fact that gasoline prices are anything but stable. If anything, $10,000 USD is a low estimate, but it certainly falls within the difference in purchase price between a Leaf and a Matrix. I used a Toyota Matrix in my comparison because the car's dimensions are almost exactly the same (the Leaf is 4cm longer but otherwise identical), and the features of the Matrix S are very similar to the Leaf's base model, with similar power output.
[2] Wikipedia references a Washington Post article that is no longer online, stating that each EV1 produced cost GM about $80,000. Another reference in book form states that they cost $100,000 each. This was far in excess of what GM was leasing the cars for, in any case.
[3] Yes, another Wikipedia reference, but they bring together many sources that do a better job of describing this phenomenon than I ever could in my short article, which is decidedly not about any such conspiracy anyway.