There has been a lot of work on this subject, most of which has even been peer reviewed! Even 30+ years ago, there were studies confirming confirmatory bias in peer review:
...we are left with little to defend [the peer review process] other than tradition.
("Confirmatory bias" is one's tendancy to agree with conclusions that agree with one's own preconceived notions and hypotheses.)
That study took 75 journal reviewers (I believe they were all from the cognitive sciences community) and presented them with papers to review. The catch was that the papers were all identical in their experimental setup and data, however, they varied in their conclusions of the results. The majority of the reviewers agreed with the conclusions that were similar to their own theoretical perspective.
Peer review of research grant proposals isn't much better; this study from around the same time period concluded that being awarded a peer-reviewed research grant falls mostly to chance:
The degree of disagreement within the population of eligible reviewers is such that whether or not a proposal is funded depends in a large proportion of cases upon which reviewers happen to be selected for it. No evidence of systematic bias in the selection of NSF reviewers was found.
From a philosophical perspective peer-review can be very effective, however, it relies mostly on the quality of the reviewer:
Peer review can be performed successfully only if those involved have a clear idea as to its fundamental purpose. Most authors of articles on the subject assume that the purpose of peer review is quality control. This is an inadequate answer. The fundamental purpose of peer review in the biomedical sciences must be consistent with that of medicine itself, to cure sometimes, to relieve often, to comfort always. Peer review must therefore aim to facilitate the introduction into medicine of improved ways of curing, relieving, and comforting patients. The fulfillment of this aim requires both quality control and the encouragement of innovation. If an appropriate balance between the two is lost, then peer review will fail to fulfill its purpose.
This more recent meta-study aggregated the results of 19 studies on peer review (having pruned away 116 other studies that did not meet their criteria). They found that each study had widely different conclusions, many of which may not be generalizable. They conclude that
Editorial peer review, although widely used, is largely untested and its effects are uncertain.
Some studies did show that peer review is effectual in improving the readability of articles.
So, in summary, there is a lot of evidence to support the fact that the effect of peer review is often indistinguishable from chance. It has been perpetuated due to tradition; until very recently, the dissemination of scholarly information was technologically and economically limited to distribution via printed journals, and peer review provided a means for limiting the size of those expensive journals.
For some more amusing reading on this subject, I suggest you check out the opinions of Doron Zeilberger (e.g., this one).