April 2-4 update(s): the WHO has now commissioned a panel to re-examine the problem of mask use by the general public:
Should more of us wear face masks to help slow the spread of coronavirus?
This question is to be assessed by a panel of advisers to the World Health Organization (WHO).
The group will weigh up research on whether the virus can be projected further than previously thought; a study in the US suggests coughs can reach 6m and sneezes up to 8m.
The panel's chair, Prof David Heymann, told BBC News that the new research may lead to a shift in advice about masks.
The former director at the WHO explained: "The WHO is opening up its discussion again looking at the new evidence to see whether or not there should be a change in the way it's recommending masks should be used.
There's a leaked CDC memo that they are also reconsidering their stance:
US health authorities are debating whether to recommend face coverings for everyone when they go out in public.
One internal memo for the Centers for Disease Control and Prevention says that even simple cloth masks could help reduce the risk of virus transmission, the Washington Post reports. [...]
Dr Fauci told CNN he would "lean towards" recommending a "much more broad, community-wide use of masks outside of the health care setting" once there was a sufficient supply of masks for healthcare workers.
However, experts have stressed that it is vital to maintain social distancing measures and thorough hand washing, even when wearing a mask.
As it turned out (April 3) this became an CDC official recommendation now, see Johnathan' own answer below. Some over governments, like the UK's found however this CDC change of stance unconvincing for them though (insofar, April 4):
Meanwhile, the Government confirmed that it does not recommend that healthy people wear face masks in light of the US adding it to their advice.
Professor Jonathan Van-Tam, deputy chief medical officer for England, said the practice seemed “wired into” some South East Asian cultures but there was no evidence it helped stop the spread.
Original answer below:
I think that relying on a single study (esp. one just on healthcare workers) can be a bit misleading here. Here's an excerpt from a 2016 review that covers that study and more: [Almost every sentence has a footnote/citation, but I'm stripping them out here since the review is open access]
In the 1919 influenza pandemic, masks were available and were dispensed to populations, but they had no impact on the epidemic curve. At the time, it was unknown that the influenza organism is nanoscopic and can theoretically penetrate the surgical mask barrier. As recently as 2010, the US National Academy of Sciences declared that, in the community setting, “face masks are not designed or certified to protect the wearer from exposure to respiratory hazards.” A number of studies have shown the inefficacy of the surgical mask in household settings to prevent transmission of the influenza virus, but Smith and colleagues, in a recently published meta-analysis, concluded that the surgical mask was noninferior to the N95 mask in terms of influenza transmission rates among health care workers. So … health care workers should wear masks to prevent transmission for reasons other than source control, but the public shouldn’t? [...]
In an annex to the Canadian pandemic influenza preparedness plan covering public health measures, the Public Health Agency of Canada (PHAC) does not recommend the use of masks by well individuals in pandemic situations, acknowledging that the mask has not been shown to be effective in such circumstances. However, this stance is complicated by the PHAC’s supporting reasons, which relate to problems of supply, cost, distribution and feasibility: panic might occur if the availability of masks were limited; public purchase of masks might limit the availability of masks in health care settings where they are required; and not all members of the public can afford to purchase masks — if masks are recommended by public health authorities, there could be an expectation that they will be publicly funded and made available by public health programs.
The sentence I bolded on household setting is citing two studies both from 2009, but neither is the one from the accepted answer (because that's on healthcare workers, which is included in the 2016 meta-analysis I'll quote from later on)
Many countries are stockpiling face masks for use as nonpharmaceutical interventions to reduce viral transmission during an influenza pandemic. We conducted a prospective cluster-randomized trial comparing surgical masks, non–fit-tested P2 masks, and no masks in prevention of influenza-like illness (ILI) in households. [...] Adherence to mask use was associated with a significantly reduced risk of ILI-associated infection. We concluded that household use of masks is associated with low adherence and is ineffective in controlling seasonal ILI. If adherence were greater, mask use might reduce transmission during a severe influenza pandemic.
INTERVENTION: Lifestyle education (control) (134 households), hand hygiene (136 households), or surgical facemasks plus hand hygiene (137 households) for all household members.
RESULTS:
Sixty (8%) contacts in the 259 households had RT-PCR-confirmed influenza virus infection in the 7 days after intervention. Hand hygiene with or without facemasks seemed to reduce influenza transmission, but the differences compared with the control group were not significant. In 154 households in which interventions were implemented within 36 hours of symptom onset in the index patient, transmission of RT-PCR-confirmed infection seemed reduced, an effect attributable to fewer infections among participants using facemasks plus hand hygiene (adjusted odds ratio, 0.33 [95% CI, 0.13 to 0.87]). Adherence to interventions varied.
Noteworthy is that this was a multi-intervention study, but only the "combo" had a noticeable effect, and hand hygiene alone appears to have been slightly more effective than masks alone, in this setting.
The "recently published meta-analysis" in the first quote in my answer is not the same as the study in the accepted answer but a more recent (2016) one albeit also on healthcare workers.
Conflicting recommendations exist related to which facial protection should be used by health care workers to prevent transmission of acute respiratory infections, including pandemic influenza. [...]
We identified 6 clinical studies (3 RCTs, 1 cohort study and 2 case–control studies) and 23 surrogate exposure studies. In the meta-analysis of the clinical studies, we found no significant difference between N95 respirators and surgical masks in associated risk of (a) laboratory-confirmed respiratory infection [...] (b) influenza-like illness [...] or (c) reported workplace absenteeism [...]. In the surrogate exposure studies, N95 respirators were associated with less filter penetration, less face-seal leakage and less total inward leakage under laboratory experimental conditions, compared with surgical masks.
Interpretation:
Although N95 respirators appeared to have a protective advantage over surgical masks in laboratory settings, our meta-analysis showed that there were insufficient data to determine definitively whether N95 respirators are superior to surgical masks in protecting health care workers against transmissible acute respiratory infections in clinical settings. [...]
Randomized controlled trials (RCTs) and observational studies comparing N95 respirators and surgical masks have not shown a benefit, but they may have been underpowered.
The lack of clarity has led to conflicting guideline recommendations regarding respiratory protective equipment for the prevention of acute respiratory infections: N95 respirators are recommended in some guidelines but not others.
[...]
None of the studies included in the meta-analysis, except the RCT by Loeb and colleagues,[this is the one from the accepted answer] independently audited compliance with the intervention. Potential confounding due to concurrent interventions (e.g., gloves, gowns and hand hygiene practices) as part of routine and additional precautions for droplet transmission were not accounted for by our meta-analysis.
There's another meta-analysis by Offeddu et al. (2017), which is broader in
scope (i.e. not just surgical masks vs N95) but subjects-wise also limited to healthcare workers:
This systematic review and meta-analysis quantified the protective effect of facemasks and respirators against respiratory infections among healthcare workers. [...] Meta-analysis of randomized controlled trials (RCTs) indicated a protective effect of masks and respirators against clinical respiratory illness (CRI) [...] and influenza-like illness (ILI) [...]. Compared to masks, N95 respirators conferred superior protection against CRI [...] and laboratory-confirmed bacterial [...], but not viral infections or ILI.
Meta-analysis of observational studies provided evidence of a protective effect of masks [...] and respirators [...] against severe acute respiratory syndrome (SARS). This systematic review and meta-analysis
supports the use of respiratory protection. However, the existing evidence is sparse and findings are inconsistent within and across studies.
Notably, since 2009 when indeed there was no RCT on "mask vs nothing" (as the accepted answer says) such studies were conducted in 2011/2015, as summarized by the following figure in Offeddu

Continuous Respiratory Personal Protective Equipment Use vs No
Respiratory Personal Protective Equipment
Two RCTs compared respiratory infection risk in HCWs
wearing rPPE continuously to convenience-selected controls
wearing no rPPE [...] or following routine care [...]. Wearing
a medical mask or N95 respirator throughout the work shift
conferred significant protection against self-reported clinical respiratory illness (CRI) (RR = 0.59; 95% CI: 0.46–0.77)
(Figure 2A) and influenza-like illness (ILI) (RR = 0.34; 95% CI:
0.14–0.82) (Figure 2B). Meta-analysis suggested a protective
but nonstatistically significant, effect against laboratory-confirmed viral infections (VRI) (RR = 0.70; 95% CI: 0.47–1.03)
(Figure 2C).
The paper does discuss the caveats of these two studies: self-reported outcomes and (obvious) non-blinded:
Compared to non-rPPE wearing HCWs, those wearing medical masks or N95 respirators throughout their work shift were
significantly protected against nonspecific respiratory infection.
However, assessment of clinical outcomes was self-reported and
prone to bias, as the intervention cannot be masked. Evidence
of a protective effect of masks or respirators against VRI, a rarer
outcome, was not statistically significant, though this may indicate insufficient statistical power in these studies, rather than
lack of a protective effect. [...]
The meta analysis also notes rather bluntly that the MacIntyre's studies "were conducted in China [...] limiting generalizability to other settings".
It's worth repeating that this latter study/meta-analysis is not about general public use. There doesn't seem to be much in the way of studies on the latter, but from expert opinion on CNN, the issue seems to be that people not used to wearing masks touch their faces a lot more often when they do wear mask... So presumably the increased risk factor from face touching "cancels out" the benefits of wearing a mask in the general/unhabituated public.
US Surgeon General Dr. Jerome Adams not only wants people to stop buying facemasks to prevent the novel coronavirus, but warns that you actually might increase your risk of infection if facemasks are not worn properly.
"You can increase your risk of getting it by wearing a mask if you are not a health care provider," Adams said during an interview on Fox & Friends on Monday morning.
"Folks who don't know how to wear them properly tend to touch their faces a lot and actually can increase the spread of coronavirus," Adams said.
I think there aren't really any studies testing out this particular [face touching] theory as to why laboratory effectiveness of masks is diminished in the general public. Probably the study by Cowling I discussed earlier has some relevance, as [the lack of] hand hygiene was found to diminish mask effectiveness in that study (to the point of statistical irrelevance.)
But the overall picture seems clear: masks are effective in the usual clinical setting in which they are accompanied by other hygiene and protective measures, and better masks might be marginally more so, but adherence to their "proper" use in household settings seems to be significantly reducing their effectiveness. (What exactly besides inconsistent use is hindering their effectiveness in the household setting is a bit difficult to say, but hand hygiene seems to be factor.)