Realistically there is little to worry about from the mercury in a broken CFL
Using the same sources as Russell Steen's post above, I conclude there is little to worry about.
First many CFLs have a lot less than 4mg of mercury. Then only about 10% of the mercury is released when a bulb breaks (the rest is adsorbed onto the surfaces in the bulb and will go away with the broken glass). AS the source says:
Most mercury vapor inside fluorescent light bulbs becomes bound to the inside of the light bulb as it is used. EPA estimates that the rest of the mercury within a CFL – about 11 percent – is released into air or water when it is sent to a landfill, assuming the light bulb is broken
Then you need to take into account the circulation of air in the room. His original calculations assume no air turnover and that all the mercury is released. Under more realistic assumptions the peak concentration in a room will be 0.4mg/27 cubic meters or about 0.015 mg/m3 or substantially less than even the strict Californian standards from occupational exposure (i.e. how much you could be exposed to every day). Even this small level will decline rapidly as fresh air circulates through the room. So the worst case seems to involve lower than occupation standard exposure for a limited period.
As to worries about accumulation, there shouldn't be any. It is mercury vapour and will be swept away with the circulation of air in the room/house. Breaking a mercury thermometer and releasing liquid mercury is much worse (for a start there might be 500mg of mercury and it will stick around for a while releasing vapour which may well build up to a dangerous level if the ventilation is not good). UK sources suggest, though, that even this is not a worry as long as it is handled sensibly
If you break a mercury thermometer or light bulb, some liquid mercury may spill out, but probably very little. Liquid mercury can separate into small beads, which can roll some distance away. You may also be exposed to mercury vapour.
However, this small amount of mercury is extremely unlikely to cause problems for your health.
A health protection agency source also offers reassurance:
Exposure to airborne mercury vapour at concentrations near 0.05 milligrams per cubic metre (mg m-3) may result in an onset of mild to moderate symptoms...The concentrations described above of mercury in air are 1000 to 40000 times greater than that which could be produced from the quantities of mercury released from a broken domestic fever thermometer in a room.
Since the amounts released from a CFL bulb are 500-1000 times smaller than this, the immediate risks are probably very small. Moreover all the mercury in a CFL is vapour so there is no liquid mercury to hang around once the air has circulated.
Really high levels of mercury vapour have occasionally been observed in domestic premises but only because large volumes of liquid mercury have been dealt with badly (like by trying to use a vacuum cleaner which splits the blobs into smaller blobs and disperses them over a wider and warmer area over which air is flowing quickly leading to a drastic increase in the evaporation rate).
Chemists often work safely with large volumes of mercury, but are careful to clean up spills and to keep good ventilation to keep the vapour levels low. But there is so little mercury in CFLs that they don't count as a significant hazard.