The answer seems to be in some cases that taking a break can help. Various studies have shown increases in productivity and focus after doing something that requires very little concentration after spending a lot of time on a a task that required a lot of concentration.
Other studies have shown that taking a break can have little effect while other studies show differences in results based on what you do during the break.
A study done at the University of Illinois showed that breaks were effective.
From Brief and rare mental ‘‘breaks’’ keep you focused: Deactivation and reactivation of task goals preempt vigilance decrements:
We asked observers to perform a visual vigilance task while
maintaining digits in-memory. When observers retrieved the digits at
the end of the vigilance task, their vigilance performance steeply
declined over time. However, when observers were asked to sporadically
recollect the digits during the vigilance task, the vigilance
decrement was averted. Our results present a direct challenge to the
pervasive view that vigilance decrements are due to a depletion of
attentional resources and provide a tractable mechanism to prevent
this insidious phenomenon in everyday life
- Link to study on Pubmed
...
A 2005 study drew a distinction between goal driven tasks and stimulus driven tasks. Since mental fatigue had little effect on stimulus orientated tasks then the effect from fatigue could be largely related to perception.
From Effects of mental fatigue on attention: An ERP study:
In summary, the effects of mental fatigue on behavior seem to a large
extent to be caused by an inability of fatigued subjects to allocate
their attention efficiently. However, a distinction must be made
between the effects of mental fatigue on goal-directed and
stimulus-driven attention. Goaldirected attention is shown to be
negatively affected by mental fatigue, while stimulus-driven attention
was largely unaffected. These results account for both the increased
distractibility as well as the decrease in flexibility that is
characteristic of fatigued people. When behavior becomes 114 M.A.S.
Boksem et al. / Cognitive Brain Research 25 (2005) 107– 116
increasingly stimulus-driven, salient stimuli in the environment will
have a greater influence on behavior. At the same time, goal-directed
control over behavior will decrease, causing behavior to be guided
more by automatic stimulus response couplings, resulting in a reduced
behavioral flexibility.
...
A study done at Stanford seemed to indicate that perception and/or emotional state could be more of a factor in mental efficieny than any potential benefit from a rest. From Ego Depletion—Is It All in Your Head? Implicit Theories About Willpower Affect Self-Regulation
Much recent research suggests that willpower—the capacity to exert
self-control—is a limited resource that is depleted after exertion. We
propose that whether depletion takes place or not depends on a
person’s belief about whether willpower is a limited resource. Study 1
found that individual differences in lay theories about willpower
moderate ego-depletion effects: People who viewed the capacity for
self-control as not limited did not show diminished self-control after
a depleting experience. Study 2 replicated the effect, manipulating
lay theories about willpower. Study 3 addressed questions about the
mechanism underlying the effect. Study 4, a longitudinal field study,
found that theories about willpower predict change in eating behavior,
procrastination, and self-regulated goal striving in depleting
circumstances. Taken together, the findings suggest that reduced
self-control after a depleting task or during demanding periods may
reflect people’s beliefs about the availability of willpower rather
than true resource depletion.
...
A study in 2008 seemed to show an improvement in attention depending on what was done on the break time. Students who walked through an Arboretum, improved their short-term memory by 20 percent, but showed no improvements after walking down city streets.
From The Cognitive Benefits of Interacting With Nature:
We compare the restorative effects on cognitive functioning of
interactions with natural versus urban environments. Attention
restoration theory (ART) provides an analysis of the kinds of
environments that lead to improvements in directed-attention
abilities. Nature, which is filled with intriguing stimuli, modestly
grabs attention in a bottom-up fashion, allowing top-down
directed-attention abilities a chance to replenish. Unlike natural
environments, urban environments are filled with stimulation that
captures attention dramatically and additionally requires directed
attention (e.g., to avoid being hit by a car), making them less
restorative.We present two experiments that show that walking in
nature or viewing pictures of nature can improve directed-attention
abilities as measured with a backwards digit-span task and the
Attention Network Task, thus validating attention restoration theory.
...
The studies indicate that in some cases taking a break was shown to have beneficial effects leading to increases in attention and short memory. Other studies indicated that perception and attitude can play an important role as well as what was done on the break. It's also important to keep in mind that the studies were limited to student populations which may not be a good representation of the wider population.
It's safe to say that for some people, taking a break can have benefits although it isn't possible to say that as a general rule taking a break will have benefits given the important role perception and attitude seems to play.