The figures vary. There may or may not be more British Muslims serving Britain than Islamist groups, but figures indicate that Muslims are under-represented in the armed forces.

From The Guardian, [British army aims to recruit more Muslims after worries over low numbers][2] (talking about the army, not the armed forces as a whole)

> There are 480 Muslims in the army out of an overall strength of 88,500
> – 0.54%, an increase from 300 in 2008. Muslims comprise about 4.4% of
> the UK population according to the 2011 census.

An opinion piece [Muslims in the armed forces: A proud tradition][3]

> Despite this, the figures in absolute terms are still small, with
> around 650 Muslims in the 200,000-strong military making up less than
> 1 per cent of its ratio strength.

From the BBC: [UK's Muslim soldiers ‘fighting extremists not Muslims'][4]

> With more than 600 Muslims in the British Armed Forces, do those that
> are deployed on the front line in Afghanistan have to reconcile their
> beliefs in order to fight hardline Islamic Taliban militants?

Caveats:

 - I haven't critically examined the number of British Muslims serving Islamist groups.
 - Could estimates about the number of British Muslims be incorrect? Maybe there's people who are Muslim, but haven't mentioned their religious affiliation to those collecting statistics.
 - Some of those serving the British Armed Forces may be women. It's possible the Islamists have more men, but the British Armed Forces have more people overall.

  [1]: http://www.independent.co.uk/voices/comment/muslims-in-the-armed-forces-a-proud-tradition-9226715.html
  [2]: http://www.theguardian.com/uk-news/2015/feb/06/british-army-recruit-muslims-low-number-iraq-afghanistan
  [3]: http://www.independent.co.uk/voices/comment/muslims-in-the-armed-forces-a-proud-tradition-9226715.html
  [4]: http://www.bbc.com/news/uk-12504162