Except for some small details, the story appears to be true.
- It was Christmas Day (not Boxing Day)
- He might have only been out there for 15 minutes after that (instead of 20).
Sam Bartram was the goalkeeper, and his autobiography details the event, parts of which have been quoted by various sources. The ESPN FC version also gives some background, in addition:
On Christmas Day 1937, Bartram was in the papers once more after a bizarre incident in a match against Chelsea at Stamford Bridge. With the score at 1-1, the game had to be called off on 61 minutes due to thick fog. Unfortunately for Bartram, he was the last to be made aware. "Soon after the kick-off, [fog] began to thicken rapidly at the far end, travelling past Vic Woodley in the Chelsea goal and rolling steadily towards me," he wrote in his autobiography. "The referee stopped the game, and then, as visibility became clearer, restarted it. We were on top at this time, and I saw fewer and fewer figures as we attacked steadily.
"I paced up and down my goal-line, happy in the knowledge that Chelsea were being pinned in their own half. 'The boys must be giving the Pensioners the hammer,' I thought smugly, as I stamped my feet for warmth. Quite obviously, however, we were not getting the ball into the net. For no players were coming back to line up, as they would have done following a goal. Time passed, and I made several advances towards the edge of the penalty area, peering through the murk, which was getting thicker every minute. Still I could see nothing. The Chelsea defence was clearly being run off its feet.
"After a long time a figure loomed out of the curtain of fog in front of me. It was a policeman, and he gaped at me incredulously. 'What on earth are you doing here?' he gasped. 'The game was stopped a quarter of an hour ago. The field's completely empty'. And when I groped my way to the dressing-room, the rest of the Charlton team, already out of the bath and in their civvies, were convulsed with laughter."
This story is also backed up by the Chelsea F.C. Chronicle (27 April 1938):

(Click for a bigger image.)
There are likely other sources that document this, such as newspapers, but it may be difficult to find them.
The image is from a different match.
The Press Association is the source of the image:

2 January 1954
Arsenal goalkeeper Jack Kelsey peers into the fog, searching for the elusive ball. The fog was so thick the game was eventually stopped.
The Guardian features the full image without the watermark.