TL;DR: Rated (old) urban legend
Based on this website, this story dates from as far back as 1897, from a magazine called "The Strand". This appears in section which looks like a collection of various jokes and "fancy stories". You can find the original article below (the one about the claim is in the bottom left corner):
This story has been attributed to Mr. Conan Doyle: A friend of his had
often been told that there is a skeleton in the cupboard of every
household, no matter how respectable that household may be; and he
determined to put this opinion to a practical test. Selecting for the
subject of his experiment a venerable Archdeacon of the Church,
against whom the most censorious critic had never breathed a word, he
went to the nearest post-office, and dispatched a telegram to the
revered gentleman: ‘All is discovered! Fly at once!’ The Archdeacon
disappeared, and has never been heard of since.
Turns out, there is a website keeping track of letters and various writings of A.C.D. which can be found here. A search with the keywords "flee", "discover(ed)", "12/twelve" wielded several results, but none matching the wording nor the context.
I also took a brief look at all the letters from before 1897, and none match the prank.
Finally, I looked up the Archdeacons of England which lived at the time, but none is listed before 1951.
Considering the absence of one of the original letters used for the prank, the uncertainty considering the number of recipients (one with the Archdeacon, 12 with the nobles, I found a few references about 5 letters, etc...) as well as the sole appearance of the prank in an article listing jokes, it appears it was already considered an urban legend even back then.
I rate the prank urban legend.