The 卍 ( "ban" ) character is common at many temples, and predates Nazism by many centuries. Also note the mirror image and the rotation compared to 卐, the swastika used by the German Nazi party.
Japan makes no particular associations between the character and the Third Reich - it appears on many street maps, tourist guides and the like.
It's carved in stone in front of Sensoji, one of the largest Buddhist temples in Tokyo and a major tourist attraction.
The maps printed in English avoid the character over prolonged complaints from the tourists. Still used extensively in Japanese online maps in areas like Kyoto, where there's a temple every few blocks.1
Other segments of Japan, like publishing, have learned the hard way what the rest of the world thinks but they aren't that interested in changing anything. After all, they had it first. There's also a "KKK Building" in Ginza2 and a popular beverage called Calpis.
- JR Kyoto Station, tourist information counter. Ask for a map.
- Chuo-dori, Ginza. West side of the street.