Before developing video games, Nintendo owned 'Love Hotels.' Private short-stay rooms that were used for sexual encounters.
It appears all modern references to this claim come from the book Game Over: How Nintendo Zapped an American Industry, Captured Your Dollars, and Enslaved Your Children by David Sheff and published in 1993.
In it, he claims:
The first product launched by the new company was a line of individually portioned instant rice. Add water and - presto! It was a dismal failure. Yamauchi then opened a "love hotel," with rooms rented by the hour. The business was, for Yamauchi, a personal passion; it was said that he was one of his own best customers (his infidelities were well known - even by his wife, who ignored them).
A taxi company Yamauchi started, Daiya, thrived, although he grew tired of negotiating with powerful taxi-driver unions, which demanded high salaries and expensive benefits for their members. he soon folded that company and closed the doors of the love hotel. he planned more changes as he move Nintendo again, this time to a larger building, a three-story structure of beige bricks with black door and window frames and bars on the windows.
Yamauchi had concluded that he wanted new businesses tha tcould take advantage of one of Nintendo's strongest assets, its karuta distribution system, which reached into toy and department stores throughout Japan. Nintendo's roots were in entertainment, and there would be no more rice or taxis or love hotels. Yamauchi set Nintendo on a new course as an entertainment company.
The Google Books version doesn't include any references, but Wikipedia notes:
The book is notable in that the author extensively interviewed numerous established figures in the industry, such as Howard Lincoln, Nolan Bushnell, Shigeru Miyamoto (spelled as "Sigeru" in the book), Alexey Pajitnov, and others, including people who spoke anonymously. Despite the title, the book is fairly neutral; it mainly relates the history of the company while looking at both the positives and negatives of their business practices.
I think it's reasonable to assume that David Sheff did have enough sources for the story of the three attempts at diversifying to justify including them in his book. However, I can't tell you what they are.
There's also an interesting reference to them here, where the author is describing Yamauchi's relationship with his daughter