There was, particularly amongst veterans who had been captured and imprisoned in harsh POW camps, a great deal of animosity which lasted decades after the war.
An 1949 article in the former Melbourne newspaper The Argus quotes an RSL spokesman saying:
Australian ex-servicemen would seriously resent the importation of Japanese cars to Australia
In 1970, Bill Bourke, the managing director of Ford Australia, was quoted in the Sydney Morning Herald saying:
I have told our dealers that we are looking forward to the day we don't see any Japanese cars in R.S.L. parking lots.
He wanted to sell more Australian-made cars, but the clear implication there is that Japanese cars could be found in RSL parking lots in the 60s.
Roger Pulver's article for Japan times may have been the source for Abbott's statement there:
For example, Japanese cars were not welcome in parking lots of clubs run by the country’s leading war veterans’ organization, the Returned and Services League of Australia.
However, Menzies retired in 1966, Bourke's quote implied that Japanese cars had not been banned from RSLs as at 1970 and Pulver first came to Australia in 1972, so it doesn't really support the speech's chronology.
At the end of the day, it seems that practicality was the determinining factor.
Not buying Japanese brands and Japanese made products was one common attitude. After the war, Jared and his wife had only item in the home that was Japanese made – their family Toyota car. Their story was a familiar one; many former Australian POWs families would have preferred not to buy Japanese cars if it had not been for their cheaper prices.