Washing your hands with plain water of a normal temperature is significantly less able to effectively sanitise against bacteria, viruses, and many protozoa. The oil on your skin will hold pathogens pretty effectively. The detergent in the soap helps to break down the cell walls in some types of prokaryotes (bacteria) and eukaryotes (other living microbes) and also to remove the oil so that virus particles can be removed in sufficient numbers. Different soaps and detergents will have varying levels of effectiveness. The CDC provides quite a bit of information about sanitisation. This page explains how to wash your hands effectively to remove pathogens.
Here is a quote from a study specifically comparing hand washing with and without soap:
Handwashing with water alone reduced the presence of bacteria to 23% ... Handwashing with plain soap and water reduced the presence of bacteria to 8% ... The effect did not appear to depend on the bacteria species (PubMed 21318017).
The above results may or may not be similar to those for viruses or protozoa. Also, the amount and type of oil on the skin before washing probably plays a large role, as probably does the method and material used to dry the skin. Different situations call for different approaches and levels of sanitation.
Interestingly, the same study mentioned that no specific instructions were given for the washing technique, other than using a paper towel to dry:
Participants assigned to handwashing were asked to wash their hands as they would normally do, without instructions on length of time or thoroughness. The volunteers allocated to handwashing were then provided with a paper towel to dry their hands ... Participants took on average 12 seconds to wash their hands with water alone, and 14 seconds to wash their hands with water and soap (PMC3037063).