Well... yes and no.
If you have good privacy settings and you trust everyone that has access to the information, you are safe.
However, many people add people as friends who they have never met and give full access to everything. I have known some companies to create fake people in order to ask/add people.
In addition, it should be said, it is possible that you have a mutual friend who told on you!
update to comments
It is just common knowledge amongst people in recruitment and IT professionals that this sort of thing goes on, I am not really sure/not seen any sources.
So many people have thousands of friends, they have no idea who is who and add anyone who sends an invite.
If you do not believe me, create a new account, put some good looking girl as a picture and add add random people with over a few hundred friends, I guarantee that the majority will add you straight away, and out of those 60-70+% will allow you full access to their pictures and everything else.