Apparently not, as the following evidence shows, quoting from various doctors.
This is Doctor Richard Ferber: he states it is not the best option, but no damage occurs:
Cry-it-out stance: "Going 'cold turkey'—putting your child in the crib at bedtime, letting him cry, and not returning until morning—is far from ideal," writes Dr. Ferber. "[But] allowing some crying … will never cause psychological damage."
Dr. William Sears, a prominent opponent of the crying it out technique,
wrote an article called Science Says: Excessive Crying Could Be Harmful to Babies.
In it he notes:
One study showed infants who experienced persistent crying episodes were 10 times more likely to have ADHD as a child, along with poor school performance and antisocial behavior. The researchers concluded these findings may be due to the lack of responsive attitude of the parents toward their babies. 1
Researchers at Pennsylvania State and Arizona State Universities found that infants with excessive crying during the early months showed more difficulty controlling their emotions and became even fussier when parents tried to console them at 10 months.2
Chad Skelton, who did research on Dr. William Sears article:
Curious, I tracked down the two studies to see if I could learn more (the numbers beside each statement corresponded to two specific studies included in Sears' footnotes).
To my surprise, neither study (which you can read for yourself here and here) appeared to have anything at all to do with the cry-it-out method.
Skelton went on to say:
In fact, rather than looking at whether leaving children alone to cry (for whatever reason) caused problems later in life, both studies instead examined whether persistent crying in infancy was a symptom of underlying problems,
Indeed, the Penn State study even made a point of noting that how responsive a mother was to her chid's cries didn't appear to make a difference, since "the infant who cries excessively in early infancy will be likely inconsolable."
Professor Cynthia Stifter, lead author of the Penn State study, states:
there is nothing in the paper to suggest either 'crying it out' is an effective or ineffective method for getting children to sleep.
In fact, Professor Dieter Wolke wrote:
Extinction is a powerful learning method. However, in clinical practice, most parents find it very difficult to apply as they are very distressed by their baby’s crying. Thus, a staged process of “crying it out”, the checking method has been developed that is much more acceptable for many parents (i.e. check after 10 mins when baby is crying – but no reinforcement, i.e. no touching or picking up; next time after 15 mins checking, next time 20 mins etc.)
It's up to you to decide from the evidence above, whether or not it is "neglectful".