Due to their chemical properties, many drugs have ill effects on the receptors that they bind to; take for example MDMA which is nucleophilic, and thus has a tendency to occasionally rip protons away from the proteins that it comes into contact with. LSD is more or less inert in your brain tissue, and thus is considered to be non-neurotoxic.
Other drugs may induce changes in the body's cardiovascular system, either increasing or decreasing heart rate or breathing to a dangerous level. LSD itself does this quite negligibly. It's important to remember, however, that the extreme emotions involved in LSD experiences have been known to induce these dangerous side effects, and cases of such fatalities have been reported.
In addition, you're correct that other psychiatric complications may occur, and contribute to LSD not being entirely harmless. While I have never distinguished any lasting mental effects from my own LSD experiences, many people report personality alterations after profound doses, whether for better or for worse (depending on the nature of the trip itself). LSD has even been purported to cause early symptoms in individuals with latent mental conditions related to the serotonin system (I.E., schizophrenia). One should only take LSD if they are confident with their mental and emotional condition.