The most prominent example of the Targaryen madness was in the last Targaryen king, Aerys II, who subsequently became commonly known as "the Mad King". Aerys II suffered from some sort of paranoid schizophrenia, hearing voices in his head which weren't real which told him to burn people alive because they were all plotting against him.
Other Targaryens inherited highly bizarre personalities, though they were not strictly speaking "insane", from the technical standpoint that they didn't have hallucinations or hear voices. Some had aggressive, sociopathic personalities, such as Aerion Brightflame - older brother of Maester Aemon. Eventually, Aerion succumbed to the delusional belief that if he drank wildfire it would transform him into a dragon: he was wrong.
The effects worsened in with each subsequent generation of compounded inbreeding. By the later centuries of their rule, it was joked that when a new Targaryen was born the gods would flip a coin to decide if it would be a brilliant statesman or insane. This does not mean that all Targaryens were as a rule mentally unstable, as seen with how Daenerys Targaryen is mentally stable while her own brother Viserys is disturbed. Daenerys' older brother Rhaegar was considered a great man by all who knew him.
Generally, if a Targaryen had multiple sons and one of them was clearly mentally unstable, they would try to skip over him in line of succession for a more stable younger son. Therefore, fewer of the early Targaryen kings had mental problems, because if a king did have a mentally unstable son he wouldn't be allowed to succeed his father. However during the last century of their rule, the Targaryen madness became increasingly common, making it more and more difficult to avoid having a mentally unstable son inherit the throne.
Known Mad Targaryens
- Prince Aerion Brightflame, who drank wildfire to turn himself into a dragon.
- King Aerys the Mad
- King Viserys, the Beggar King
In the books
On a few occasions, however, Targaryens were born who had benign but nonetheless delusional personalities, such as Baelor "the Blessed". Baelor was admittedly a well-intentioned man but lost his grip on reality, becoming obsessed with religion - albeit not in a puritanical way but by carrying altruism and pacifism to an absurd degree. An overzealous fool, he prayed more than he ruled, wore only a crown of flowers, and died without heirs because he never consummated his marriage for fear of sinning, and even locked his three sisters away in a tower (mockingly called the "Maidenvault") to avoid carnal temptations. He made a stonemason the new High Septon due to the beauty of the religious statues he carved - despite the fact that he was illiterate and knew no prayers - believing him the Smith in human form. After the stonemason died, Baelor replaced him by appointing an eight year old boy as High Septon.
There are also examples of Targaryen children who were born mentally retarded; however because they were never seriously considered in the line of succession little was recorded about them and thus they are given only scant mentions in the royal histories. The daughter of Aerion and Aemon's oldest brother is recorded as being born "feeble-witted".
The generations of inbreeding practiced by the Targaryens usually resulted in mental health problems, though at times they also produced severe detriments to physical health. Some Targaryens were intellectually and emotionally normal, but possessed such fragile health that they suffered from numerous ailments and died young. A prominent example of this was King Jaehaerys II, a good and honorable ruler who possessed frail health and died after ruling for only three years (however, Jaehaerys II has been officially cut from the TV series continuity).
Nor was the "Targaryen Madness" always initially present, but sometimes grew worse with age. At first, Daenerys wasn't willing to believe the tales of her father's madness as anything more than rumors spread by Robert Baratheon, but after Barristan Selmy joins her service, at her request he gives her a frank report on the nature of Targaryen madness. Aerys II himself, while remembered as "the Mad King", was perfectly sane even as a young adult, but as the years passed had lapses into unstable behavior which gradually grew worse and worse, and longer in duration. Many were willing to ignore these eccentric episodes when they passed, but by his mid-40's Aerys had become completely unstable, hearing voices that weren't there, becoming irrationally paranoid, and burning men alive for his own amusement.
Daenerys becomes deeply concerned when she learns that Targaryen madness was often a late-onset condition: her own infamously insane father showed little sign of mental instability into his late-20's, thus there is no real guarantee that the apparently-sane teenaged Daenerys will not suddenly turn violently insane at some point in the future; either one year in the future or twenty years in the future.
House Baratheon in the current generation is closely related to the Targaryens, as Robert Baratheon's paternal grandmother was a Targaryen, thus making Robert's father the first cousin of King Aerys II. Every now and again when certain people disparage the three Baratheon brothers, they sometimes speculate that their "drop of dragon-blood" from their partial Targaryen descent made them a little odd. For example, Olenna Tyrell thought this was a contributing factor in what made Renly Baratheon so rash that he thought he could proclaim himself king even though Robert had two sons and a middle brother Stannis before Renly in the line of succession.
- Targaryen madness at A Wiki of Ice and Fire (MAJOR spoilers from the books)