Following Varys's escape from Westeros with Tyrion Lannister, however, the office was left vacant. When Queen Cersei reorganized the mall council after Tyrion murdered their father Tywin, she took control over the council and stacked it with sycophants. One of her actions was to appoint Qyburn as the new Master of Whisperers - despite the fact that he is a disgraced former maester who was expelled from the Citadel after it was learned he was conducting human experimentation, dissecting living men. Cersei ignored Grand Maester Pycelle's objections to this controversial choice- including asking what qualifications Qyburn even had for such a post - by simply saying that he had demonstrated his loyalty to her, more than Varys ever did. Minutes after this announcement, along with Cersei's other choices in restaffing the council, her uncle Kevan Lannister bluntly declared that she was stacking the Small Council with sycophants, and left for Casterly Rock in disgust.
Varys, a eunuch from the Free Cities, called "the Spider". Began serving in the office during the reign of the Mad King, and continued in it through the reigns of King Robert, King Joffrey, and the early months of King Tommen's reign. Although Varys is exceptionally skilled at gathering information, he is secretly plotting to support a Targaryen restoration, believing this will bring about peace and justice in the Seven Kingdoms, which eventually led him to desert his post and flee Westeros with Tyrion Lannister. He now serves as Master of Whisperers to Daenerys Targaryen, Queen of Meereen and a claimant to the Iron Throne.
Qyburn, an ex-maester expelled in shame from the order, appointed by Cersei because she needed a loyal crony on the council. Also currently serves as Hand of Queen Cersei.
In the A Song of Ice and Fire novels, Varys was Master of Whisperers under King Aerys II Targaryen at the time of Robert's Rebellion. He cautioned the king not to let Tywin Lannister's army within the gates, but Grand MaesterPycelle (secretly Tywin's agent) convinced the Mad King that Tywin had come to turn the tide of the war back in his favor, and to let his army inside the gates. As soon as the army entered the Lannisters turned on the Targaryens, in the destructive Sack of King's Landing. After Robert Baratheon was crowned king, he pardoned Varys and kept him in his position as Master of Whisperers, which he would maintain through the rest of Robert's rule. Thus, by the beginning of the narrative, Varys has been the head of the realm's spy network for over twenty years. It is unknown specifically why Robert pardoned Varys, much less kept him in such a sensitive position, though it may simply be because he was very good at his job (though this is revealed to be detrimental to Robert, as Varys secretly remains a Targaryen loyalist, in contact with Illyrio Mopatis in orchestrating the affairs of Prince Viserys Targaryen, and the marriage of his sister Princess Daenerys Targaryen to Khal Drogo).
The World of Ice and Fire sourcebook (2014) mentions other masters of whisperers in passing. The Targaryens actually had a long unofficial tradition of recruiting unpopular Masters of Whisperers from the Free Cities, so Varys's situation is actually not that unusual. The other prominent holders of the office were actually Mistresses of Whisperers, being women who seem to have held the title openly (and were also actual mistresses to the kings in question).