Following Varys's escape from Westeros with Tyrion Lannister, however, the office was left vacant. When Queen Cersei reorganized the small council after Tyrion murdered their father Tywin, she took control over the council and stacked it with sycophants. One of her actions was to appoint Qyburn as the new Master of Whisperers - despite the fact that he is a disgraced former maester who was expelled from the Citadel after it was learned he was conducting human experimentation, dissecting living men. Cersei ignored Grand Maester Pycelle's objections to this controversial choice- including asking what qualifications Qyburn even had for such a post - by simply saying that he had demonstrated his loyalty to her, more than Varys ever did. Minutes after this announcement, along with Cersei's other choices in restaffing the council, her uncle Kevan Lannister bluntly declared that she was stacking the Small Council with sycophants, and left for Casterly Rock in disgust.
In the A Song of Ice and Fire novels, Varys was Master of Whisperers under King Aerys II Targaryen at the time of Robert's Rebellion. He cautioned the king not to let Tywin Lannister's army within the gates, but Grand MaesterPycelle (secretly Tywin's agent) convinced the Mad King that Tywin had come to turn the tide of the war back in his favor, and to let his army inside the gates. As soon as the army entered the Lannisters turned on the Targaryens, in the destructive Sack of King's Landing. After Robert Baratheon was crowned king, he pardoned Varys and kept him in his position as Master of Whisperers, which he would maintain through the rest of Robert's rule. Thus, by the beginning of the narrative, Varys has been the head of the realm's spy network for over twenty years. It is unknown specifically why Robert pardoned Varys, let alone kept him in such a sensitive position, though it may simply be because he was very good at his job. In truth, this was very detrimental to Robert, as Varys secretly remains a Targaryen loyalist, in contact with Illyrio Mopatis in orchestrating the affairs of Prince Viserys Targaryen, and the marriage of his sister Princess Daenerys Targaryen to Khal Drogo, although the only Small Councilors who ever learned the truth did so in their last moments of life.
The World of Ice and Fire sourcebook (2014) mentions other masters of whisperers in passing. The Targaryens actually had a long unofficial tradition of recruiting unpopular Masters of Whisperers from the Free Cities, so Varys's situation is actually not that unusual. The other prominent holders of the office were actually Mistresses of Whisperers, being women who seem to have held the title openly (and were also actual mistresses to the kings in question).