- "Ser Arthur Dayne, the Sword of the Morning... led the attack on the Kingswood Brotherhood. Defeated the Smiling Knight in single combat."
- ―King Joffrey Baratheon
The Kingswood Brotherhood was a band of outlaws, cutthroats, bandits, and exiles who plagued the Kingswood, a forest south-east of King's Landing, approximately twenty years before the series begins. They were wiped out at the order of King Aerys II Targaryen, the Mad King.
Their leader, Simon Toyne, was killed by Ser Barristan Selmy in battle, while Jaime Lannister, then a squire of sixteen years, killed another member of the band. The Smiling Knight, another prominent member, was killed by Ser Arthur Dayne.
King Robert Baratheon is chatting with his Kingsguard and asks Jaime Lannister who the first man he ever killed was. Jaime says it was a member of the Kingswood Brotherhood, whom he decapitated with a swing from his sword. Barristan Selmy interjects and says he was there that day too. Jaime reverently says he still remembers how Ser Barristan killed Simon Toyne, the finest display of swordsmanship he had ever seen. Barristan modestly responds that Toyne was a good fighter, but he lacked stamina.
While imprisoned in Robb Stark's army camp, Jaime recounts in awe to fellow prisoner Alton Lannister how he squired for Ser Barristan and witnessed him the day he and the other Kingsguard defeated the Kingswood Brotherhood. Jaime recalls that Barristan moved like a master painter during the fight - who painted only in red.
King Joffrey Baratheon is flipping through The Book of Brothers and reading through some of its accounts about the exploits of all the great Kingsguard. He reads aloud part of Ser Arthur Dayne's entry, noting that he led the attack on the Kingswood Brotherhood (alongside Ser Barristan) and killed the Smiling Knight in single combat.
In the books
In the A Song of Ice and Fire novels, the Kingswood Brotherhood was a notorious outlaw gang founded by the brothers Toyne, two outcasts whose brother, a member of the Kingsguard, had been killed by King Aegon IV Targaryen for sleeping with one of his mistresses. They ambushed and killed Prince Aemon Targaryen, the infamous Dragonknight, before disappearing into the vastness of the Kingswood. They and their descendants troubled the realm for several generations before Ser Arthur Dayne led the force of knights and soldiers - which included Ser Barristan Selmy, Lord Sumner Crakehall, and his squire Jaime Lannister who was fifteen then - who finally wiped them out.
The Kingswood Brotherhood used to kidnap nobles for ransom. They were sheltered by the smallfolk of the Kingswood, who believed that the Brotherhood was the only who protected their rights. However, Ser Arthur gained the confidence of the smallfolk by offering them better rights, making sure that the royal forces searching for the outlaws in the region now paid the commoners for all goods and provisions they commandeered. This decision was crucial, as it turned the allegiance of the smallfolk to the royal forces, and afterwards they stopped helping the outlaws to hide (and possibly even gave the royal forces information about the outlaws' movements when they could).
Without the smallfolk to shelter them, the Brotherhood members were eventually brought down. Selmy killed the gang leader Simon Toyne, while Arthur Dayne killed the Smiling Knight.
Unlike what was mentioned on the TV show, Jaime Lannister did not kill any of the Brotherhood during the fight. He did, however, save his master Lord Crakehall from the outlaw Big Belly Ben, and crossed swords with the Smiling Knight. As a reward for his bravery, Arthur Dayne knighted Jaime on the battlefield. In the TV continuity, some of Arthur Dayne's actions were given to Barristan in early seasons, though starting in Season 4 Arthur Dayne started to be mentioned. The Season 4 Histories & Lore animated featurettes reconciled this by having Jaime explain that (in the TV continuity) he was knighted by both Barristan Selmy and Arthur Dayne at the conclusion of the fight against the Kingswood Brotherhood (more than one veteran knight leading the dubbing ceremony is not unheard of in the novels).
Other known members of the Brotherhood were Wenda the White Fawn, Fletcher Dick, Oswyn Longneck the Thrice Hanged, and Ulmer (an expert archer). All of the Brotherhood members are presumed dead except Ulmer, who joined the Night's Watch to avoid execution, and currently serves as a ranger.
Ulmer is a master archer. He claims that once put an arrow through the hand of Lord Commander of the Kingsguard Gerold Hightower to steal a kiss from a Dornish princess (the name of the princess in question is not revealed - perhaps it was Elia Martell) - then stole her jewels and a chest of golden dragons, but it was the kiss he liked to boast of in his cups. In the present day, though Ulmer is stooped and has a grey beard he is still one of the finest archers in the Night's Watch.
Ulmer takes part in the Great Ranging, and is among the survivors of the Battle of the Fist of the First Men. During the Mutiny at Craster's Keep, he is one of those who remain loyal to Lord Commander Mormont, and is one of the barely twenty men who actually manage to return to Castle Black alive - where he also manages to survive the ensuing assault of Mance Rayder's wildling horde. After the battle, and Stannis Baratheon's arrival at the Wall, Jon Snow tasks Ulmer with leading the archery lessons for new recruits. Samwell Tarly remarks to Jon that Ulmer has had a very long and interesting life, met all sorts of historical figures, and he'd really like to write his stories down in a book if he gets a chance (but then Samwell has to depart for Oldtown).