- "...everyone hates the Cave people."
- ―Mance Rayder
The Cave people are a clan of Free Folk.
The Cave people now march in the army of King-Beyond-the-Wall, Mance Rayder, who has united all of the different wildling clans to assault the Wall and force their passage to the south. Mance points out that it was difficult to unite the numerous different wildling clans into one army, as one clan is always feuding with another clan for one reason or another - Thenns against Hornfoots, Hornfoots against Ice-river clans - though everyone hates the cave people.
In the books
In the A Song of Ice and Fire novels, there is indeed mention of Cave people - though the books call them "Cave dwellers" - as one of the groups of wildlings in Mance Rayder's army. However, they have been barely mentioned at all in the text, except for two or three occasions when lists are given describing the diversity of the wildling clans in Mance's army, i.e. "Thenns, Hornfoots, Ice-river clans, Cave dwellers..." etc.
The most that has ever been said about them is that they make frequent use of face-paints and dyes: "cave-dwellers with their faces dyed blue and purple and green". They are apparently one of the more isolated wildling groups, from the extreme fringes of the lands beyond the Wall, from the caves of the Frostfang Mountains.
Because they've been mentioned so rarely, it is never particularly stated that "everyone hates the Cave people", nor why they would. If anything, the wildling tribe that all of the other wildlings universally hate tends to be the Ice-river clans, as they are openly known to be wild and savage cannibals, who survive in the polar regions northwest of the Frostfangs by literally hunting people from neighboring tribes to eat them. Even so, Mance managed to make peace with the Ice-river clans and convince them to join his army.