The Fist of the First Men is a landmark in the wilderness beyond the Wall. It is an ancient ring fort located at the crown of a defensible round hill with an excellent view of the surrounding countryside. Ranging parties from the Night's Watch use it to orient themselves and also as a possible fallback position in case of combat with the Free Folk. It can only be reached by a steep climb on stony ground. It has served as a natural defensive position since the time of the First Men.
The Fist is located many days north of the Wall, deep in territory held by the Wildlings. It is located at the western edge of the Haunted Forest, where the foothills of the Frostfang mountains begin. It commands a view overlooking the Milkwater river.
Physically, the ring fort is a simple circle of large megalithic stones around the top of the hill. The stones are incredibly ancient, the site having been used as a defensive position for well over six thousand years. While the Fist isn't really a castle or even a structure, due to its positioning on a lone hill commanding the surrounding flat plain, it is one of the more defensible positions that the Night's Watch can use north of the Wall.
In the books
In the A Song of Ice and Fire novels, the ring fort was built by the First Men well over six thousand years ago, centuries before the Andals invaded Westeros. It is called "The Fist" of the First Men, because the hill looks like a fist thrust into the sky from the flat landscape surrounding it, and the jagged remaining stones circling the top look like the knuckles of a fist.
In the books, the trees of the Haunted Forest come very close to the base of the Fist. In the TV series there is no forest visible near the Fist. This is probably due to the fact that these scenes were filmed on a glacier in Iceland, and there were no trees nearby.
- Fist of the First Men at the Wiki of Ice and Fire (major spoilers from the books)