Physically, the ring fort is a simple circle of large megalithic stones around the top of the hill. The stones are incredibly ancient, the site having been used as a defensive position for well over six thousand years. While the Fist isn't really a castle or even a structure, due to its positioning on a lone hill commanding the surrounding flat plain, it is one of the more defensible positions that the Night's Watch can use north of the Wall.
In the books
In the A Song of Ice and Fire novels, the ring fort was built by the First Men well over six thousand years ago, centuries before the Andals invaded Westeros. It is called "The Fist" of the First Men, because the hill looks like a fist thrust into the sky from the flat landscape surrounding it, and the jagged remaining stones circling the top look like the knuckles of a fist.
In the books, the trees of the Haunted Forest come very close to the base of the Fist. In the TV series there is no forest visible near the Fist. This is probably due to the fact that these scenes were filmed on a glacier in Iceland, and there were no trees nearby.