The Nightfort is a ruined castle on the Wall, west of Castle Black. It was abandoned because of the high cost of its upkeep. It is reportedly the oldest castle on the Wall, predating Castle Black by centuries, if not millennia. It is not manned by the sworn brothers at the time of the series. It is ill-omened, but some stories say it holds a great secret of the Watch.
According to legend, the Nightfort served as the seat for the Night's King, who had been seduced by a woman from beyond the Wall. He brought her back to the Nightfort and performed dark sacrifices to the White Walkers. However, he was defeated by the King in the North and the King-beyond-the-Wall, Joramun, who joined forces to destroy him. 
Bran Stark, Hodor, Summer, Jojen Reed and Meera Reed arrive at the Nightfort and camp in the chamber containing the well. Bran tells his friends the story of the rat cook, a former cook at the Nightfort who was insulted by a king who stayed at the castle. He killed the king's son and served his flesh to his father inside a pie. The gods, furious at this insult to the ancient laws of hospitality, turned the cook into a white rat cursed to feed on its own offspring forever.
The group then encounter Samwell Tarly and Gilly, who show them how to leave the castle via the tunnel.
In the books
In the A Song of Ice and Fire novels, the Nightfort is the oldest and largest castle on the Wall, and was likely the first major stronghold of the Night's Watch after its founding, which according to myth was almost eight thousand years ago. The Nightfort became an ill-reputed place over the centuries, with dark tales surrounding it (of varying truthfulness). One of the oldest of these is the legend of the Night's King, a Lord Commander in the early years of the Watch who declared himself king of the Wall after falling in love with a white demon woman from the lands beyond. Ultimately the Stark King from Winterfell and a King-Beyond-the-Wall named Joramun united to defeat the Night's King, and his name was cursed and erased from history.
In one of the Nightfort's stories, an Andal King visited the Nightfort. The cook, for some unspecified offense, killed the King's son and served his flesh to his father in a pie. The gods, angry at this violation of the laws of hospitality, transformed the cook into a massive rat that would only feed on its offspring. The tale of the Rat Cook became an infamous myth about a violation of the sacred law of Guest right. In another story, a young girl disguised herself as a boy called Danny Flint to serve on the Wall, but was discovered, raped and murdered by persons unknown. In other stories, seventy-nine sentinels stood guard at the Nightfort, and it is where King Sherrit called down a curse on the Andals. It is where a ranger called "Mad Axe" went insane and killed several of his sworn brothers. Symeon Star-Eyes saw hellhounds fight at the Nightfort in the Age of Heroes. Finally, three centuries before the Targaryen invasion, the commanders of the garrisons at the Nightfort and Snowgate went to war against one another, but they were both executed by the Starks for betraying their vows.
The Nightfort remained occupied up until two centuries ago. By this time the Nightfort had become mostly unoccupied and fallen into disrepair, with most of the administrative functions of the Watch transferred to the more centrally-located Castle Black to its east. When Queen Alysanne Targaryen visited the Wall on her dragon, Silverwing, she convinced the Watch that the Nightfort had to be abandoned. She convinced her husband, King Jaehaerys I, to fund a new, smaller castle to be built at Deep Lake a few miles to the east, and the Nightfort was abandoned. By the time of the books, Deep Lake itself has also been abandoned.
In the Nightfort there is a hidden gate, known as the Black Gate, that allows passage to the other side of the Wall. It is set deep in a wall of the well at the center of the kitchens and is made of white weirwood with an old wrinkled face on it. Only a brother of the Night's Watch can open the Black Gate, by reciting the vow of the Watch.
There are actually two other abandoned castles between the Nightfort and Castle Black. From west to east, they are Deep Lake and Queensgate. Deep Lake was actually built as a smaller replacement to guard the section of the Wall near the Nightfort, until it was also abandoned. In the TV series, of course, Samwell does not explicitly state that the Nightfort is the castle immediately west of Castle Black, he only says that it is some distance west of it (without mentioning whether or not there are intervening castles).
- Nightfort at A Wiki of Ice and Fire (MAJOR spoilers from the books)