It is often called just "Ibben" for short (it is called the "Port of Ibben" in much the same way the phrase "port city of King's Landing" is used).
The port of Ibben is located in the middle of the southern side of Ib island.
In the books
In the A Song of Ice and Fire novels, the port of Ibben is the primary city on Ib island, where most of the population is concentrated. A much smaller second city also exists at the north of the island, Ib Nor.
Since the dawn of Ib's history, the Port of Ibben has been its capital. It is the hub of all the Ibbenese whaling fleets, and thus its main exports are whale blubber, oil, and bone. This industry has made it the largest and richest city in the Shivering Sea (though that isn't saying much, given how few cities are in it at all). Any foreigners who dock at the Port of Ibben are legally obligated to remain within its confines; venturing beyond into the heart of the island is forbidden without express invitation (rarely given) and must be done in the company of an Ibbenese host.
Physically, the Port of Ibben is "grey and gloomy", a city of cobbled alleys, steep hills, and teeming docks and shipyards. Unsurprisingly, it is lit by hundreds of whale-oil lamps suspended over the strees on iron chains. The largest structure in the city is the colossal ruin of the God-Kings' castle, an imposing palace of rough-hewn stone. It was the home to a hundred Ibbenese kings throughout history, but their local line of God-Kings was overthrown in the aftermath of the Doom of Valyria, leaving the palace ruined and abandoned. Afterwards all of Ib, including the city, is ruled by the Shadow Council, a council of magisters similar to those governing several of the Free Cities.
The Ibbenese are only mentioned in passing in the earlier novels, and their homeland was not described in much detail for some time. The references were mostly to the "Port of Ibben" - though in this context it wasn't clear if the text was indicating that "Ibben" is the name of the island, or referring to "the port-city named Ibben" - using the same turn of phrase also used to describe "the port city of King's Landing" / "the port city named King's Landing".
The Lands of Ice and Fire (a companion map collection for the novels, not the TV series, released in 2012) seemed to indicate that the island as a whole is officially named "Ib", and the major port city on it is named "Ibben". The World of Ice and Fire sourcebook (2014) supports this naming pattern, outright titling the sub-chapter devoted to the Ibbenese as "Ib", not "Ibben". The World book also consistently refers to the "Port of Ibben" specifically with regard to their primary city. The world map given to the TV series by Martin starting in Season 2 contained labels for "Port of Ibben", and also "Ibben" for the island, but this was apparently just an incorrect assumption.
This issue was settled when the island was first referred to by name in the TV series continuity, in the Season 6 Histories & Lore video for "The Summer Sea", in which the island is referred to as "Ib", not "Ibben".