Walk west along Calle Norzagaray in the late afternoon and the whole logic of Old San Juan lays itself out in front of you. On your right, the Atlantic, grey-blue and restless. Ahead, a green apron of lawn rising to a low grey mass of masonry that seems to grow out of the headland rather than sit on it. Behind you, seven blocks of pastel facades, wrought-iron balconies and streets paved in dark blue-grey stone that turns treacherous in the rain.

This is the point of the place. San Juan was not built to be charming. It was built because whoever held this islet held the eastern gateway to the Caribbean, the first deep, defensible harbour that a ship crossing from Europe could reach, and therefore the hinge of an empire. Everything the visitor now finds beautiful about the old city, the walls, the sentry boxes, the narrow shaded streets, was originally a military calculation.

The City That Moved

The story does not begin on the islet. In 1508 Juan Ponce de Leon established a settlement called Caparra on the almost landlocked bay to the west, and it was, by every contemporary account, a mistake: unhealthy, badly ventilated, and inconveniently far from the dock where the ships from Spain actually arrived. The residents complained until they got their way. In 1521 the settlement was moved to the rocky islet at the harbour mouth, and the city we now call Old San Juan was born.

Then came one of the great administrative accidents in Caribbean history. The island had been named San Juan Bautista and the harbour was called Puerto Rico, the rich port. Within a year or two the two names had swapped places, apparently through simple habit, and they never swapped back. The island became Puerto Rico and the city became San Juan. Almost every visitor arrives without knowing that the names are the wrong way round.

You will read in a great many places that this makes San Juan the oldest city in the United States. That needs a qualifier to be true. San Juan, founded 1521, is the oldest European-founded city in United States territory, and the second oldest European-founded capital in the Americas after Santo Domingo. Saint Augustine in Florida, usually given the title, was founded in 1565, forty-four years later, and is the oldest continuously inhabited European settlement in the continental United States. Both claims are correct; the difference is the word continental, and it is usually left out.

A related legend concerns the founder. Ponce de Leon is remembered for hunting the fountain of youth, and it is the one thing most people can tell you about him. He never mentioned it. The story attaches to his name only in chronicles written after his death, and historians treat it as a literary embellishment rather than a motive. He died in 1521, the year the city moved, and although Casa Blanca on the western edge of the old town was built for his family and stayed in their hands into the eighteenth century, he almost certainly never lived in it.

The Fort That Never Fell and the City That Did

Defences came in layers. La Fortaleza, begun in the 1530s, was the first real fortification, and it was badly sited: it guards the inner bay rather than the harbour entrance. The Spanish recognised the error and in 1539 began work at the tip of the headland on Castillo San Felipe del Morro. El Morro was not finished for two hundred and fifty years, reaching its final form around 1790, by which point a promontory with a cannon on it had become a six-level fortress designed to demoralise a ship's captain before he was in range.

Here is where the tourist narrative needs correcting. El Morro is routinely described as impregnable, the fortress that never fell, and as far as the castle itself goes that is broadly true. The city behind it is another matter, and it fell twice.

In 1595 the English privateer Francis Drake attacked through the harbour entrance and was beaten off. Three years later the Earl of Cumberland learned from Drake's mistake, landed to the east and took San Juan from behind by land. He held it, and then lost it to dysentery, abandoning the prize because his men were dying in it. In 1625 a Dutch force under Boudewijn Hendricksz seized the town and La Fortaleza itself, but could not crack El Morro; when the Dutch withdrew they burned what they could, including much of the governor's palace. Twice the flag came down over the city. Twice the castle held, and the castle is what got remembered.

There is a third siege that produced the city's favourite legend. When the British returned in 1797 under Ralph Abercromby, the story goes, the bishop led a night-time rogativa, a procession of women, the elderly and children carrying torches through the streets, praying for deliverance; the British, seeing long files of lights moving across the city, concluded that reinforcements were arriving from the interior and sailed away. It is a wonderful story and it is told all over Old San Juan. Treat it as tradition rather than as the documented cause of a British withdrawal.

The Blue Cobblestones Are Not Spanish

Now for the detail every guide gets wrong, and it is worth spelling out because it is repeated in travel writing to this day. The famous blue-grey paving of Old San Juan, the adoquines, is almost always explained as ballast: iron slag carried in the holds of Spanish galleons to steady them on the outward voyage, unloaded at San Juan before the ships were filled with the wealth of the New World.

Archival research in the municipal records tells a different and much less romantic story. The slag blocks were an industrial product, ordered commercially from England in the last decade of the nineteenth century. The paperwork survives: motions in the city council, correspondence with export agents in Manchester and Liverpool, shipments dispatched from Middlesbrough on the northeast English coast and invoiced to the ayuntamiento. One consignment arrived in San Juan by steamer in February 1894. The first street laid with them was Calle de la Tanca. Before that the streets were surfaced with siliceous stones known locally as chinos.

So the stones underfoot are not colonial, not Spanish, and not ballast. They are late Victorian industrial waste bought from a catalogue, which does not make them any less beautiful when the light goes gold at six o'clock, but does mean that the sentence in your guidebook is wrong.

What to See

Give yourself two days if you can. The two forts, El Morro and Castillo San Cristobal at the eastern end, are administered together as the San Juan National Historic Site and a single ticket covers both, so do not pay twice; check current hours and fees before you go. San Cristobal is the larger and the less visited, a maze of tunnels and outworks built to defend the landward approach, which is to say the direction Cumberland came from.

La Fortaleza, still the governor's residence, is the oldest executive mansion in continuous use in the Americas, and it is approached along the Paseo de la Princesa, a promenade under the south wall that ends at a fountain and gives the best view of the ramparts from below. Casa Blanca is now a museum. The Cathedral of San Juan Bautista, begun in 1521 and rebuilt more than once, holds the tomb of Ponce de Leon. San Jose Church, begun in 1532 and among the oldest churches in continuous use in the hemisphere, reopened not long ago after roughly two decades of restoration.

Two lesser-known stops repay the walk. The Santa Maria Magdalena de Pazzis cemetery sits outside the walls directly below El Morro, its white marble crowded between the fortress and the sea, and it is the most photographed and least understood corner of the city. And the walls themselves can be followed for long stretches; the Puerta de San Juan, the surviving sea gate, is where dignitaries once stepped ashore and walked uphill to the cathedral to give thanks.

Practical Notes

The old city is compact and entirely walkable, but it is built on a hill and the adoquines are slick after rain, so bring shoes with grip. Mornings are cooler and quieter; cruise ships disgorge into Calle Fortaleza in the middle of the day. La Perla, the brightly painted neighbourhood tucked between the north wall and the sea, became globally famous through a music video, and it is worth remembering that it is somebody's home rather than an attraction. Hurricane season runs roughly June to November.

The old town became part of a UNESCO World Heritage listing in 1983, alongside La Fortaleza and the forts. El Morro itself spent the first half of the twentieth century as a United States Army installation, complete with a concrete observation post added during the Second World War, before passing to the National Park Service in the early 1960s. Puerto Rico had come under United States sovereignty in 1898, the year an American squadron shelled the city in May and Spain ceded the island by treaty in December.

Which is the last thing worth carrying up onto the ramparts at sunset. Five hundred years of flags have gone up and come down over this islet, and the walls were built by people who assumed the fighting would never stop. They were nearly right. What survives is a fortress that outlived its purpose and became, of all things, a place to watch the light change over the sea.