The expanding Portuguese colonial forces had their sights set on the coastal Moroccan city of Tangier throughout the fifteenth century until they finally succeeded in taking it in 1471. It remained in European hands throughout the following centuries, being passed to Spain in the seveteenth century as part of the union of the crowns between the two countries, and then in 1661 it was given to King Charles II as part of the dowry given to him with the Portuguese infanta Catherine of Braganza. The King immediately sent embassies to assess his new land and by January 1662 English Tangier was fully occupied, with Charles declaring it "a jewell of immense value in the royal diadem" even though the departing Portuguese had done their very best to strip the city of everything valuable. A garrison was established in Tangier, which also received a charter making it equal to other English towns.

Nonetheless the regiment stationed there was constantly under attack by local mujahideen offended by the Christian presence in their country. An uneasy truce was struck between 1663 and 1664, but was called of when 470 English soldiers were killed in an ambush. A longer-lasting peace treaty was made in 1666, and the English took advantage of the respite to greatly improve the defences first established by the Portuguese. They also planned to build a mole in the harbour much like that in Gibraltar, however incompetence, waste and outright fraud caused costs to swell. Although it eventually extended almost half a kilometre through the waters, the mole was destroyed and the £340,000 spent on its construction was wasted.

On one of the missions sent to assess and plan for the development of English Tangier was Wenceslaus Hollar, who travelled to Morocco in 1668. During and after his visit he produced 15 views of the city from different perspectives. Among these, many focus on Tangier Castle, the name given to the Portuguese fortifications, atop which the English flag is flown from 'Peterborough Tower', another familiar name the colonists had given to the buildings they inherited in Tangier.