The need for a reliable harbour to service the frontier region of the Eastern Cape began to surface during the 1830s, when the British military found that the lines of supply and communication between Port Elizabeth and the Albany settlements had become dangerously over-extended. Although efforts had already begun in 1825 to develop a harbour at the Kowie river mouth, this was found to be too shallow and liable to silting, and the attention of the military turned further east. The need became more pressing after 16 June 1835 when Governor Sir Benjamin D'Urban proclaimed a broad belt of land between the Keiskamma and Buffalo rivers as the Province of Queen Adelaide. In November 1836 a survey was made by John Rex and John Bailey of the Buffalo river mouth. Their report was positive and the area was immediately named Port Rex. However, the new province was never annexed to the Cape and plans regarding the new harbour had to be abandoned.
During the frontier war of 1846-47, better known as the War of the Axe, the British made use of Waterloo Bay, located near the Fish river mouth, but found this unsatisfactory and the military authorities ordered a second survey of the Buffalo river mouth. Again, the report was favourable and this time plans for its use were implemented. In 1847 a post, known as Fort Buffalo, was built on the west bank of the river. On 14 January 1848 the new Governor, Sir Harry Smith, formally annexed the port and its surrounding territory to a radius of about 3km to the Cape Colony and named it East London. At about the same time Smith decided to establish a second and more substantial fort only a short distance above Fort Buffalo. The structure, designed by Capt J Walpole, Officer Commanding the Royal Engineers, was built in 1848 under the supervision of Lt Ferrois, and was named Fort Glamorgan in honour of Col Henry Somerset, eldest son of Lord Charles Somerset and Commander of British troops on the eastern frontier in 1819-52.
A stone jetty was also built in 1848, and by the end of 1849 at least four streets had been laid out. By the mid-1850s the village had a population of 124 Europeans settlers and 300 troops. In 1857 members of the British- German Legion, who had previously fought in the Crimean War, were recruited for service as soldier-settlers in South Africa. They were settled in British Kaffraria, but a number of them eventually took up residence in East London, building their homes on the eastern bank of the Buffalo river. By 1873 three villages were clustered about the mouth of the Buffalo: East London West Bank; East London East Bank, and Panmure, named after Lord Panmure, British Secretary for War in the 1850s. On 22 April 1873 the three were merged into a single municipality. Construction on the main harbour began in 1872, and in 1873 work began on the breakwater.
The 1875 census indicated that East London had a population of 2,134. In 1891 this number had risen to 6,924, and by 1904 it stood at 25,220, of whom 14,860 were literate.