Broomfield Lodge was reputedly built as a hunting lodge for James I in the 16th century, but it has been altered and expanded repeatedly over the years. A City merchant, Joseph Jackson, owned the estate by 1624, and the Powys family from 1816 to 1902. Both house and estate were improved and enlarged in the early 18th century, probably including the formal gardens to the west.
The formal garden, ponds and house are enclosed by red brick walls which date from the 16th and 18th centuries. The eastern wall, behind the remains of the house, has an early-18th-century summer house with wooden Ionic columns.
In 1902 the house and 54 acres of its grounds were purchased by Southgate Urban District Council and opened to the public in 1903. The public could swim in one of the lakes until 1911, when the council discovered that the water was polluted. The house, which became a local museum in 1925, was badly damaged by a number of fires in 1984, again in 1993 and then 1994.