The Barony of Howard de Walden was created by writ of summons in the Peerage of England, by Queen Elizabeth I for Admiral Lord Thomas Howard, a younger son of the 4th Duke of Norfolk, in 1597. The title was reportedly granted for the Admiral's role in the defeat of the Spanish Armada in 1588.

The baron eventually went on to obtain the title of Earl of Suffolk from Elizabeth I's successor King James I, which latter title continues in his male-line descendants. The barony Howard de Walden however eventually passed out of the Howard family with the death of James Howard, 3rd Earl of Suffolk in 1688, and came briefly to the 4th Earl of Bristol before passing to his great-grandson, the four-year old Charles Augustus Ellis in 1803.

The title actually fell into abeyance between 1688 and 1784 between the heirs of the 3rd Earl's two daughters _ Lady Essex Howard (a daughter by his first marriage) and Lady Elizabeth Howard (a daughter by his second marriage). Lady Essex Howard married Edward Griffin, 1st Baron Griffin, and had descendants. Her granddaughter Hon. Ann Griffin was the only surviving daughter of the 2nd Baron Griffin, and her son John Griffin Whitwell, later Field Marshal John Griffin Griffin (when his surname was changed) inherited one-half or a moiety of the barony when his maternal uncle the 3rd Baron Griffin died without legitimate issue. In 1784, the barony was called out of abeyance in his favor. In 1788, the new 4th Baron Howard de Walden was also created 1st Baron Braybrooke with special remainder to a kinsman Richard Aldsworth Neville who had married a daughter of the powerful Grenville family (and thus a cousin of William Pitt the Younger).

At the death of Lord Howard de Walden & Braybrooke in 1797 without any issue, the barony Howard de Walden passed to the only other heir - the representative of the 3rd Earl of Suffolk's younger daughter Lady Elizabeth Felton. Her daughter Elizabeth Felton had married as his 2nd wife John Hervey, 1st Earl of Bristol, and their grandson was Frederick Augustus Hervey, 4th Earl of Bristol ((1730-1803). He had inherited the earldom only upon the death of two older brothers. At his death, his second but eldest surviving son inherited the earldom as 5th Earl, and eventually became the 1st Marquess of Bristol. However, the Barony Howard de Walden passed out of the Hervey family once again, via two deceased heirs - John, Lord Hervey (eldest son of the 4th Earl) and his daughter Hon. Elizabeth Catherine Caroline Hervey (later Hon. Mrs Charles Rose Ellis) - to a great-grandson Charles Augustus Ellis, 6th Baron Howard de Walden who was also 2nd Baron Seaford by right of his father's title (granted 1826).

The young Lord Howard de Walden (1799-1868), aged four when he inherited the title from his great-grandfather, eventually married Lady Lucy Cavendish-Scott-Bentinck, youngest daughter of the 4th Duke of Portland and sister and co-heiress of the reclusive 5th Duke of Portland. It was through her that the Barons Howard de Walden inherited substantial properties in London, and adopted the name of Scott-Ellis instead of Ellis. The 6th Baron was succeeded by his son the 7th Baron (1830-1899) and the 8th Baron (1880_1946). The great-grandson was John Scott-Ellis, 9th Baron Howard de Walden and 5th Baron Seaford(1912-1999)).

In 1999,while a cousin succeeded as Baron Seaford, the barony of Howard de Walden fell into abeyance among the 9th Baron's four daughters and co_heiresses, who also inherited jointly substantial estates in London. By Royal Warrant dated June 25, 2004, The Queen called the Barony of Howard de Walden out of abeyance in favor of the eldest daughter, Mary Hazel Caridwen Czernin (born 1935).

The property associated with the barony - Audley End at Saffron Walden - which was obtained by Thomas Howard, 1st Earl of Suffolk from his maternal grandfather Thomas Audley, 1st Baron Audley was separated from the title, when it was left by the 4th Baron Howard de Walden & 1st Baron Braybrooke to his relative Richard Aldworth Neville, 2nd Baron Braybrooke. The house Audley End was actually built by the 1st Earl of Suffolk.

Barons Howard de Walden (1597)

• Thomas Howard, 1st Earl of Suffolk, 1st Baron Howard de Walden (1541-1626)
• Theophilus Howard, 2nd Earl of Suffolk, 2nd Baron Howard de Walden (1584-1640)
• James Howard, 3rd Earl of Suffolk, 3rd Baron Howard de Walden (1619-1689) (abeyant 1689)
• John Griffin Griffin, 4th Baron Howard de Walden (1719-1797) (abeyance terminated 1797; abeyant 1797)
• Frederick Augustus Hervey, 4th Earl of Bristol, 5th Baron Howard de Walden (1730-1803) (became sole heir in 1799)
• Charles Augustus Ellis, 6th Baron Howard de Walden (1799-1868)
• Frederick George Ellis, 7th Baron Howard de Walden (1830-1899)
• Thomas Evelyn Scott-Ellis, 8th Baron Howard de Walden (1880-1946)
• John Osmael Scott-Ellis, 9th Baron Howard de Walden (1912-1999) (abeyant 1999)
• Mary Hazel Czernin, 10th Baroness Howard de Walden (b. 1935) (abeyance terminated 2004)