Lillywhite was born in Hove, Sussex on 7 July 1829.
He was the third son of (Frederick) William Lillywhite (1792-1854), the Sussex bowler known as the "Nonpareil" ("unrivalled"), younger brother of the batsman and umpire John Lillywhite (1826-74), and cousin of JamesLillywhite (1842-1929) who captained England in the first Test match played against Australia in Melbourne in 1877.
The Lillywhites' father died in 1854 and was buried in Highgate Cemetery, North London.
Frederick William Lillywhite (born 13 June 1792 in Westhampnett, Sussex; died 21 August 1854 in Islington, London) was a famous English cricketer during the game's roundarm era.
Lillywhite, who played for Sussex county cricket teams and latterly for Sussex CCC, was one of the greatest bowlers in cricket history and his status is borne out by his nickname: The Nonpareil.
He was the father of John Lillywhite and Fred Lillywhite and uncle of JamesLillywhite.