The son of the Rev. Hon. Richard Byron (1724-1811) and grandson of the fourth baron Byron. He married Margaret Powditch in 1803; their daughter Eliza is not recorded in Burke's Peerage, Baronetage & Knightage.