Derby, per AHD3, after Edward Stanley, 12th Earl of Derby. Which only gets us one step closer to the answer.

OED puts it 23 years later than Bowler, 1883 and 1861 respectively. But it calls bowler colloquial, which it does not say of derby.