According to AHD4 haberdasher is perhaps from the Anglo-Norman hapertas, 'petty wares'.