But my guess is that it changed to controller because deleting the p (and changing the m to an n) made it much easier to pronounce.

Actually comptroller used to be pronounced the same as controller, but its spelling started to influence the way people pronounced it. (Cf. perfect which came into Middle English from French, and was pronounced as spelled, parfit; the later etymological spelling soon changed how people pronounced it, too.) The reasoning behind the -mpt- spelling was the thought (in the minds of those who knew little Latin and less Greek) that the word controller had something to do with count (var. compt < Latin comptus < computo 'to calculate'). Control, OTOH, came from Medieval Latin contrarotulo 'to check by duplicate regist' < contra 'against' and rotulus 'roll, register' (< rota 'wheel').