I've never heard 'going off' to mean getting angry. Yes, it's quite common to talk about milk and a few other things (mostly butter, cheese and other dairy products) as having 'gone off'. General British Isles English-speak.