My experience of 'gone off' has always been food related, though '(somebody) having an off day' is used quite a lot as well. Another usage for 'gone off' that I've heard used quite a lot is when someone gets annoyed, bored or put off with someone/thing. E.g: 'He's so annoying, I've gone right off him' or 'Yeah, I've gone off milk ever since I drank some that was off.' ()