I agree with you, but unfortunately, most people who use "I could care less" are thinking "I couldn't..." Think about someone who just slammed you: "I couldn't care less about what you say" could be said coldly, calmly, or in anger.

In a different situation, using the ironical "I could care less" and intending it as irony, would only say it calmly - not in an angry demeanor - so as to give it the colder feeling intended = perhaps even sneer or smirk a little while saying it, thereby possibly angering the recipient - a manipulative play.