The use of, "annoys me to no end" , irritates me limitlessly, deanjens (by way of a welcome )
If your annoyance is without destination or an object that you hope to achieve, then why allow it to annoy you?

Fairly obviously, you cannot mean that your annoyance is going to "no end", because that is clearly impossible - you can, I suppose, travel toward "no end" but you can never get to it.