In reply to:

The expression implies, to me, that you can't do whatever it is well enough even to be given a piece of toffee for it.

That's a plausible explanation, dxb.