It seems that numbers are more difficult to learn...does anyone share this experience?

As it happens, my personal experiece is the exact opposite of your problem - I find numbers easy to learn, remember and use. I find it very easy to think of numbers, and visualise them, in each of the four languges in which I have learned to count. Of those four languages, I can converse in only one, and have varying degrees of aural and written comprehension in the other three. Yet counting, and thinking numerically, is easy in all of them.