Well, talking for the civilized portion of English speaking world, we here in the South had trouble with "talking to". We kept getting it mixed up with "talking too" as in "much", which is understandable.

Being a most considerate subset of English speakers we solved the problem by using the idiom "talking at". In example...

Jerrie Lou was talking at Sammy.

See, now it becomes crystal clear to whom Jerry Lou was talking.

If we had used the phrase "Jerrie Lou was talking with Sammy" we would have known that Jerrie Lou, while she was with Sammy, was talking; but properly speaking, we would not have known to whom she spoke.

English is a right fine language; it just takes a little adjusting to make it work.