However when I go back to the town where I grew up I hear a very strong and very different accent from the Raleigh/Durham accent that I'm used to

Yes, xara, this is also my experience over in UK. Agreed that the differences in pronunciation in UK are much wider and over much smaller distances - but despite that, there is a noticeable difference between both pronunciation and usages betweeen villages and towns only six or eight miles apart. A Galgate, Lancashire, person will immediately spot a person from Garstang, Lancashire, by his or her accent. It is even apparent to observant incomers to the area (like myself!)
I first noticed the phenomenon in Northampton shire, when I travelled all over that small county - it is c40 miles long and perhaps twenty-five wide - and there are very distinct differences between Daventry, at the West end and Irthlingboro at the East, with a dozen variations in between. Even the main river, the Nene. changes its name from "Nen" West of Oundle, to "Neen" East of that town.

It is a fascinating study, and it makes you realise that Prof. Henry Higgins was not making an idle boast when he said he could spot whch street you were born in!