OK, here's a couple of examples.

If you live in Manchester then London is to the South. Most people would go "down" to London.

If you live in the South East of England, regardless of whether you live to the North, South, East or West you would go "up" to London.

If you live in Edinburgh which is 50 miles (80km) to the East of Glasgow - how would you describe your journey?