Speaking from a Scottish perspective, I agree entirely. I have never done anything except travel down to London and I do not believe any of my acquaintances are any different.
Geographically, many people do go down to London. 'Up' is perfectly fine for the clue, though, as Chambers defines 'up' as 'towards a centre (such as a capital, great town, or university)'.