Kitty corner, catty corner, catta corner. And yes, I would use it (almost) solely for the description of a physical relationship based on location at a street intersection. I have heard it used in a context in which I would have used diagonal or cattawampus, to describe a building that was not square to the nearest road, but would not consider such usage to be at all common.