And why do I have no problem using "without" in Jackie's example if I leave out the "you"?
Because "without" is understood to refer to "you" when they are together. Without as a place is old and seldom used but I do remember reading about something or someone being "without the house" in a very old novel. ?Jane Austin? or some such.