I'm sure the more gramatical members will give you the exact rule, but the way I see it is this.

If you use the word "which" you are asking a person to choose from a set number of things, to choose from a specific group. So the question must be followed by the list to choose from.

i.e. Which format should I put this file in? jpg, gif or tif?


If you use "what" you are not providing a list of choices. You don't know what the choices are.

i.e.

"I scanned a picture of my puppy, and I want to save it in my computer." "What format should the file be in."