To me let's go to lunch feels more formal and implies that the food is prepared and waiting. For lunch is looser, merely meaning that it is the middle of the day and I want food whether I plan to buy it, make it or open my lunch box.