If you want people to eat with you, whether you all have your food or are going out somewhere, I would just say "Let's go eat lunch", and not worry about the "to" or "for" part. If you are definitely going out, you could use "Let's go get lunch", and if the food is already present and accounted for, "Let's eat!" works just fine. :0)