You just had to push it, didn't you.

In the situation given, the non-striker would not be out as the ball was not touched by a member of the fielding side. Even if it was, I believe the earlier of the two would take precedence. There are indeed a number of hypothetical ways is which more than one batsman could be dismissed with one delivery. However, most of these relate to "ungentlemanly" dismissals - "Timed Out", "Obstructing the Field", "Hit the Ball Twice" and "Handled Ball", only the last of which have I seen.