The OED gives /wou/ as the pronunciation. Says it's from who also /wou/ meaning stop. Examples of usage: come hither whoa, gee-whoa, hait whoa, whoa back. I've always heard it pronounced /wou/ to rhyme with woe and oh. And who comes from ho. There's definitely some interesting stuff going on here.

There are two Shakespeare citations in the OED entry, but for a different meaning than stopping a horse, i.e., calling attention from a distance: "Whoa hoe, hoe, Father Paige." Merry Wives Windsor V.v.187. "He hallow'd but euen now, Whoa-ho-hoe." Winter Tale III.iii.79.