OED2 supports the buffet interpretation, giving bluff as an *early spelling variant:

bllind-man's-buff Also 6-7 blindman-buff(e, (-buffet, -bough, -bluff,) 7-9 blindman's-buff. [f.
BLIND-MAN + BUFF = buffet, blow, stroke.]

1. A game in which one player is blindfolded, and tries to catch and identify any one of the others, who, on their part, push him about, and make sport with him.

1600 ROWLANDS Let. Humours Blood iv. 64 At hot-cockles, leape-frogge, or blindman-buffe. 1628 GAULE Pract. The. (1629) 231 Others make him [Christ] no better then their Pastime, at no more discreet a Sport then Childs, or Fooles Blind-man-Buffet: Prophecie vnto us, who is he that smote thee? 1634 J. TAYLOR (Water-P.) Gt. Eater Kent, Gregorie Dawson, an English-man, devised the unmatchable mystery of Blind-man-buffe. 1696 Month. Mercury VII. 55 They oblig'd him to play with 'em at Blindman-Buff. 1766 GOLDSM. Vic. W. xi, Mr. Burchell..set the boys and girls to blindman's buff. 1866 R. CHAMBERS Ess. Ser. I. 186 The whole parlour put into disorder by blind man's buff.

2. fig.

1590 Three Lords Lond., Ile to my stall; Love, Lucre, Conscience, blindman buffe to you all. 1643 BRAMHALL Serpent Salve §1 We desire to see what they have done, before we go to blindman's buffet one with another. 1648 C. WALKER Hist. Indep. I. 55 Me thinks..we are compelled to play at blind-man-bough for our lives. 1837 CARLYLE Fr. Rev. I. VI. iii. 278 Government by Blind-man's-buff.