I found a site with evidence that Coleridge got his idea from a previous publication. I think this really shoots the "sex fantasy" idea squarely in the ass.

"'In Xamdu did Cublai Can build a stately Palace, encompassing sixteene miles of plaine ground with a wall, wherein are fertile Meddowes, pleasant Springs, delightfull Streames, and all sorts of beasts of chase and game, and in the middest thereof a sumptuous house of pleasure.'-- Purchas his Pilgrimage: Lond. fol. 1626, Bk. IV, chap. XIII, p. 418." A few years later Coleridge made two memoranda quoting Purchas: "Cublai Chan began to reign 1256, the greatest Prince in Peoples, Cities, & Kingdoms that ever was in the World" Notebooks, I, 1840. "Kublaikhan ordered letters to be invented for his people" Ibid., 1281.