Your original surmise may well be correct. There was no such sound in Norman French.

Modern French uses a similar sound, the uvular fricative [X], in final position after a voiceless consonant: e^tre = [EtX], Ypres = [ipX]. This is a positional devoicing of the uvular fricative or roll that is the normal value of French r. That pronunciation replaced the normal rolled [r] starting in, I believe, vaguely, Paris in the late Middle Ages (and spread through France, then into Germany and Denmark).

There was no such sound in Latin, nor in any of its descendants that I can think of, at any time. Several of them were influenced by Germanic invasions: for example, the Franks took over Roman Gaul. But the Frankish language was lost and they adopted the local Gaulish Latin, the precursor of Old French, leaving only a few Germanic words in it as borrowings.

Frankish would certainly have had [h], and presumably also had [x], in the same positions as in OE and German. If I had books by me I might be able to find an example, and then we'd know what had happened to them. But Germanic [x], if it was there, did not survive to the OF (Norman) period. Clearly [h] did, since English borrowings from French pronounce the [h] where it is silent in Modern French.

The [x] was lost in Middle English; perhaps around 1400 in London speech, later in the country. (I'm being very vague here.) You could tell by looking at poetry: at what point did light begin to rhyme as [li:t] rather than [lixt]?

Of course the change never spread fully to Scotland.

[x] has reappeared in contemporary London speech, replacing [k]: so come is [xam], look is [lux], and so on. In many speakers the change is consistent.