We agree not to refer to OR We agree to not refer to.

[…] the difference in meaning


For me, they are different because there is a sort of ghost-parsing or an echo of archaic usage in the first example: it could break down into
> We [agree not] [to refer] [to x]
which would render the meaning as “we don’t agree to refer to x”, in contrast to the clear and certain meaning of the second example (“we {positively} agree to not refer to x”)

English word order tends to prioritise important information to the head of the sentence. When you encounter a negative word in third place, the brain is trying to apply it to the preceding statement, hence leading to ambiguity.