My maternal grandmother used to use an expression that was similar. When she wanted to say that something was not as serious as we thought, she would say, “Les Sarrasins sont pas à la porte.” (the Sarasens are not at the door).

The Sarasens were the name given to Muslims in the middle ages. Perhaps the expression came about in the age of sea conquests, pirates and privateers when the sea going ships of one religion would, with the blessing and encouragement of their respective countries, invade the ships and port villages of other religious affiliations. I wonder if the Muslims had an expression that went “the Christians are at the door”?