I would say that le simple fait de + infinitive = Even just + -ing:

Le simple fait de marcher me fait mal au genou.

Even just walking hurts my knee.

So le simple fait de l’envisager means even just thinking about it.

To insist on “simple”, one can use sheer, as a direct translation:

The sheer fact that

and turning it differently, by converting the initial verb “envisager” into a substantive:

The sheer thought of it