Whether you pronounce /e/ or /ɛ/ in agression, terrorisme and many other words doesn’t make that much a difference. There are regional or just personal variations so native speakers might use one or the other.

For example, here is a map showing if raisonner is more pronounced /re…/ (dark blue) or /rɛ…/ (light blue) depending on the département of origin:

In any case, even if you use a vowel that is seldom used for a given word, for example /uver/ instead of the mainstream /uvɛr/ for ouvert (open), everyone will understand you.

Source: https://cartopho.limsi.fr/

I think I can say that your “Robert” has a mistake, because I own the “Grand Robert”, edition of 1985, and in my Robert, “terrorisme” is given with [ɛ]. And in fact, before a double consonant, the letter “e” has to be pronounced that way (“terre”, for example). The fact of pronouncing it [e] is a often a childish mistake and not an academic possibility.