The orthography reflects the pronunciation.

One of the rule of French orthography is that if two sounds that would be written i occur next to each other, they’re not written with ii, but rather with y. This can be either the vowel /i/ followed by the glide /j/ (as in fuyez); or a digraph ending in i (oi, ai, ei) followed by the glide /j/ (as in foyer) or by the vowel /i/ (as in pays).

Ennuyer is pronounced (in my dialect, but the principle applies to Northern French French as well, just with different values for the vowels) /ɒ̃.nwi.je/. Notice how the letter y stands for both the vowel /i/ and the glide /j/?

Ennuie however is pronounced /ɒ̃.nwi:/. Here, there’s only the vowel /i/ (elongated by the “mute” e, in Belgian, Swiss and eastern French French), without the glide, so the letter i is used.

This is also the case with, say, choyer (/ʃwa.je/) where two orthographic i would stand next to each other: the one of the digraph oi (for the sound /wa/) and the one for the glide /j/ . They fuse into y (i.e choi + ier = choyer rather than choiier). Choie is pronounced /ʃwa:/, so there’s only one orthographic i, and no fusion into y.

With verbs in -eyer, the infinitive and the indicative present both have a digraph ending in i (ei pronounced /ɛ/) followed by /j/, so they’re both written with the letter y. For example, grasseyer (/ɡʁa.sɛ.je/) et grasseye (/ɡʁa.sɛj/).

With verbs in -ayer, two pronunciations are accepted and thus two orthographies exist. The verb payer (pai + ier = /pɛ.je/) in the indicative present can be pronounced either /pe:/ and written paie or pronounced /pɛj/ and written paye. (In Parisian French this would be /pe/ and /pej/, with the same effects on orthography)