The English translations of:

Je n’y suis presque plus allé.

sounds odd to me. This sentence means neither “I hardly went there” (Je n’y suis presque pas allé) nor “I seldom went there” (J’y suis rarement allé) but:

“I almost stopped going there” (literally “I went almost no more there”).

You suggest the imparfait should be used because the action wasn’t instantaneous. It was actually kind of instantaneous. The statement relates a final decision or fact true from a single point of time.

With the imparfait:

Je n’y allais presque plus.

a follow-up is expected and it might even invalidate the statement. There is no guarantee he didn’t change his mind later.

A third possibility for a past event verb is the passé simple.

Je n’y allai presque plus.

The meaning is fine, very close to the passé composé. One obvious issue is that this first person is homophone with the imparfait so introduces some ambiguity.