Same rule, the event is real so there is no reason to use the subjunctive.

I would use the present and passé composé. You saw him once but you might see him again in the future:

C’est la seule fois que je l’ai vu.

Or, if you really want to use the imperfect which leaves the possibility that you saw him later:

C’était la seule fois où je l’avais vu.

or, using the passé simple. You saw him once and you’ll never see him again. This is written/formal French:

Ce fut la seule fois où je le vis.