Le sens n’est pas vraiment le même.

Il y avait de rares passants = Il y a avait quelques passants, pas beaucoup.

Il y avait eu de rares passants = Mais au moment où se passe l’histoire, la situation a changé (il n’y a plus personne ou alors les rues sont bondées).

Donc pour répondre à votre question, il faut utiliser “Il y avait de rares passants”.

One first possible source of misunderstanding could be interpretation of "two different points in time" (what you call t₀ and t_-1). If you really have a story in a story, in a different timeline, this is not a reason to suddenly switch to a different past tense. Compound past tenses are only used to indicate that something already happened at the current point in a timeline (which could be present or past).

I will now assume that we are dealing with a past timeline and literary French offers you in this case 4 different indicative past forms:

• il y avait (context)
• il y eut (event)
• il y avait eu (context, already happened)
• il y eut eu (event, already happened)

Choosing between the first two or the last two depends on whether what you want to express already happened at the current (past) narration point, or if it’s happening/relevant at that point.

Choosing between the imperfect (avait), or the simple (eut) form depends on the role of this proposition or sentence in the narration. The imperfect form introduces contexts while the simple form introduces additional “events”/“records” in the narration. The use of y avoir as an event is possible but very uncommon. It’s use is generally descriptive.

Here is an example that illustrate the use of the plus-que-parfait (avait eu):

Elle se réjouissait de l’effervescence soudaine de la rue. La veille il n’y avait eu que de rares passants.

This sentence only introduces contexts to the narration, one which is current (elle se réjouissait) and one which was only relevant the day before (il n’y avait eu). The use of the imperfect (resp. plus-que-parfait) is typical to descriptions. It doesn’t add any “events” to the storyline.

It’s maybe not relevant to your question, but in the unlikely case of an event, to complexify matters, passé simple (eut) and its composed form passé antérieur (eut eu) are never used in contemporary spoken French. The “eut” form is generally substituted by “a eu”.

Il y eut des accidents. (literary)
Il y a eu des accidents. (casual)

Quand il y eut eu une explication, les esprits se calmèrent. (literary)
Quand il y a eu une explication, les esprits se sont calmés. (casual)

I hope now that you will be able to chose from the different tenses, but from your description it’s hard to tell which is the one suited to your situation.