The expression is sounding right enough to me, but yes the confusion is logically possible. If the context makes it unclear, or if one of the interlocutors is learning french, or anything alike, just go for the long version, saying out loud what you put between parenthesis, it’s slightly heavier but correct also and clearer.

On a assisté à la parade, ma nièce et moi, il y a cinq ans, mais elle était trop jeune à l’époque pour s’en souvenir aujourd’hui.

The phrase makes sense. “pour s’en souvenir” is a complément circonstanciel de conséquence i.e it expresses the consequences of “elle était trop jeune”.
There are nothing requiring the consequences to be at the same moment.

Also the phrase is not ambiguous:

• “elle était trop jeune” can only mean “elle était trop jeune il y a 5 ans”, this is by the construction of the phrase.
• and “pour s’en souvenir” cannot be “il y a 5 ans” as it would not make any sense (that would mean she was remembering what she was living at the same time). Without any other time elements, it can only be understood as “pour s’en souvenir aujourd’hui”.