You can use:

‘Il n’y a aucune raison que je me pollue l’esprit […]’ but also,

‘Il n’y a aucun intérêt à me polluer l’esprit […]’

It means that, this is not worth for you to take in consideration all of these questions.

“Il n’y a pas matière à ce que je…” is not right.

• “Il n’y a pas matière à” must be followed by a positive idea.

What you mean is this;

• La matière de cela n’est pas assez substantielle pour que je me pollue l’esprit avec toutes sortes de questions accessoires qui s’y trouvent.

The simplest and closest synonym of the expression, however, is “Il n’y a pas de quoi se…”.

• Il n’y a pas de quoi se nourrir l’esprit, de plus qu’on y trouve un bon nombre de questions qui ne font que le polluer.

ADDED CLARIFICATION (suggested by user suiiurisesse)

It must be admitted that this point concerning the would-be “positiveness” of the action is a little abstruse and it is a good idea to make the caracterisation of this concept more specific.

“Matière”, in the sense considered is defined as “ce qui constitue l’occasion d’une action”. If the matter in question serves a purpose (s’instruire, se suicider, etc) it is “positive”, as I put it.

• Il n’y avait pas là matière à glaner la moindre information.
• Il y a là matière à se suicider pour le boursier et il lui arrive de le faire.

If that idea of evident purpose is missing, there is not a logical basis anymore for using the word. In the instance from the original post, “se polluer l’esprit” can’t serve a purpose, if taken as usual: there is an interest in the accumulation of more information, an interest in suicide since the act involved aims at doing away with one’s misery, etc., but not in the act of making one’s mind murkier.

Il n’y a pas de quoi se polluer l’esprit would be another way to phrase it.

"Il n’y a pas matière à…" implies minimization, and can be used to express that something is too trivial, too small to bother with.