Rather than trying to translate the word “à”, it might be better to understand the phrase as a whole.

The phrase passer son temps means “to spend a lot of time, and even much more than one should reasonably expect”. There is a nuance of reproach or complaint, so it differs from the neutral phrase passer du temps.

Mes enfants passent leur temps sur leur téléphone

You can use this phrase with a verb to specify “what” one spends too much time on. As often in French, this is done by adding the preposition à with an infinitive.

J’en ai marre, je passe mon temps à nettoyer la maison

As to your question when to use à or de in such cases (I presume with an infinitive form), I am afraid there is no easy answer: that is something that native speakers learn naturally, but that you, as non-native, will unfortunately need to study… There are some trends you can remember (see here) but for a lot of adjectives, nouns or idiomatic phrases, you just need to learn it case by case.