You can say both.
Ex: Désolé pour le retard que j’ai eu ce matin.
Ex: Désolé du retard que j’ai eu ce matin.

Both do work, you would use pour for day to day speech and du for the formal one.

I’ve been scratching my head for a minute trying to figure out an actual difference between the two, but even in terms of formality, I’d have to say that both are in effect perfectly equivalent. Use whichever one, no sweat.

I would preferably use

désolé de + "infinitif" (je suis désolé d'être arrivé en retard)

and

désolé pour + noun/pronoun (je suis désolé pour mon retard)

But this is not a strict rule (désolé de + noun is also correct as far as I know) and both are formal.

As a remark, it is more formal to say "Je suis désolé.e …" than "Désolé.e …")

It doesn’t seem to me that there are any rules.

In familiar language (with friends, family), we will use one or the other. In a more formal context, we will rather use "de". However, in either case, we are sorry for someone: "Je suis désolé pour Paul qui vient de…".

The problem with "désolé pour" comes from the rest of the sentence. For example, if I replace "retard" with "la mort de quelqu’un" (someone’s death):

• "Je suis désolé de la mort de Paul."
• "Je suis désolé pour la mort de Paul."

The second sentence sounds less good. So even if there is no particular rule, "de" will be preferred.

In any case, as the others have specified, in a formal context, we will not use "Désolé" but "Je suis désolé". And in the case of "retard", we will instead use "Veuillez m’excuser de mon retard".