Avoir à means must, have to. The construction is similar to the English have to, and in this particular sentence it works as a translation, although I think a better translation would be “In summary, I didn’t need to apologize”.

Avoir à + verbe is mostly used in negative sentences: “Je n’ai pas à faire cela” = “I don’t have to do this”. “Ne pas avoir à …” means that the action is not compulsory. There is also a related construction Avoir + nom + à + verbe which doesn’t have this restriction: “J’ai quelque chose à faire” = “I have something to do”, expressing a moral or practical obligation or even something that one wants to do.

The equivalent of ”I’m not going to apologize“ would be “Je ne vais pas m’excuser”. The equivalent of ”I wasn’t going to apologize“ would be “Je n’allais pas m’excuser”. “*Je n’avais pas m’excuser” is not a French sentence. “Je ne m’étais pas excusé” would be “I didn’t apologize”.