(Se) targuer can only be used in limited contexts.

• It is literary while vanter can be used in all registers.
• It needs to be pronominal, we can’t targuer quelque chose but we can vanter quelque chose.
• It needs to have a complement, “Il se vante.” is possible but “Il se targue.” is not.

Se vanter is to say/pretend anything that makes you appear good, or better than you are. One can se vanter of having done something, being something, owning something, … anything you want.

Se targuer would be more about a quality, a property, an ability you have. However the nuance is slight because one can consider that having done something proves a quality or an ability, and then se targuer can be used.

Se targuer might be more about showing an extra value, focusing on the added value more than on what brought it, while se vanter can focus on either a value or an action. In the end the idea is the same though.

If you look at etymology, targuer comes from targe which was a shield (definitely an extra value in a battle). Vanter comes from latin vanus = vain.

Note : [This answer could just be my opinion, I could’nt find any proof of this nuance while I found many counterexamples… But it’s often the problem with nuances]