Il se brosse les dents is the usual sentence you’ll heard.

Il se brosse son dents is doubly incorrect.¹

First the possessive must agree with dents so that should be ses dents (or sa dent if he only has one left 😉 ) and second, se brosse already tells this is about his own teeth so ses is redundant.

You might have said:

Il brosse ses dents.

although in that last sentence, it is not 100% guaranteed his is brushing his own teeth. The context might tell he is brushing someone else’s teeth, or an animal teeth.

¹ _{The original (i.e. unedited) question was proposing son dents.}

Possessive pronoms are not always used when talking about something happening to one or more body parts.

Il s’est cassé la jambe : he broke his leg.
Il se frotte les mains : he is brushing his hands (together)
Il se gratte le nez : he is scratching his nose.
il a la peau rouge : His skin is red

The absence of context here implies that we’re talking about his body parts, not someone else’s

But

Il prend soin de ses cheveux/ses dents/ses mains : He takes care of his hair/his teeth/his hands

The presence of ‘de’ in the last sentence then needs a possessive to clarify which hair/teeth/hands he is taking care of.

Both “il se brosse les dents” and “il brosse ses dents” are correct, no reason to use one over the other.

The closest example I can find in english would be like “he is brushing his teeth” and “he is brushing himself on the teeth“. Even if the latter is not correct in English it corresponds to “il se brosse les dents“. Since himself or se already specifies that it is about him you don’t need to specify again that it is his teeth.

But when you said “he’s brushing his teeth” “il brosse ses dents” you have to specify it.