Il semblerait que ce ne soit pas de Rousseau mais de Fénelon, Les aventures de Télémaque.

To build upon @oldergod’s answer:

Though the quote is often attributed to Jean-Jacques Rousseau, it is always unsourced (not attached to a book).

As oldergod found out, the quote appears first in Fénélon’s Les aventures de Télémaque (1699) as follows:

Les injures sont les raisons de ceux qui ont tort.

This work is known to have influenced Rousseau’s work, so it is possible that he later said or wrote it himself, but this is likely a reference to Fénélon’s thoughts.

This idea is quite common nowadays and can be easily found over the internet in one form or another, often left without an author.

Another example by Pierre-Claude-Victor Boiste was found in the Dictionnaire Universel (1843):

Les insultes sont toujours les raisons de ceux qui ont tort.

You could find more if you search for it.