Your sentence should read:

Mes amis dont je déteste les enfants…

Unlike in English, the order in French within a "dont" clause is always:

subject –> verb –> direct object (with article)

In English, it’s equivalent to "of whom" and sounds rather stilted to my native ears:

My friends of whom I hate the kids…

Another example is:

L’homme dont je connais la sœur…

The man whose sister I know…

Not quite. It’s important to note that the embedded clause is a full sentence:

Mes amis dont les enfants sont si mignons viennent en vacances avec nous.

Les enfants sont si mignons. The kids are so cute.

This will also be true of any other clause you want to embed.

So to decide what it should be, just construct the normal sentence for “I hate the kids” and insert it.

I hate the kids. Je déteste les enfants.

→ Mes amis dont je déteste les enfants viennent en vacances avec nous.

Your attempt is not entirely ungrammatical, but it’s a different construction in which dont is followed by a nominal group that makes this particular sentence semantically doubtful (“My friends, among whom are the kids I hate, are coming with us”). But this isn’t what you had in mind anyway.

In fact dont doesn’t always attribute the subject of the relative clause to its antecedent; it can also apply to the object. Unlike in English, the word order in the relative clause isn’t affected. So, the sentence you’re looking for goes as follows:

Mes amis, dont je déteste les enfants, viennent en vacances avec nous.