Que s’utilise avec un verbe transitif :

Tu m’as donné un livre. Je lis le livre que tu m’as donné.

Dont s’utilise avec un verbe intransitif :

Tu m’as parlé d’un livre. Je lis le livre dont tu m’as parlé.

Que replaces a direct object (COD in french ‘Complement d’objet direct‘ ).

Dont replaces an object (or person) after ‘de’ like in this example:

Voici l’enfant. J’ai trouvé le ballon de cet enfant.
Voici l’enfant dont j’ai trouvé le ballon.

As a side note to the answer given by Laurent, even a French native speaker sometimes needs to think about the inverse sentence “Tu m’as donné un livre” to deduce the correct insertion of “que” w.r.t “dont”.

In the second sentence “Tu m’as parlé d’un livre”, mind the “d'” (or “de” depending on the case), that will lead you to the insertion of “dont”.

Not sure it’s helpful, though. (That’s my first answer here.)

I can’t explain the grammatical rules (too much technical terms) but an easy way to remember the difference is that you can translate ‘dont’ with ‘of which’.

(Note that in your two proposed translations, the ‘which’/’that’ are elided).

It can sometimes help to think of dont as an invariable synonym for duquel (or de laquelle, desquels, desquelles).

Que = that

Dont = of which, of whom.

“Je lis le livre que tu m’as donné.” = I am reading the book that you gave me.

“Je lis le livre dont tu m’as parlé.” = I am reading the book of which you spoke.

Here as in all relative clauses it’s a case problem (yes there are cases in french). Both dont and que (as duquel, auquel, etc) replace the noun in the relative clause :

• Dont is the genitive form and it’s used to mark a complément du nom

• Que is the accusative form and marks a complément d’objet direct, a direct objet of the relative clause. “Que” is also an ambivalent word equivalent to the English “that” or even “what” as in “je sais que j’ai raison” (i know that i’m right) or in “les mots que j’ai dits” (the words that i said) (and i’m not sure about that s after dits)

• Auquel would be the equivalent for an indirect object.