The following words are apparently difficult to rhyme in French:

simple, quatorze, belge, quinze, goinfre, larve, monstre, meurtre, pauvre

I just found this by searching “le mot qui ne rime pas” in Google.

There is a list in an annex on Wiktionary.

Other than the words already mentionned, there’s huître, chanvre, sceptre, scepulcre, …

But there is no “famous word” like “orange”, that is known for not rhyming.

There are quite a few words that have no rhyme in French as well as in English. In English, orange and silver achieved some sort of celebrity, but nothing of this exact nature has happened in French.

Other answers listed some of these lone words of French. I will try to explore other types of rhymes that achieved a little fame in French, at least in some enthusiastic circles.

Rare rhyme

As far as rare rhymes making it to (a mild form of) celebrity goes, there could be some conjugations of the present of the subjunctive of the verb “perdre” (to lose), that is “perde(s)” as in “que je perde, que tu perdes, qu’il/elle perde”, that only rhyme with a single word of the French vocabulary, “merde” (shit).

Claude Gagnière, a compulsive lover of words and humor, and a witty collector of wordplays, introduced me to the concept of “fausse rime” (false rhyme) in his book Pour tout l’or des mots. To introduce his subject, he used this rare rhyme in a funny way, in which everyone is to think about “merde”, but where the word itself is never pronounced. Here it goes:

Voici l’auteur d’une comédie en vers en train d’écrire la scène au cours de laquelle un avare donne ses instructions au cuisinier qu’il vient d’engager.

_{L’AVARE
     Je veux qu’on garde tout et que rien ne se perde
     Car tout peut se manger…}

_{LE CUISINIER (effaré)
                    … Tout ?}

_{L’AVARE
                                   Tout ! Même la…}

STOP ! À ce point précis, notre auteur se trouve confronté au dilemme suivant :
— Ou bien il va autoriser son interprète à prononcer le seul mot français rimant avec le subjonctif présent du verbe perde […]
— Ou bien il renoncera à faire entendre le mot que, déjà, le public avait sur le bout de la langue, et il lui substituera hypocritement un autre mot n’ayant avec le vers précédent qu’un semblant de raison en guise de rime.
Choisissons la fausse rime sournoise et reprenons au départ :

_{L’AVARE
     Je veux qu’on garde tout et que rien ne se perde
     Car tout peut se manger…}

_{LE CUISINIER (effaré)
                    … Tout ?}

_{L’AVARE
                                   Tout ! Même la soupe
     Faite d’eau de vaisselle et de trognons qu’on coupe.}

La règle du jeu exige que soit marquée une légère hésitation avant de prononcer le mot substitué à la rime attendue.

Not rhyming

There is an expression in French that states “rimer comme hallebarde et miséricorde” (=not rhyming at all, or very poorly). Though this expression is a little dated, it is still around. It compares two words that don’t rhyme, but that were allegedly put as a rhyme in the XVI^th in an epitaph to a friend by a man who didn’t know much about literature, and who had been told that two words sharing the same final three letters were constituting a rhyme. Thus the (in)famous poem, in which no two verses rhyme :

_{Ci-gît mon ami Mardoche
Qui fut suisse de Saint-Eustache.
Il porta trente ans la hallebarde
Dieu lui fasse miséricorde.}

A cunning trick to complete a rhyme

Victor Hugo composed a long and well-known poem named “Booz endormi”, in which he invented the Chaldean city of Jerimadeth :

_{Tout reposait dans Ur et dans Jérimadeth ;
Les astres émaillaient le ciel profond et sombre ;
Le croissant fin et clair parmi ces fleurs de l’ombre
Brillait à l’occident, et Ruth se demandait}

The name flows well and has a sonority that could very well fit in the local geography. But as no one was able to pinpoint its real location, different ideas were explored to justify the name. The most realistic is that Hugo was only playing and created the name to be homophonic to “J’ai rime à -dait” (I have a rhyme to -dait), -dait being, of course, the rhyme required from demandait in the last verse.

Hugo didn’t do it because the French language lacks of this type of rhyme, but he nevertheless achieved celebrity with this poem, not only from its sheer beauty, but also and mostly from this witty pun.

In addition to the other answers, I would add the verb “sourdre”.

… with the additional challenge to manage to actually know its existence and use it in poetry.