Both sentences are possible but with a potential difference in meaning. The first one is more common:

Ce livre a plus de quarante pages

just states the fact the book has more than 40 pages. It is an approximation, not a comparison. The final S of plus is generally not pronounced.

Ce livre a plus que quarante pages

might be contradicting a previous statement telling the book has (only) 40 pages. Unlike in the first sentence, a true comparison is made here between the number of pages of the book and the number 40:

Ce livre a plus de pages que quarante (pages).

If the expected meaning is the same than the first sentence (a statement, not a refutation), the register can be perceived as more colloquial.

Note that the final S of plus need to be pronounced in plus que quarante page. Otherwise, the meaning is “this book has only 40 pages left” (i.e. ce livre [n’]a plus que quarante pages (dans sa dernière édition or maintenant que tu en as arraché la moitié).

See this detailed answer about when to pronounce the final s of plus.

Ce livre a plus que quarante pages

Formation classique d’un comparatif de supériorité. Plus est ici adverbe.
En tant que tel c’est au verbe qu’il se rapporte. Avoir plus, avoir davantage.

• Ce livre a davantage de pages que quarante.

Ce livre a plus de quarante pages

Plus est ici adjectif.
En tant que tel il se rapporte à la quantité. C’est la quantité qui est supérieure.

• Le livre a des pages en quantité supérieure à 40

Avec un verbe comme avoir qui n’exprime pas particulièrement de process, les deux formulations portent rigoureusement le même sens.
Il en irait différemment avec d’autres verbes.

Littré prend son exemple avec porter :

Ce cep portait plus que vingt grappes C’est le port, l’effort de porter qui est plus important.

Ce cep portait plus de vingt grappes Le nombre de grappes était supérieur à vingt.

English traduction courtesy of @Laure

Ce livre a plus que quarante pages

It’s the usual way the comparative of superiority is built. In this case plus is an adverb and, as such, refers to the verb : it means avoir plus and plus is a synonym of davantage.

• Ce livre a davantage de pages que quarante.

Ce livre a plus de quarante pages

Plus is an adjective in this case, it refers to the amount of pages, the amount of pages is superior to 40.

With a verb like avoir, which shows no real process, both sentences have the same meaning, but it would not be the case with other verbs. This is the example we find in Littré :

Ce cep portait plus que vingt grappes The stress is on the effort of carrying. The vine stock carried over twenty bunches of grapes.

Ce cep portait plus de vingt grappes The number of grapes was superior to twenty.