Les trois premiers livres or les trois premiers is the standard order.

Les premiers trois is odd and unusual, but it might be a shortcut for “le premier groupe de trois”. I guess it can be used orally to avoid ambiguity when the objects are grouped already.

J’ai aimé les trois premiers livres.
J’ai aimé les trois premiers.

It is not possible to put premiers before trois in these sentences. The number has to come very early in the order of adjectives. Although the number is classified as an adjective, it behaves a lot like a part of the article, and could be an article on its own (j’ai lu trois livres cette semaine).

I can’t think of any context where you could say *les premiers trois livres. The only circumstance I can think of where les premiers trois is possible is if the number is part of a continuous unit of measure (all the examples in modern French on the first few pages of Google Books search hits fit this):

Les premiers trois kilomètres furent faciles.
Les premiers trois mois d’abonnement sont à mi-tarif.

Even in these cases, it would be common to say les trois premiers mois, etc. The syntax les premiers trois mois implies that the three months are taken as a single unit. If the subscription is payable monthly, then les trois premiers mois is the only possible word order.