Il n’a rien pu faire is the standard and most common ordering.

Il n’a pu rien faire is rarer but used, especially in spoken French.

Il n’a pu faire rien is incorrect.

As Aerovistae writes, the first one appears to be far and away the most common wording.

In the past, however, the second appears to have been more common and still sees some use today. This is not surprising, since the syntactic status of rien is not simple (see CNRTL entry), overlapping with noun phrase types, and parsing it as something closer to a direct object would of course place it after pu and any other modal. It seems that for several decades it has been parsed much more as a negative particle, like pas.

When you see lines cross on an Ngram like that… you’re watching diachronic (historical) linguistics in action 🙂

You’ve got a lot of answers about “rien” but if you want to use “pas” then a little thing has to change , you need to refer to something :
“Il n’a pas pu faire quelque chose”
“Il n’a pas pu le faire” where “le” refers to the action.

You can’t just write or say “Il n’a pas pu faire.”

Exept the case where your sentence looks like “there’s something he couldn’t do” which can be translated as “Il y a quelque chose qu’il n’a pas pu faire” but here again you refer to an action. It’s just that you just do it before.