Your guess is correct if you mean “Would he not have woken up?”

Il n’aurait pas se réveillé

is indeed incorrect.

Il ne se serait pas réveillé

is much better and the interrogative form is indeed also correct:

Ne se serait-il pas réveillé ?

If you want mean: "He wouldn’t have woken up."

"Pronominal verbs" have to be conjugated with the auxiliary verb "être" and not "avoir".

Thus, you are correct with :

Il ne se serait pas réveillé.

Ne se serait-il pas réveillé?

For more information about conjugating past participle with pronominal verbs here is French article from the CCDMD (Subject which is indirectly related to the question).

Pronominal verbs use exclusively être as an auxiliary verb for composed tenses. Therefore, independently of the word order, your first phrasing can’t be correct.

Your second guess was right. To augment your level of confidence in it, you might want to proceed step by step. To avoid potential confusion due to homophones “est” and “ai”, best is to also consider the 1^st person.

Start with present indicative for the auxiliary verb:

Je me suis réveillé.
Il s’est réveillé.

Add negation:

Je ne me suis pas réveillé.
Il ne s’est pas réveillé.

Turn this into question form, with subject–verb inversion:

Ne me suis-je pas réveillé ?
Ne s’est-il pas réveillé ?

Switch to conditional:

Ne me serais-je pas réveillé ?
Ne se serait-il pas réveillé ?

You may of course consider different paths. Yet, your final guess was right.