You were close. Something like this would do

Je n’ai pas hâte d’être à Pâques.

In addition to your near miss in your question and the good accepted answer, the following constructions could also capture the general sentiment of “not looking forward to something”:

Je ne suis pas pressé(e) de voir Pâques [arriver]. (‘I’m not in a
hurry to see Easter come.’)

or

Je n’ai pas vraiment envie de voir Pâques arriver. (‘I really don’t
want Easter to come.’)

or

Ça ne me dit rien de voir Pâques arriver. (‘I don’t care much about
the coming of Easter.’)

The following one would probably be hyperbolic exaggeration and probably not extremely suitable when discussing religious holidays (unless one is terrified of some of the unbelievable paranormal events historically associated with these holidays, e.g., egg-bearing rabbits; chocolate-bearing flying bells; red-suited intruders descending chimneys), but I’ve heard Students use it at Summer’s end:

J’ai la hantise de La Rentrée (‘I absolutely dread the beginning of
school’ [which in most cases actually just means ‘I’m not looking
forward to the end of Summer’]).

Je n’ai pas hâte à Pâques.

Indeed avoir hâte à + (nom, infinitif) is simply more usual in Québec (see “hâte” in Dictionnaire historique de la langue française, sous la dir. d’A. Rey, ed. Le Robert), but correct nevertheless.