Actually it depends on the auxiliary verb.

• When you use “être” the past participle varies in terms of gender
  and number.
• With “avoir” the past participle is invariable.

Source (la-conjugaison.nouvelobs.com) : être, avoir

The auxiliary used for the verb être is avoir. Like for all other verbs that use avoir as an auxiliary the agreement of the past-participle is done with the direct object (only if it appears before the verb).

Elle a pleuré. (no direct object)
Ils ont posé des questions. (direct object after the verb)
Les questions qu’ils ont posées. (direct object before the verb)

For aller the auxiliary is être, and in this case the agreement is done with the subject.

Elle est allée dans le jardin.
Ils sont venus rapidement.

French uses two auxiliaries for its compound tenses : être and avoir.
être is mostly associated with intransitive verbs, and all reflexive constructions:

• Paul est parti (‘Paul has left’) intransitive
• Paul s’est vu dans le miroir (‘Paul saw himself in the mirror’) reflexive

avoir is mostly associated with transitive verbs (and never used in reflexive constructions)

• Paul a mangé (‘Paul ate’) lexically transitive verb
• Paul a été brave (‘Paul was brave’) être selects avoir as an auxiliary even though it is not a transitive verb…

Now for the past participle agreement, French has only 3 cases:

1. Reflexive constructions behave exactly as if they selected avoir
2. with the être selection, the past participle agrees in gender and number with the subject argument
3. with the avoir selection, the past participle can agree in gender and number with the direct object if it is placed before the participle and it is not the clitic pronoun en

Examples:

• être agreement with the subject
  • Paul est mort vs Marie est morte vs elles sont mortes
  • il est arrivé vs ils sont arrivés vs elle est arrivée vs elles sont arrivées
• avoir agreement with the direct object on the left (except en)
  • Marie a mangé des pommes and Marie a mis des robes rouges
    direct object on the right => default agreement: M.SG
  • Marie en a mangé and Marie en a mis des rouges (*mises)
    direct object = en => default agreement: M.SG
  • Les pommes que Marie a mangées étaient bonnes and les robes que Marie a mises étaient jolies
    direct object on the left => agreement in gender and number with the object: F.PL
• reflexive and reciprocal constructions => same as avoir
  • Marie s’est écrit un post-it
    direct object = un post-it
    direct object on the right => default agreement: M.SG
  • Marie s’en est mis sur le col
    direct object = en => default agreement: M.SG
  • Marie s’est vue dans le miroir
    direct object = se
    direct object on the left => agreement in gender and number with the object: F.SG
  • Les roses que Marie s’est offertes sont jolies
    direct object = les roses
    direct object on the left => agreement in gender and number with the object: F.PL

As mentioned by @Sifu, these are the rules for formal written French. Colloquially speaking the agreement is welcome but not realized with avoir most of the time. The proportion tends to be higher with être. In very formal context, some people tend to use what we could call hyper-corrected French and make mistaken agreements even with the en clitic…