If the next word starts with a vowel sound, the spelling l’, d’, m’, etc. is mandatory, and the word only contributes its consonant sound. This mandatory contraction also applies to la (la + école → l’école, je la vois → je l’ai vue).

If the next word does not start with a vowel sound, the formal pronunciation includes the mid central vowel sound /ə/, which is commonly classified as a schwa. This sound is often omitted in most variants of spoken French, except in poetry and in very formal contexts. Some regional variants of French, especially south-western France, realize /ə/ as [ø] and don’t normally omit it. In most variants of French, there is no rule for when to omit it: it’s a free choice.

Sounding every /ə/ makes you sound formal, but it isn’t wrong. Omitting every /ə/ can be hard for a native French speaker when it leads to a long series of consonants, but it isn’t wrong either. For example, “il faut que je voie”, which has two /ə/ in a row, is indifferently pronounced [il.fo.kʒə.vwa] or [il.fo.kə.ʒvwa]¹ in colloquial French, because the consonant cluster [kʒv] is possible but difficult to pronounce.

¹ _{Or, often, with “il” pronounced just [i], or even with the word “il” omitted altogether.}

Indeed there are no rules about our weird habit of skipping this /ə/ sound (except the one mentioned above), it’s all up to you according to 2 criterias: your level of language and the rhythm of the sentence.
Basically the more you skip it the more informal you will sound, and on the contrary if you never skip any you will sound very formal, maybe even too much – if you listen closely to formal speeches, you will see that they skip some too.
When you have several consecutive sounds with this syllable, like in the sentence "je me le suis rappelé" we normally (this is not a fixed rule but more a habit) skip one out of two, otherwise saying "j’m’l’suis rapp’lé" is impossible!

So you can say either "j’me l’suis rapp’lé / rappelé" (here the last sound /ə/ is not consecutive so you can do both) or "je m’le suis rapp’lé / rappelé".

None is better than the other, and it’s not a choice we even make consciously. Right now I would be tempted to tell you that I would probably go for the first option but tomorrow I might prefer the second one. It’s all about the rhythm of the sentence in itself, and also according to the previous and the following sentence, who you are talking to, and also just you and your personal way of talking that is specific to each one of us.