Here, the sentence has the meaning of:

Je fis comme mon père m’avait dit de faire.

I did as my father had told me to do.

And implicitly, "l’" stands for that.

It could be translated by:

I did as my father had told me.

Don’t worry for the "l’", English doesn’t need it to have the same meaning. The tenses used in French indicate that the moment my father told me is past relatively to the moment I did, that’s why I used plu-perfect "had told" (not sure if it is the actual name for this tense in English)

Note that

Je fis comme mon père m’avait dit.

can be heard in French, even if it sounds bad for conservative linguists.

"Le" is a pronoun, it means "it" (accusative).

There is an equivalent in English, but it is easier to see it in a simpler construction:

Je te l’avais dit.

I had told you so.

Arguably, "so" is an adverb, but in that case it used as a pronoun (and American Heritage Dictionary considers it as a pronoun).