Neither of the explanations provided do work.

En is a pronoun that refers to something that is not in the sentence (similar to of it) so:

Il faut d’abord en définir les termes.

means

“Their terms need to be defined first”

or

“We need to define the terms of it first”, i.e.:

Il faut d’abord définir les termes de ce dont nous parlons [ex: du contrat] or whatever this is about.

About your second question. Being a native speaker is not a guarantee of expertise. Of course, having a lifetime of practice in a language helps a lot and gives a strong edge compared to a non native learner, but outside linguists, grammarians and the likes, most of the knowledge average native speakers (including myself) have about their language, whatever it is, is made of habits, intuition, reflexes more than theory. In that case, the intuition of both of your friends misled them although the essential meaning wasn’t lost.