The word “en” is not always a preposition, in this case it refers to the situation, described in the previous sentence (“…s’asseyait toujours dos à la fenêtre.”).

An easier and more visual way to figure this is to imagine what the sentence would look like without this word : “S’il avait été autrement …”, it would be a little confusing because “il” could refer to “Mr Dursley”.

Hope I helped you with that 😉

You can find more information about “en” and its different use cases on this page

This could be translated as, “if it hadn’t been so” or “if this hadn’t been the case”. Thus, as Martin explains, “en” refers to the situation being described.