It is a little too strong and colloquial, but an expression that cames to mind is:

péter les plombs (to freak)

Otherwise, you might say:

avoir le cerveau en surchauffe

Méninges is old fashioned and fondre isn’t very idiomatic when applied to the brain.

Here is how I would arrange your sentence:

Apprendre l’arabe m’obligeait à me creuser la tête comme une folle ! Mon cerveau n’était pas loin de la surchauffe…

To express something difficult for someone to understand I sometime use the sentence which means thinking a lot about something.

Se faire des noeuds au cerveau en étudiant l’arabe

The same idea can be conveyed this way too :

Se prendre la tête en étudiant l’arabe

With the sentence se creuser les méninges you get the idea, but the meaning is not pejorative