This sentence uses more big words and subordinate pile-up than needed for the job. It is sadly typical of writing about pedagogy in France.

First, TS is terminale S (scientifique), i.e. the last year of high school with a science curriculum.

Now let’s start breaking down the sentence:

(Solliciter (de ses élèves de TS) (la narration (de leur recherche))) permet
(à l’enseignant de mathématiques)
(d’inciter (le plus grand nombre) (à se lancer (dans la résolution (d’un problème ouvert)))).

Your second translation gets the meaning right at the beginning but gets the relationships between actors wrong in the middle. Your first translation completely botches the beginning but gets the end broadly right.

Here’s a translation for meaning, keeping the grammatical structure of the original:

Requiring from his TS students that they narrate their research allows
the math teacher
to entice the largest number into attempting to solve an open problem.

Or rearranged into more idiomatic English: if a math teacher requests that his TS students narrate their research, this provides an impetus for many of them to commit themselves into tackling an open problem.

I’ll leave the styling and polishing up to you.