Both sentences are correct, but don’t exactly carry the same idea. Here, “pense” means “believe”. “I don’t believe” means I’m not sure. “I believe he has not” is a stronger statement.

Le sens est un peu différent, les règles sont précises:

subjonctif, car forme hypothétique, après ne pas penser:

Je ne pense pas qu’il ait compris.

ou affirmatif, sens plus fort

Je ne pense pas qu’il a compris.

affirmatif obligatoire avec forme positive, sens encore plus fort:

Je pense qu’il n’a pas compris.

Both sentences are correct, but they aren’t synonyms. The nuances are the same as their literal English translations.

Je ne pense pas qu’il ait compris.

The second clause is negated: it’s a hypothetical. Thus we use the subjunctive mood.

Je pense qu’il n’a pas compris.

The second clause is not a hypothetical. There is some doubt regarding its veracity, but this isn’t the relevant issue to choose the mood. Thus we use the indicative mood.

Like in English, the actual meaning of both sentences is that I hold the belief that he hasn’t understood to a some level of certainty. The second statement is a stronger-held belief.

A neutral statement would also use the indicative mood:

An explicit expression of uncertainty might use the indicative or the subjunctive depending on the formulation. The following two sentences express about the same level of doubt. The first uses the indicative because it is formally neutral about the statement. The second uses the subjunctive because it is expressing a level of doubt about a hypothetical.

Je ne sais pas s’il a compris.
Je ne suis pas sûr qu’il ait compris.

All I’ve stated so far are grammatical rules. There’s a trend for people to use the indicative instead of the subjunctive, but a French teacher would mark a sentence like « *Je ne pense pas qu’il a compris » as wrong, and it isn’t common even in informal speech.

In a question, usage is more variable. Both of these questions are grammatically correct:

Penses-tu qu’il a compris ?
Penses-tu qu’il ait compris ?

The choice of mood conveys the opinion of the speaker. With the indicative, the speaker either implies that they think that he understood or remains neutral. With the subjunctive, the speaker implies that they think that he did not understand.