Apparently you can have an inanimate subject for the verb "exploiter", but such examples are pretty rare. The most reliable dictionaries I have checked, exclusively give examples with animate subjects. I almost got tricked by this one from TFLi, but I realised the subject is a stingy person:

Le calculateur avare exploite sans pitié l’intelligence et le travail (Vigny, Chatterton,1835, p. 242).

Interestingly enough though, I did find such an instance (with an inanimate subject) in an article on grammar which does seem advanced, so I suppose the author knows what he is doing when he says:

En revanche, les constructions en "se faire" suivi d’un verbe transitif peuvent donner lieu à ambiguïté
puisque la valeur passive exploite aussi abondamment ce type de verbes.

This is not a personification, here the action of the verb "exploiter" is really performed by the inanimate subject "la valeur passive".

As for your sentence, "Ce nouvel algorithme complexe exploite les anciens algorithmes plus petits", I should have more knowledge in the field to discern if you can say that about algorithms. Google doesn’t find one instance of it and neither does Gngram. That doesn’t mean you cannot be the first, but I’d go careful with that nonetheless.

Yes, you can but it is not an optimal way to say it, for instance Ce pays (cet état) exploite une mine d’or (this country is exploiting a gold mine). I think a better solution would be profiter de, prendre avantage de.

A better translation to your sentence would be Ce nouvel algorithme complexe profite des (resultats des) anciens algorithmes plus petits if it is implied that it is building its results on older algorithms.

There are references in various domains, in particular that of science and computer systems.

(ref. 1) Here, the subject is "la mise en œuvre". in the first occurrence; in the second it is "GPS", which is a navigation system.

Par ailleurs, la mise en œuvre de la définition de la seconde, grâce aux horloges atomiques, exploite depuis déjà plusieurs années l’augmentation de précision permise par les atomes froids.

(ref. 2) In this second reference the subjects are "social networking" and "calcul intensif".

(ref. 3) The subject is now the word "forme".

La relation que cette forme exploite n ‘ est donc pas de l ‘ ordre …

(ref. 4) The subjects are "emploi de …" and "car".

It appears that you can use "exploiter" with an inanimate.

If the algorithms are all complex in the context of your discussion, then the translation is correct.

• Ce nouvel algorithme complexe exploite les anciens algorithmes plus petits.

If only this new algorithm is complex, then the following sentence is the translation.

• Cet algorithme nouveau et complexe exploite les anciens algorithmes plus petits. (more common than the following possibility)
• Ce nouveau et complexe algorithme exploite les anciens algorithmes plus petits.