I believe this is conditionnel passé II simply because it is not a “que” construct, and does not express a desire, wish, emotion, obligation, doubt or uncertainty.

Using the imparfait in this case would be equivalent to using the conditionel présent, and would be incorrect. The other option you have is to use the conditionel passé I, or “comme si elles avaient essayé”.

After having delivered hasty and thus unprecise analyses, I have calmly considered the problem you raised: It shows you’re quite keen at French, because indeed you detected a mistake: I can affirm that, in that sentence, Alexandre Dumas effectively made a mistake (but after all, the law “errare humanum est” can apply even to a great author, can’t it?).

First, we can definitely assert that “eussent essayé” can represent 2 different tenses:

• Plus-que-parfait du subjonctif
• Passé 2e forme du conditionnel

But here, the proposition is:

“Comme si elles eussent essayé”

So, “comme si” introduces the hypothesis (first clause) of a conditional sentence. In that first clause, you cannot use the conditionnel which is reserved to the conclusive clause (the second one), but only a past tense of the indicatif (imparfait, plus-que-parfait, passé antérieur):

• Imparfait:

Si je gagnais au loto, je cesserais de travailler (If I won at the lottery, I would leave my job)

• Plus-que-parfait:

Si j’avais gagné au loto, j’aurais cessé de travailler (if I had won at the lottery, I would have left my job)

• Passé antérieur:

Si j’eus gagné au loto, j’eusse cessé de travailler (same translation as plus-que-parfait).

So, firstly we realize that the preposition “si” forbids the use of the conditionnel, because it introduces the first clause of a conditional phrase, and that clause compulsorily relies on the indicatif mode.

Then we must investigate the other possibility: plus-que-parfait du subjonctif. It is true that the idea of the phrase can correspond to the subjonctif, because subjonctif, like conditionnel, relies on events that may be unreal, like hypotheses.

But here again, the introduction “Comme si” firmly requires the first clause of the conditionnel, and “si” can’t match a subjonctif. Let’s try to shift the subjonctif from plus-que-parfait to imparfait:

“Commme si elles essayassent”

We can clearly see that it doesn’t fit. Here you just could say:

Comme si elles essayaient

You can say:

“Il faudrait qu’elles essayassent”

but you can’t place”essayassent” after “si“, because you cannot use the subjonctif after “si”, but just a past tense of the indicatif mode.

The conclusion is surely that Alexandre Dumas made a mistake, and the quite literary form “eussent essayé” drives this mistake invisible, since it has become a general use, but we can consider, though, that it is a mistake. Dumas should have written:

Comme si elles eurent essayé

The plus-que-parfait du subjonctif form could be used without “si“, with the inversion of the subject’s place:

Eussent-elles essayé cela, il en serait advenu ceci (or il en fût advenu )

This is a correct subjonctif, but you can’t use it after “si”, though the use you mentioned from Dumas was probably widely made in the 19th century, since Grevisse (Le Bon Usage, 2715) says that “comme si” can be used with the subjonctif plus-que-parfait (instead, in fact, of the indicatif passé antérieur). In that case, the sentence of Dumas contains no mistake, but one must admit that this use of subjonctif is not grammatically logical: it is purely arbitrary, probably a mistake that became a general use, just like the adjective “glauque”, for example, which today means “troubled”, a new sense taken only in the last 50 years, meanwhile the real sense is “green-blue”: It used to be applied to the eyes or to waters. Not knowing the real meaning of “eaux glauques“, some readers thought it meant “troubled” and in the end, that sense became generally accepted. We may suppose it was the same for that unlogical use of the subjonctif.