I’m not sure if this is a proper rule, but I suppose a way to distinguish between the two is by looking at what follows the ‘que’.

If the que is there to introduce a subordinate clause, like in your example, then it is the ne littéraire.

If the que introduces a verb, a noun, an action etc (ie je ne mange que de la viande, je n’aime que dormir sur le dos), it is the classic ne…que

Your understanding is correct, and Nico Mezeret’s rule of thumb is actually the correct answer.

Litterary form

This is the dated/litterary form of "ne" (actually "ne … pas" with "pas" being omitted), followed by a "que".

Thus, the (simplified) sentence

On ne saurait en conclure que l’alchimie est une religion.

could be written

On ne saurait pas en conclure que l’alchimie est une religion.

The real rule here is that the "que" is not related to "ne", it only serves as to introduce a subordinate clause (which as a consequence includes a conjugated verb).

Then, the translation uses not.

Restrictive form

Some examples:

Je ne mange que de la viande. (I only eat meat.)
Ce ne sont que des insectes. (They’re only bugs.)
Je ne fais que passer. (Just passing through!)

The thing here is that there is no conjugated verb after que. Infinitive verbs and nouns (nominal groups, with sometimes a partitive as in our first example) are possible, not subordinate clauses.

As BBBreiz noted, the most generic translation for "ne … que" would be "nothing but".

Trap form

As the asker noticed, if you remove "pas" before "quoi", "quoi" becomes "que". The translation however is not the same as in the first case. Let’s have examples:

Je ne sais pas quoi faire. > Je ne sais que faire. (I don’t know what to do.)
On ne saurait pas quoi répondre. > On ne saurait que répondre. (You wouldn’t know what to do.)