-eur/-euse is a suffix that attaches to verbal roots, thus creating an agentive noun. It’s a fully productive suffix, if I borrow swipe from English I can instantly produce the infinitive verb swiper and the agent noun swipeuse/swipeur.

-(a)teur/-(a)trice is also an agent noun suffix, but it’s not used to create agent nouns from native verb roots, but usually borrowed directly from Latin (géniteur, salvateur, oratrice, cantatrice), or backformed from a noun in -(a)tion (action -> acteur/actrice; promotion -> promotrice/promoteur; direction -> directrice/directeur)

Note that those words in -(a)teur/-(a)trice often don’t have a corresponding verb sharing the same stem: there is no * génir from which géniteur could come, and the pairs salvateur – sauver; directrice – diriger or promoteur – promouvoir can’t be derived from each other by a productive process.

-(a)teur/-(a)trice is thus a lot less productive than -euse/-eur, and often limited to administrative, technical, legal or religious vocabulary.

Coming back to menteur, you should hopefully now see it contains the -eur suffix, not the -(a)teur one. It’s transparently formed from the verb mentir (stem=ment-) and isn’t a Latin borrowing, and that there isn’t a corresponding -(a)tion noun (mention exists but is semantically distinct and monomorphemic: its verb is mentionner, conserving the whole noun as its root while you’d expect any derivational suffixes to have been chopped off).

Furthermore, in some nouns the suffix -teur appears as -seur instead (défenseur, possesseur, oppresseur) because of a sound change dating back to Latin. The feminine form of those nouns is usually uncertain because the expected -srice isn’t used in any other word while -euse feels wrong since it’d imply the existence of verbs like défenser, possesser or oppresser, which don’t exist.

In short, don’t trust the presence or absence of a t to know which suffix it is, look for the verb root instead.