No. When grammar calls for an adverb, you need to use an adverb. You can’t use an adjective, even in colloquial French.

Il parle lentement.
* Il parle lent.

Vite looks different because it’s one of the few adverbs in French that don’t have the suffix -ment. It is in the company of many common adverbs: bien, mal, beaucoup, peu, très, toujours, jamais, …
Adding this suffix (with the proper inflection) to an adjective is the only productive rule to make new adverbs.

Note that vite is not an adjective in modern French (except in Québec).

A way to choose between adjectives and adverbs is to remember that an adjective qualifies a noun and agrees with it, whereas an adverb qualifies anything but a noun (verb, adjective, adverb) and is invariant (except for tous).

There is one exception which is sometimes classified as “variable adverbs”, although it could equally be classified as “adjectives qualifying adjectives”.

des fleurs fraîches cueillies = des fleurs fraîchement cueillies
des portes grandes ouvertes

Apart from a few frozen idioms like “porte grande ouverte”, this is not in common use and the real adverbial form is preferred (at least in France): “fleurs fraîchement cueillies”.

Le français classique n’a fait que progressivement la distinction entre adjectif et adverbe ; par exemple “coûter bon” signifiait “beaucoup” – on dit encore familièrement “coûter gros”. Il ne faut donc pas s’étonner de voir chez les grands auteurs ce qui serait proscrit aujourd’hui (sans raison véritable : l’anglais est très libéral à ce sujet, sans que cela lui nuise).