Usually the verb represents a movement, the fact of going somewhere, or about to do something.
Here, the verb is used in its imperative form to represent a wish, exhortation, threat or indignation.

The following wiki page has more detailed information on it.
“À l’impératif, sert également à faire des souhaits, des exhortations ou des
menaces et à marquer de l’indignation.”
https://fr.wiktionary.org/wiki/aller

Hope that helped.

Here “allez” is an interjection translating the impatience of the locutor. It comes from the imperative of “aller”, of course, but here it is only used for his interjection value. A good example of an interjection in English is “c’mon”, as one can say “C’mon, why is this happening to me ?” without anyone wanted to come actually.

“viens” is a real imperative of the verb “venir”, it means you want the person you a talking with to come.

Very interesting note, it’s queer, as you stress it, to juxtapose two imperative forms having opposite meanings (to go is the opposite of to come) and juxtaposing also two differents persons (“tu” and “vous“).

In fact, it shows that the expression “Allez!” has shifted its original meaning towards the fact that we use it to take the lead on the person(s) to whom we are talking. Then after having caught their (or his) attention, we can give an order, which can be of any kind, including to come, though we first seemed to give the opposite order to go, which in fact we did not.