Both are rude, but « Ferme-la » is the strongest. These are very disrespectful and you should avoid using them. Actually, as mentioned by Simon Déchamps in the comments, it’s rude because it’s imperative. Asking someone to stop talking is in most cases rude. To stay polite the good way would be to ask the person to make less noise (Pourriez-vous faire moins de bruit s’il vous plait ?).

« Tais-toi ! » is sometimes used to ask a child to be quiet, and in an adult-to-children talk there’s no problem using it, but in an adult-to-adult conversation it’s disrespectful due to its imperative aspect, though the verb itself isn’t rude. « Ferme-la » is disrespectful even when talking to children, and it’s rude anyway.

Note that even though « Tais-toi ! » can be used toward a child, there are still more respectful ways to tell it, by example :

Pourrais-tu te taire s’il te plaît ?

Which translates to « Could you please stop talking ?`

If you want it to sound more like an order, adding « s’il te plaît » will make it more polite :

Tais-toi s’il te plaît

Ferme-la ! (or La ferme !) is very rude although you might be even more rude with Tu vas la fermer ta gueule ! or just Ta gueule !.

Tais-toi ! or Taisez-vous ! is only acceptable if you are talking to kids or to people under your orders, better to add s’il te plait / s’il vous plait to slightly soften the request.

With friends, you might say Camembert (for ferme ta boite à camembert…)
but it’s more a joke than anything, and only used in France.

If you are talking to someone you don’t know, directly asking to that person to stop talking is rude, whatever the way you say it. This is true in English too.

In such case, you’d rather indirectly ask like one of these ways:

Est-ce que je peux vous demander un peu de silence s’il vous plait ?

Excusez-moi, pourriez-vous parler un peu moins fort s’il vous plaît ?

Pourriez-vous faire un peu moins de bruit pendant que je … ?

Finally, chut ! or just the index in front or the lips is a common way to ask for silence, exactly like the "shhh!" gesture.

Femme la bouche – shut your mouth

Both are direct orders, so depending on the context they may both be rude. You wouldn’t go up to a stranger and tell them either “tais toi” or “ferme la” (or even “taisez vous” ou “fermez la”) unless you wanted to aggravate them.

“Tais toi” on its own is not really rude; it’s the presumption that you can give someone an order when you can’t that’s rude. A teacher can certainly tell their students “taisez vous” so that they make less noise, because the teacher has authority over the students, for example. But even in this case “Fermez la” would be rude, as “la fermer” is rude in any case, it’s definitely not elevated language. If you’d like an analogy in English, “tais moi” means “stop talking” while “ferme la” means “shut up” (it’s not a perfect translation obviously).

If you want a more polite manner of telling someone to stop talking, find a way to tell them in a non-imperative manner and add a “please”, such as “Pourrais-tu te taire s’il-te-plaît”, or if you use “vous” with the other person, “Pourriez-vous vous taire s’il-vous-plaît”.

If you want to be even ruder, a common expression is “Ferme ta gueule”, or simply for short “ta gueule” — “gueule” being the word used for the mouths of most animals, so you’re at the same time ordering them to stop talking and implying they’re an animal. If you say that to a random stranger, prepare for at least a verbal fight. “Ferme ta bouche” is the same thing but a bit less nasty, about on the same level as “ferme la” I would say.

I think the best way to say that is:

Un peu de silence, s’il vous plaît.

Which means:

Be quiet, please.