« Il faut la jouer fine » means that someone has to act wisely, to be smart.
The « la » isn’t used because some idea of limit is implied. You can use that expression in a sentence where the idea of limit isn’t present, for instance :

Mes chances de succès sont faibles, je vais devoir la jouer fine.

« la » and « fine » are refering to the case, l’affaire, which gender is féminin. You could write your sentence this way :

Ne confonds pas hâte et précipitation. Il faut jouer l’affaire de manière fine.

Why say « la jouer fine » instead of « le jouer fine »?

Because the second sentence is incorrect, should be le jouer fin.

Why la instead of le?

It is not 100% clear what la refers to, but this recent set expression has always been using the feminine. As it is something that is played, my first guesses would be either la partie
(sports) or la pièce, la scène, la comédie (theater).

Here, fine doesn’t means “careful” but avec finesse, subtilité.

After some research, I found in Le bouquet des expressions imagées (1990) by Claude Duneton, that la is stated to probably refer to une bille (boule) de billard – peut-être de billard électrique, i.e. a ball used to play billiard or maybe pinball. This refers to an expression used in billiard: prendre la bille fine, i.e. effleurer (to kiss), which is the opposite of prendre la bille pleine (full ball hit). This perfectly matches the current usage of “la jouer fine“, i.e. instead of frontally attack some task, use finesse and tact to achieve it.