This is the distinction I would make (caveat: I am Belgian, I would not be surprised there would be other opinions in other French-speaking areas, esp. in Québec):

Un petit malin is someone smart, who uses their skills for finding subtle, practical solutions to tricky issues (and often with some mischief)

Ex:

Un petit malin a réussi à entrer par derrière.

C’est un petit malin, il a trouvé comment payer moins cher.

Un gros malin is actually not very smart, even if they think they are; gros "negates" the quality of malin. Typically, someone who does more harm than good when trying to play it smart.

Ex:

Ce gros malin a réussi à tomber en panne d’essence.

Mais tu te trompes complètement, gros malin !

So neither of the two phrases are really used the sense of "know-all"; you may want to use then eg monsieur/madame je-sais-tout (a person that knows it all and likes to show it).