According to Wikipédia:fr it did refer to manual workers, who first wore pantalons (trousers) instead of culottes (breeches). It had no direct links to wealth, since even some upper-class craftsmen wore pantalons.

Historically, the name sans-culotte refers to some zealots of the French first revolution, who were indeed manual workers.

The bottom line, so to speak, is that the phrase sans-culottes was coined during the most radical period of the French Revolution to designate people of the lower classes
who did not wear the culotte.

The culotte was an elegant pair of knee-breeches typically worn by members of the gentry and the bourgeoisie alike.

For mere practical reasons the working class favoured the more robust long trousers called “pantalon” (named after a greedy, gullible and libidinous Venetian character of the Comedia dell’Arte). Knee-breeches would not cover the lower leg which would thus be encased into silky stockings. No need to say that such delicate garment could not have sustained the daily activities of the “working class” for whom they were unaffordable anyway.

Although the expression sans-culotte is attested prior to the Revolution it really appears to have gained momentum in the early 1790s.

In early 1792, after a few months of existence, it became evident that the Revolution had been confiscated by an elite made of an alliance of progressive aristocrats, wealthy bourgeois and members of clergy. This state of affairs was seen as an opportunity by the more radical elements who stirred up the turbulent parisian crowd to their advantage. The sans-culottes was an iconic figure of this faction because their costume would stand in stark contrast with that of a bourgeoisie aping the aristocracy. Anyone wearing a culotte was a réactionnaire (a counter-revolutionary) and anyone who wasn’t was instead a true revolutionary.

This is the official etymology. The Littré however, reports an interesting anecdote possibly seizing the very instant when the phrase was coined. FWIW of course.

SANS-CULOTTE. ÉTYM. L’origine du nom de sans-culotte paraît être différente de celle qui est indiquée dans l’article. Un jour que les femmes qui occupaient les tribunes de la Constituante étaient encore plus bruyantes que de coutume, l’abbé Maury dit au président : Monsieur le président, faites taire ce tas de sans-culottes. De là le nom de sans-culotte adopté par les patriotes d’alors. BOURLOTON et ROBERT, la Commune, Paris, 1872, p. 169.

Translation for NNS:

The origin of the phrase sans-culotte appears to be different from what is indicated in the [main] article. One day, as the women occupying the higher stands of the Constituent [Assembly] were even more noisy as usual, Cardinal Maury addressed the Speaker: Mister Speaker, could you please silence this throng of “sans-culottes”. From thence the name of sans-culotte was adopted by the patriots.