The proper sense is this one, found in the TLFi ;

A. Abondance de biens, de moyens, de revenus; état, condition d’une personne qui possède des biens très importants, qui a beaucoup de ressources, de revenus. Synon. fortune, opulence; anton. besoin, dénuement, misère, pauvreté. Signe extérieur de richesse; transmission héréditaire de la richesse.

If the word is used in the plural, it is difficult, if not impossible to conceive this next one to be proper ;

C. P. méton., le plus souvent au plur.
1. Tout ce qui est susceptible de combler, de satisfaire les désirs, les besoins de l’homme. Les richesses de la nature; richesses naturelles, végétales. L’eau se trouve être la richesse économique par excellence; elle est, pour les hommes, plus richesse que la houille ou que l’or (BRUHNES, Géogr. hum., 1942, p. 31).

The reason is that the fundamental idea is that of “fortune” ; if it is used in the plural there is a subtle difference to be made: “richesse” is not quite “fortunes”. If you say “les richesses amassées ces quarante dernières années” you are not saying les fortunes amassées ces quarante dernières années“. In the first instance you are talking about property in all sorts of fields (financial, arts, technology, real estate) and you are talking about it either as regards that of a single individual or legal entity, either as that considered generally in a certain domain. So, if you say “les richesses accumulées par les firmes de l’industrie de l’ordinateur” one is bound to understand that you are talking about an undifferentiated whole of whealth, without concern for form or ownership, whereas if you are saying “les fortunes accumulées par les firmes de l’industrie de l’ordinateur” one is bound to understand by this the seperate amounts of important property belonging to individuals or legal entities in that field, this property being moreover not considered as marketable values but rather as its current monetary worth.

Therefore, “fortune” can be used in the second possibility; but we must understand that we are not saying quite the same thing as when we use “les richesses”; it remains to determine if we can use “la richesse” as well. That is more difficult and the answer is that in the présent context only “les richesses” is really proper.

On ne dira pas très naturellement “la richesse qu’il a accumulée” et “la richesse qu’il a amassée” mais
les richesses qu’il a accumulées et les richesses qu’il a amassées (ou “la fortune qu’il a accumulée/amassée”).

Why this is so is the consequence of an elusive notion: “richesse” is a less pragmatic term, a term that has been dedicated to abstract contexts, whereas “richesse” in the plural has been specialised to the basically material. This can be seen from another ngram for the word “langue“; of course, there does not exist a taboo, but the idea that imposes itself in that domain is tant communicated by the singular, that is the idea of a general, essentially abstract caracteristic, rather than that of several, specific, albeit abstract, features.

• la richesse de son vocabulaire et non les richesses de son vocabulaire
• la richesse de leur pensée et non les richesses de leur pensée
• la richesse des nuit d’été en inspiration poétique et non les richesses des nuits d’été en inspiration poétique

I see two use cases here:

• Considering that there is potentially only one form or fortune (monetary) I will suggest to use the singular form:

  Ça ne sera pas une entreprise facile. Cela dit, elle ne doit pas s’appeler la "mine d’or" pour rien. Imaginez un peu la richesse à côté de laquelle vous risquez de passer en refusant ce contrat !

• Considering that there are potentially various forms or fortune (monetary, happiness etc..) I will suggest to use the plural form:

  Ça ne sera pas une entreprise facile. Cela dit, elle ne doit pas s’appeler la "mine d’or" pour rien. Imaginez un peu les richesses à côté desquelles vous risquez de passer en refusant ce contrat !