By tout, Montesquieu is referring to the liberty (mentioned earlier: il n’y a point de liberté), and beyond that, the justice, the legitimacy of the regime.

If the executive, legislative and judiciary powers are not separated, liberty and justice would be lost and lead to what we call today a dictatorship.

I think it can be said that Montesquieu could have been more specific; his formulation gives an impression of vagueness and is somewhat a slipshod characterization that the reader might be tempted to impute to a real writer’s block in the process of materializing what actually corresponds to it, if not worse. We can’t but speculate, inspired by the spirit of his discourse, that he means by that "all possibilities of a salutary form of government would be foreclosed".

Text. For the same reasons set forth in another answer, la liberté. We’re dealing with details on the same topic, the preceding lines are about la liberté politique and its requirements and then what stems from the confusion of those branches. Then you have your quotation and a few lines further down you have "voyez quelle peut être la situation d’un citoyen dans ces républiques […]" and you’re exploring examples where the aforementioned requirements might not have been met and the impact of this. Of course la liberté is not something construed in a vacuum etc. and that’s really what the text and its topic are all about. In my opinion the author completely masters the language and the topic in this famous work, and takes it exactly where he wants to, showcasing its vital importance in the process.

Grammar and usage. That being said, generally tout as a pronoun stands for "toutes les choses", and in context of what would be lost that would certainly highlight the utmost importance and seriousness of the consequences of not meeting the conditions discussed.

Furthermore, the pronoun tout may be used in a literary turn of phrase to mean "everyone" (« Tout avait fui, même les médecins », Chateaubriand) (LBU14 §766). In context that would mean the same as "tous seraient/nous serions tous perdus" (everyone would be/we would all be lost), not unlike "we would fail" in a way, as in we would fail at having this liberty, again, if…

Incidentally, tout can also be used to mean l’essentiel in certain cases. Like in avoir tout pour plaire, tout is not construed as meaning to have every single thing to please, but rather about having what it takes to please, what’s essential to pleasing, whatever that may be. But in context, once again, what is of the utmost importance, what is essential for everyone and which would be lost, that liberté, makes sense.