In Glanville Price’s A Comprehensive French Grammar, the function of que in the construction que de + infinitive is thus explained:

261 In the construction c’est + complement + infinitive, when the infinitive is the ‘logical subject’ of the verb (as in ‘It would
be a mistake to leave’ which is the equivalent of ‘To leave would be a
mistake’), the infinitive is introduced by de or que de, e.g.:

C’est une erreur (que) de répondre à cette lettre.

Ce serait manquer de tact (que) de partir maintenant.

C’est agaçant (que) d’être mécompris.

Thus in such constructions que is always optional.

It is a more correct and formal way for your sentence.
There is another case where you can encounter a similar form

Je préfère courir plutôt que de marcher.

Most people would say

Je préfère courir plutôt que marcher.

Here you can see that “de” is often missing, but the correct form is the first one.
There is no reason why, it is this way and no another, don’t ask why 😉

Source : Projet Voltaire (French advanced certification)

“The Structure of Modern Standard French” by Maj-Britt Mosegaard Hansen goes further than ΥΣΕΡ26328 (above) and suggests the following rules. (They appear in a discussion of infinitive clauses as postponed subjects.)

1. In [infinitive] clauses that contain a subject attribute, the infinitive marker will take the form que de if the subject attribute is also an infinitive clause. Thus:

   Ce serait aller à l’encontre de mes principes que de rentrer sans payer ma dette

2. If the subject attribute takes the form of a noun phrase, either de or que de may be used. Thus:

   Ce serait une bonne idée que de rentrer sans payer ma dette, or

   Ce serait une bonne idée de rentrer sans payer ma dette