The word "de" in French is used for many different things that have nothing in common most of the time — you should not try to look for a common sense between these usages. In particular it "de" can be used to introduce a description.

Here your problem seems to be the difference between these two sentences for example:

Il est bon d’être chez soi. (It is good being at home)

Être chez soi est bon. (Being at home is good)

In one case you have "de" ("d’" here) and in another you have nothing. In the first sentence "bon" is a vague notion that is detailed by a description: "d’être chez soi", whereas in the second we use "Être chez soi" as the subject so "bon" doesn’t need more details.

In your example, "difficile" is detailed by a long description: "de comprendre (…)", it is the same mechanics. It works even when you infinitive is used for describing something that is not an adjective:

J’enrage de m’être trompé -> I am angry I made a mistake

Some differents examples that work the same way (translations are done fast, I don’t guarantee they are good):

Tu essaies d’être à l’heure -> You try to be on time

Nous sommes remplis de l’espoir de gagner -> We are filled with the hope to win

Vous êtes déçus d’avoir perdu -> You are disappointed to have lost

“de” followed by an infinitive usually just means “to + verb”. Of course, “de” has other purposes (e.g. property “le chat de ma tante”), but they’re usually not followed by an infinitive.

By the way, your sentence:

Il n’est pas difficile de comprendre pourquoi le samedi est ma journée favorite.

is translated this way in English:

It’s not hard to understand why Saturday is my favourite day.

In this case, removing de would be exactly the same mistake as removing to in the English version:

It’s not hard understand why Saturday is my favourite day.

As other posters have mentioned, there are many different uses of "de" in French. To address your question specifically:

"de" is used in an impersonal form : Il est facile de + infinitif

Il est facile de se perdre = It’s easy to get lost.

Il est facile de comprendre l’anglais = It’s easy to understand English.

There is also "à":

"à" is used when it introduces a complement to facile

Quelque chose/ Quelqu’un est facile à + infinitif

L’anglais est facile à comprendre -> English is easy to understand.

John est facile à comprendre -> John is easy to understand.

Source: WordReference thread (I’ve adapted the answer a bit to better fit your question. The answer refers to facile (simple, easy) but it works exactly the same as difficile (difficult))

So if you’re talking about impersonal and general things (It is…), you would use difficile + de.

If you’re talking about specific things or want to emphasize something, you use Thing/Person est difficile à…