The first phrase is grammatically correct, the second one is not. In the second case you would have to use le (which would elide to l’). The reason for this is that the de following the verb essayer is part of the object of that verb and not the verb itself. In the first phrase, de is an inherent part of the construction être capable de.

This becomes more clear if you replace the object clause (what you are trying to replace with en) with cela:

• Est-il capable de cela ?
• Oui, il en est capable.

In this case de is part of the verb construction. In the following phrase it is not:

• A-t-elle essayé cela ?
• Oui, elle l’a essayé.

As you can clearly see in the phrases above, the correct construction is être capable de quelque chose and essayer quelque chose. As a result, when replacing the phrase with a pronoun in the answer to the question, you cannot use en in with essayer.

More specifically, in the first case, en replaces faire ses devoirs since de is part of the verb construction. In the second case, le is used to replace de faire ses devoirs, since de is not part of the verb construction.

Some French verbs are followed by de only when their object is a (subordinate) clause and not when the object is a noun. However, there are some that do not take de in that case either:

Il espère/souhaite/veux/doit faire ses devoirs.

Yet, as stated earlier, some verbs do:

Il essaye/envisage/tente/oublie de faire ses devoirs.

There is no grammatical rule to figure out when a verb does or does not take de with a clause as a subject. The rule of thumb is to just know your verbs and replace the object clause by cela or quoi to figure out whether or not de belongs to the verb or to the clause. If it is an inherent part of the verb, you replace the object by en. If not, you use le instead.