This question of whether to use the partitive article "de" has nothing to do with the use of an adverb.
You are dealing with forms that are idiomatic; whereas in English, for instance , you say "I am cold" in French you say "J’ai froid" and there is no way to make sense of "je suis froid" along the line of feeling the cold in your body ("je suis froid" has a meaning but it has to do with the person’s psychology). There is a number of such idiomatic forms; here are some of the most important ones.

• avoir peur, avoir faim, avoir soif, avoir mal, avoir envie, avoir froid, avoir chaud, avoir le vertige (here "le" is idiomatic, there is no explanation to justify it), …

• on a beaucoup peur de la douleur, ils ont un peu faim, elle a très peu chaud en été, …

Apparently, those do not follow the principle of treatment of mass nouns, although they seem to have to a certain extent the characteristics of the nouns of that class. It can be noticed, however that those names describe phenomena that have to do with effects which are rather of a physiological nature, whereas for the names that are treated according to the principle applied to mass nouns, some of those being listed next, the effects are of a psychological nature, but there is no perfect rule ("peur" and "envie" have to do with effects of a psychological nature).

The following nouns, which are similar to those in the preceding list, do follow the principle; I suspect that there is a greater number of these.

• avoir de la haine, avoir de l’amour, avoir du courage, avoir de la colère, avoir de la patience, …

• elle a beaucoup de haine, ils ont assez de courage, nous avons très peu de colère, …

In conclusion, I think that the syntax that is proper for these locutions has to learned on a case by case basis, without trying to rely too much on the imperfect principle of classification stated above, merely keeping it in mind as a queer guideline.

I would like to insist on the rules of thumbs given hereabove by Laurent S. in his comment:

• When something can be quantified you would (almost always) use "un peu de". Examples: un peu de pain, un peu d’argent, un peu de temps, un peu d’attente. Please note here one is using (almost always) nouns after un peu de.

• When it is about emotions or feelings, or you describe a certain state of someone or something, you would use "un peu". Examples: un peu peur, un peu froid, un peu admiratif, un peu penché. This would be especially the case when un peu is followed by an adjective (as in un peu admiratif) or a form from a verb (as in un peu penché).

I cannot guarantee it works in all cases, but this should allow you to be correct in 95% of cases.