It is not colloquial but it is far from giving those using it pretentious airs. In fact, when speaking rather formally (to employees in the administrations, to people you don’t know, etc.) it is fine to use this form, which is standard. There is another one that is formal enough for that purpose.

• Quand est-ce que je dois … (also perfect to use among friends)

If you feel absolutely unconfortable with those the following colloquial options might change that but only the first is still acceptable in a formal context.

• Je dois le retourner quand ? (colloquial)
• C’est quand que je dois le retourner ? (colloquial)
• Quand c’est que je dois le retourner ? (very colloquial)